Meeting Title: Forge Lab Success Metrics Discussion Date: 2026-04-10 Meeting participants: Greg Stoutenburg, Miranda Wen
WEBVTT
1 00:01:47.660 ⇒ 00:01:48.729 Miranda Wen: Hi, Greg!
2 00:01:48.940 ⇒ 00:01:50.180 Greg Stoutenburg: Hey, Miranda!
3 00:01:50.180 ⇒ 00:01:51.929 Miranda Wen: Hi, nice to meet you.
4 00:01:51.930 ⇒ 00:01:54.800 Greg Stoutenburg: Nice to meet you, too. Where are you located?
5 00:01:55.050 ⇒ 00:01:57.749 Miranda Wen: Yeah, I mean, San Francisco right now, how about you?
6 00:01:58.130 ⇒ 00:02:09.879 Greg Stoutenburg: Oh, cool. You got a lot… you got a whole contingent out there. I’m in York, Pennsylvania, so East Coast, but not a major city. So, I’m just a little north of Baltimore.
7 00:02:10.310 ⇒ 00:02:12.400 Miranda Wen: Oh, yeah, I know where it is, yeah, yeah, yeah.
8 00:02:12.400 ⇒ 00:02:13.440 Greg Stoutenburg: Yeah. Yeah, yeah.
9 00:02:13.440 ⇒ 00:02:15.630 Miranda Wen: A lot of friends at Johns Hopkins, yeah.
10 00:02:15.830 ⇒ 00:02:23.619 Greg Stoutenburg: Oh, no kidding! Oh, cool. Nice. Yeah, we’re… we’re fortunate that their medical center is here. We all benefit from it.
11 00:02:24.360 ⇒ 00:02:28.699 Miranda Wen: Oh, yeah, that’s great. It’s definitely, like, the best in COVID, right? Like…
12 00:02:28.700 ⇒ 00:02:33.139 Greg Stoutenburg: Yeah, I’ll say. I know, they really, like, they sort of cracked the code.
13 00:02:34.300 ⇒ 00:02:34.840 Greg Stoutenburg: Yeah.
14 00:02:34.980 ⇒ 00:02:44.190 Miranda Wen: Yeah, yeah, thank you so much for hopping on this call with me. Yeah, like, I’m right now, like, working as a contractor for the AI product manager side.
15 00:02:44.190 ⇒ 00:02:56.850 Miranda Wen: working for this project called, like, Forge Lab, which is, like, gonna later be integrated into, like, the platform. And as I was, like, developing this plan, definitely a very key part is, like, we want to iterate rapidly, and…
16 00:02:56.860 ⇒ 00:03:03.010 Miranda Wen: what is based off, definitely the metrics we’re tracking here. But I don’t have, like, much…
17 00:03:03.060 ⇒ 00:03:10.279 Miranda Wen: experience in, like, setting, like, success metrics. My main question on a higher level is, like.
18 00:03:10.820 ⇒ 00:03:20.779 Miranda Wen: what are some things that are, like, really trackable, like, like, bringing forward? What are some metrics that are really trackable and related to, like.
19 00:03:20.990 ⇒ 00:03:24.499 Miranda Wen: and actually, like, good matches, like, North Star matches.
20 00:03:24.710 ⇒ 00:03:32.750 Miranda Wen: To, define whether or not it’s, like, a good match to track with that can influence our decisions in the future.
21 00:03:33.040 ⇒ 00:03:33.770 Greg Stoutenburg: Yeah.
22 00:03:33.770 ⇒ 00:03:38.100 Miranda Wen: Yeah, so right now in the doc, I can share the…
23 00:03:38.570 ⇒ 00:03:41.809 Greg Stoutenburg: Yeah, I’m in GitHub right now, so I have not,
24 00:03:42.090 ⇒ 00:03:48.690 Greg Stoutenburg: I have not read this, so if you could just, like, talk me through what the… what Forge Lab is?
25 00:03:48.690 ⇒ 00:03:49.210 Miranda Wen: Yes.
26 00:03:49.210 ⇒ 00:03:53.240 Greg Stoutenburg: And, you know, who would use it, and why would they use it?
27 00:03:53.360 ⇒ 00:03:58.080 Greg Stoutenburg: then I can help with, defining some success metrics, for sure.
28 00:03:58.460 ⇒ 00:04:09.139 Miranda Wen: Yes, yes, totally, yeah. That would be awesome, yeah. So basically Forge Lab is this, surface that can…
29 00:04:09.330 ⇒ 00:04:22.549 Miranda Wen: help with, like, the AI skills. So right now, we are using AI skills in cursor, right? It was, like, pretty… it’s not bad, like, we can… you can easily build one, and also you can easily, like, call it out.
30 00:04:22.610 ⇒ 00:04:30.570 Miranda Wen: But when we are syncing, like, packaging to, like, clients, it needs a surface to that. And also, like.
31 00:04:31.030 ⇒ 00:04:32.050 Miranda Wen: It just doesn’t…
32 00:04:32.050 ⇒ 00:04:40.399 Greg Stoutenburg: It means, like, a UI. Like, they open Forge Lab, and there are, like, different skills they can choose from in this UI, is that what you mean?
33 00:04:40.400 ⇒ 00:04:54.159 Miranda Wen: Yes, yes, I think that’s a part of what, like, the surface means. But obviously, like, it has, like, the models integrate itself, the, workflow, the automations that be integrated into this, like, UI. Yeah.
34 00:04:54.770 ⇒ 00:05:12.400 Miranda Wen: And now, like, I think ideally what we want is, like, to make it, like, a part of the platform, but right now, we run it independently, open work, and we’ll only, like, be making it, like, a part of the platform after this being fully tested out. It makes sense, yeah.
35 00:05:13.430 ⇒ 00:05:14.020 Greg Stoutenburg: Okay.
36 00:05:14.260 ⇒ 00:05:24.940 Miranda Wen: Yeah, and so, like, why are we making this? It’s because, like, Utem Singh’s, like, cursor is definitely…
37 00:05:25.040 ⇒ 00:05:28.950 Miranda Wen: Like, an overkill, and, it’s not like the…
38 00:05:29.140 ⇒ 00:05:46.929 Miranda Wen: most ideal place where, to let people… guide people through the… the whole, AI skills. So, like, for example, cursor, like, we need education on, like, what’s… how to, like, really communicate with cursors, as well as, like.
39 00:05:47.310 ⇒ 00:06:05.799 Miranda Wen: what are some skills available, like the discovery part, as well as, like, how does it work, the guided part. So, what we want is, like, to make it, like, very much easy to discover, guided, self-explanatory. It’s kind of, like, more chat-native, more, like, kind of like ChatGBT, like, you don’t need, like,
40 00:06:05.800 ⇒ 00:06:16.639 Miranda Wen: full onboarding tutorial for that. It’s just, like, naturally, you’re in this, you are able to, like, start working on this, and know what it’s, like, the output gonna be like, and repeat, and…
41 00:06:16.650 ⇒ 00:06:23.390 Miranda Wen: really, Support, like, the workflow, reuse, and make people reuse on that.
42 00:06:23.520 ⇒ 00:06:31.169 Miranda Wen: Yeah, so… right now, like, what we are making is, like, we’re first gonna, like, make sure this, like.
43 00:06:31.210 ⇒ 00:06:44.520 Miranda Wen: entry surface fully available, and then, like, we were gonna roll out for our internal user group adoption, and the first user group we’re choosing is, like, the go-to-market team. The reason we chose them is because
44 00:06:44.620 ⇒ 00:06:49.900 Miranda Wen: Right now, they have a lot of parts that they need.
45 00:06:50.160 ⇒ 00:07:00.289 Miranda Wen: AI automations for, and also, like, they are less technical, like, they are… so we… I kind of consider this, like, the operators, go-to-market teams.
46 00:07:00.350 ⇒ 00:07:11.190 Miranda Wen: the, CSO strategy, delivery team, and then the builders, like, engineers, like, more so power users. So the go-to-market team, like, ideally, they shouldn’t be the one who, like, need to…
47 00:07:11.190 ⇒ 00:07:21.819 Miranda Wen: build a skill from scratch, like, orchestrate… this shouldn’t be, like, their main focus. Their focus should be, like, really sell the product, communicate to clients, make sure everything’s organized on their end.
48 00:07:21.830 ⇒ 00:07:35.800 Miranda Wen: So, we think they would be the ideal, like, first user group to roll out to, and then, like, maybe each person from strategy, CSO delivery, or just one group from them, and then gradually roll out to, like, the whole company.
49 00:07:35.900 ⇒ 00:07:40.559 Miranda Wen: So that’s, like, the initial plan here.
50 00:07:40.600 ⇒ 00:07:58.200 Miranda Wen: And after we evaluate, like, first building on, like, some repeatable workflows for go-to-market team and test them out, we’re gonna be… have the, strategy CSO Deliverer wants to test it out, and if the feedback’s great, everything works fine, we will consider whether or not it makes sense.
51 00:07:58.200 ⇒ 00:08:06.619 Miranda Wen: For us to, like, really move to the builder side, which would be kind of different from this tool, because, like, we do expect builders to be the one who, like.
52 00:08:06.710 ⇒ 00:08:25.170 Miranda Wen: do the orchestration, more integrate with the systems. And, after that, at the end, ideally, we wanted to be, like, a part of, like, the package service when we are selling to clients, and also maybe can be, like, a…
53 00:08:25.390 ⇒ 00:08:29.950 Miranda Wen: product by itself, as well, to sell. For example, we had this, like, go-to-market
54 00:08:30.230 ⇒ 00:08:41.570 Miranda Wen: virtual app that really works well for our go-to-market team, and we will sell it to, like, the go-to-market team for our clients, which we also have the service for. So, that’s, like, the…
55 00:08:41.730 ⇒ 00:08:42.299 Greg Stoutenburg: Yeah.
56 00:08:42.470 ⇒ 00:08:44.049 Miranda Wen: Yeah, general overview.
57 00:08:44.860 ⇒ 00:08:45.440 Greg Stoutenburg: Okay.
58 00:08:45.740 ⇒ 00:09:05.419 Greg Stoutenburg: Okay, so the end goal… okay, so Forge Lab is a UI that enables the user at a client, regardless of who the client is, the client can be Brainforge, that enables the, the client… the users at the client to,
59 00:09:07.530 ⇒ 00:09:10.910 Greg Stoutenburg: create skills, or just use skills? Use skills.
60 00:09:11.660 ⇒ 00:09:18.089 Miranda Wen: Like, for now, at least for the first two user groups, I think directly use skills, and probably they can do.
61 00:09:18.090 ⇒ 00:09:19.359 Greg Stoutenburg: new skills. Okay.
62 00:09:19.360 ⇒ 00:09:21.950 Miranda Wen: Like, rather than available for them, like…
63 00:09:21.950 ⇒ 00:09:31.039 Greg Stoutenburg: So, like, speed up their work… yeah, okay. So, speed up their work, speed up workflows, by using skills, rather than just…
64 00:09:31.310 ⇒ 00:09:37.840 Greg Stoutenburg: generic text entry as one would on ChatGPT or other AI services.
65 00:09:38.030 ⇒ 00:09:38.640 Miranda Wen: Yes.
66 00:09:38.640 ⇒ 00:09:42.509 Greg Stoutenburg: That’s the goal, right? Okay. Yes. Okay. And then,
67 00:09:43.130 ⇒ 00:09:54.529 Greg Stoutenburg: The standards are self-explanatory, meaning just, like, really easy to use. Repeatable workflow execution, so you run a skill, and the same workflow is delivered
68 00:09:54.640 ⇒ 00:09:58.369 Greg Stoutenburg: Provided that you selected the skill appropriately.
69 00:09:58.730 ⇒ 00:10:02.190 Greg Stoutenburg: Low support adoption, can we look at that one again?
70 00:10:02.500 ⇒ 00:10:07.290 Miranda Wen: Yes, so low support adoptions, like, when I was doing the product review yesterday.
71 00:10:07.290 ⇒ 00:10:08.040 Greg Stoutenburg: Hi.
72 00:10:08.040 ⇒ 00:10:14.529 Miranda Wen: B asked me the question, like, how do you make sure, like, the people at the open, and how much effort do we need to, like.
73 00:10:14.910 ⇒ 00:10:26.540 Miranda Wen: take for our… internally, our clients to adopt it. But actually, I think, like, the core value that, Forge Lab ideally provides to it, like, the product itself, it needs, like.
74 00:10:27.510 ⇒ 00:10:34.580 Miranda Wen: very little, like, education on how to use it, compared to cursor onboarding for someone who never used cursor. Okay.
75 00:10:34.580 ⇒ 00:10:35.479 Greg Stoutenburg: Yeah, got it.
76 00:10:35.480 ⇒ 00:10:36.610 Miranda Wen: Faster than that, it’s true.
77 00:10:36.610 ⇒ 00:10:37.350 Greg Stoutenburg: Okay.
78 00:10:37.350 ⇒ 00:10:38.240 Miranda Wen: Salvation.
79 00:10:38.240 ⇒ 00:10:48.670 Greg Stoutenburg: Yeah. So an engineer at the client needs to have connected Forge Lab to whatever context is going to be relied upon, so that the end user doesn’t have to worry about that.
80 00:10:49.190 ⇒ 00:10:49.740 Miranda Wen: Yes.
81 00:10:49.740 ⇒ 00:10:50.460 Greg Stoutenburg: Okay.
82 00:10:51.130 ⇒ 00:11:01.769 Greg Stoutenburg: workflow reuse across teams, so teams use similar workflows, like, so for GTM, for example, if there’s something like, you know, they get an export of a bunch of emails or something like this.
83 00:11:02.320 ⇒ 00:11:11.380 Greg Stoutenburg: they run a support… they run a skill that’s like, turn all these CSVs, these emails in the CSV into…
84 00:11:12.360 ⇒ 00:11:20.370 Greg Stoutenburg: contacts and put them in, like, an MQL workflow or something like that. Could be a skill.
85 00:11:20.710 ⇒ 00:11:31.140 Greg Stoutenburg: And then clear operator experience without specialist tool changes. Yeah, okay, same. Okay, alright, okay, I think I understand what it’s supposed to do, and I understand what the initiatives are.
86 00:11:31.740 ⇒ 00:11:33.660 Miranda Wen: Okay, awesome. Okay.
87 00:11:33.880 ⇒ 00:11:38.090 Miranda Wen: Yeah, yeah, these are just, like, more detailed explanations of, like, those,
88 00:11:38.090 ⇒ 00:11:38.650 Greg Stoutenburg: Yeah.
89 00:11:41.190 ⇒ 00:11:46.510 Greg Stoutenburg: Alright, so let’s look at, yeah, let’s look at success metrics. So, is this the part you want my help with?
90 00:11:46.510 ⇒ 00:11:50.999 Miranda Wen: Yes, yes, definitely. Like, this right now is, like, more so…
91 00:11:51.410 ⇒ 00:11:58.380 Miranda Wen: in very much ideal scenarios, but definitely, like, a lot of metrics. I think the part that’s kind of hard is, like.
92 00:11:58.480 ⇒ 00:12:04.549 Miranda Wen: I’m not sure how to, like, track them, and what are the ones who are trackable? It’s my… his,
93 00:12:04.720 ⇒ 00:12:08.759 Miranda Wen: I mean, it’s just, like, hard to monitor, like…
94 00:12:08.760 ⇒ 00:12:10.470 Greg Stoutenburg: Yeah.
95 00:12:10.470 ⇒ 00:12:16.980 Miranda Wen: like, an indirect metrics to test, like, some ideal cases, and also I want to get your feedback on this.
96 00:12:16.980 ⇒ 00:12:19.070 Greg Stoutenburg: Yeah, I think…
97 00:12:20.130 ⇒ 00:12:35.109 Greg Stoutenburg: Because I’ve not been in the loop on the development of Forge Lab, I don’t know what’s possible to measure. I think I can help weigh in on what would be a good idea to measure and consider a success metric. So, maybe we just go one at a time.
98 00:12:35.220 ⇒ 00:12:38.340 Greg Stoutenburg: Selected internal users, that means the GTM team, right?
99 00:12:38.980 ⇒ 00:12:39.680 Miranda Wen: Yes.
100 00:12:40.020 ⇒ 00:12:40.750 Greg Stoutenburg: Okay.
101 00:12:41.600 ⇒ 00:12:44.480 Greg Stoutenburg: Complete meaningful, repeatable workflows.
102 00:12:44.950 ⇒ 00:12:46.300 Greg Stoutenburg: successfully.
103 00:12:46.410 ⇒ 00:12:53.049 Greg Stoutenburg: I would… I mean, how fine-grained should I be here?
104 00:12:54.930 ⇒ 00:12:57.299 Greg Stoutenburg: I, I, I… yeah, sorry, I…
105 00:12:57.460 ⇒ 00:13:00.579 Greg Stoutenburg: I shouldn’t have asked that question yet. So, I think for the first one, I would say.
106 00:13:01.810 ⇒ 00:13:05.859 Greg Stoutenburg: Selected internal users, run a workflow.
107 00:13:07.150 ⇒ 00:13:15.330 Greg Stoutenburg: I would just say they run a workflow. And then the target would be, I would take that 70 down to,
108 00:13:15.870 ⇒ 00:13:18.059 Greg Stoutenburg: I would take that down to, like, 50.
109 00:13:18.690 ⇒ 00:13:21.940 Greg Stoutenburg: The reason being, if you think of this as, like, a…
110 00:13:21.990 ⇒ 00:13:37.320 Greg Stoutenburg: free SaaS tool. Getting to, like, 40 is pretty darn good. So the goal would be, like, they complete a workflow successfully on the first run. That’s gonna give us, like, okay, we’re getting a little bit of adoption here.
111 00:13:37.340 ⇒ 00:13:42.380 Greg Stoutenburg: I do like the idea of a return for a second workflow.
112 00:13:42.670 ⇒ 00:13:49.400 Greg Stoutenburg: I think I’d like to see some… I’d actually like to see a higher percentage there, so maybe… actually, if it were me, I would flip those percentages.
113 00:13:49.580 ⇒ 00:14:04.540 Greg Stoutenburg: And go, alright, so we get, you know, we get, like, half the team to try this thing, and then the ones who tried it, if they come back at a high rate, that’s an indicator that they found value when they ran it.
114 00:14:05.080 ⇒ 00:14:06.830 Miranda Wen: Oh, I see, I see.
115 00:14:07.290 ⇒ 00:14:16.640 Greg Stoutenburg: Yeah, I would, I would, yeah, I would not want to see drop-off there. Yeah. Time to first success is short enough to use to understand.
116 00:14:25.910 ⇒ 00:14:35.449 Greg Stoutenburg: Median time from first visit to successful workflow completion, 15 minutes is a, I think, a very long time compared to, like, running a cursor skill.
117 00:14:35.530 ⇒ 00:14:47.159 Greg Stoutenburg: So, if you understand what the cursor skill is, then you just, like, open an agent, you type slash, name of skill, and then you just say whatever else you want to say, and you hit enter, and it’s typically done in more like 1 minute.
118 00:14:47.400 ⇒ 00:14:52.899 Greg Stoutenburg: So, I think… I think I would… I think I would just reduce that to, like.
119 00:14:54.110 ⇒ 00:14:56.040 Greg Stoutenburg: I think I’d say 4 minutes.
120 00:14:56.460 ⇒ 00:15:04.269 Miranda Wen: Mmm, I see. Yeah, I was thinking of the 15 minutes, and cool, like, they just, like, just throw them this product, and they… they need to figure out, like…
121 00:15:04.270 ⇒ 00:15:04.820 Greg Stoutenburg: Yeah.
122 00:15:04.910 ⇒ 00:15:06.390 Miranda Wen: Contacts, like, just, like.
123 00:15:06.390 ⇒ 00:15:09.309 Greg Stoutenburg: Yeah, if I’m imagining, like…
124 00:15:13.030 ⇒ 00:15:13.980 Greg Stoutenburg: So…
125 00:15:14.300 ⇒ 00:15:32.859 Greg Stoutenburg: I’m gonna share my screen just for a second, just to talk about UI. If I imagine that I’m a brand new user of ChatGPT, and the Forge Lab interfaces like this, right? Like, if I come here, and I hit, like, this plus button, and it’s like a list of skills, you know, and right here is, like.
126 00:15:32.940 ⇒ 00:15:41.129 Greg Stoutenburg: turn, turn emails into leads.
127 00:15:41.250 ⇒ 00:15:53.170 Greg Stoutenburg: For example. And I just, like, click that button, and then drag a file in and go boom, right? Then, like, the amount of time that it took me to go from landing on the page to finding value is…
128 00:15:53.620 ⇒ 00:16:11.799 Greg Stoutenburg: is the amount of time that it took me to find the skill, plus the amount of time that it took the skill to actually run and turn them into leads. So I think… so I do think that the time can be much shorter. If the UI is something… if the UI is something like this, I think that the time to value can be much shorter.
129 00:16:12.570 ⇒ 00:16:15.490 Miranda Wen: Okay, could you share your screen? I can…
130 00:16:15.590 ⇒ 00:16:24.449 Greg Stoutenburg: According to Zoom, I am sharing my screen. Are you able to see it? Maybe the… maybe you have to be on a different tab or something? Is that possible?
131 00:16:24.730 ⇒ 00:16:27.129 Miranda Wen: Oh, yeah, yeah, I got it, I got it.
132 00:16:28.300 ⇒ 00:16:32.280 Miranda Wen: Yeah, yeah, yeah. I see, like, this, session thing, yeah.
133 00:16:33.850 ⇒ 00:16:34.400 Miranda Wen: Yeah.
134 00:16:35.920 ⇒ 00:16:47.600 Greg Stoutenburg: Yeah, so, yeah, so all I was trying to say is, like, if the UI is something like this, and someone comes in, then they go, like, you know, click this button, and it’s like.
135 00:16:47.910 ⇒ 00:16:56.290 Greg Stoutenburg: turn contacts into leads, or something like that. And then they… they select the skill they want to run, and then they drop in the CSV here.
136 00:16:56.960 ⇒ 00:17:01.130 Greg Stoutenburg: And then hit go, you know? Then the amount of time… then the time to value is, like.
137 00:17:01.290 ⇒ 00:17:01.900 Miranda Wen: Yeah.
138 00:17:01.900 ⇒ 00:17:04.200 Greg Stoutenburg: The amount of seconds that it takes to
139 00:17:04.480 ⇒ 00:17:06.519 Greg Stoutenburg: Find the skill they want to run.
140 00:17:06.670 ⇒ 00:17:22.839 Greg Stoutenburg: Which is, you know… I don’t know how patient you are, I think I have an average amount of patience. If I click this button, and there’s, like, 10 things on this list, and I feel like I can’t locate the one I want in a minute, I’m never coming back. You know what I mean?
141 00:17:22.849 ⇒ 00:17:24.209 Miranda Wen: True, true, true, yeah.
142 00:17:24.210 ⇒ 00:17:29.949 Greg Stoutenburg: So… Yeah, and then there’s just the run itself, right? So they select it, and then they go.
143 00:17:30.330 ⇒ 00:17:31.680 Greg Stoutenburg: you know, enter.
144 00:17:31.820 ⇒ 00:17:40.890 Greg Stoutenburg: Yeah, so, anyway. That’s a long-winded way of saying I think I would take that 15… I would take the 15 way down. Oh, shoot. Did I cut your screen share? I hope not.
145 00:17:40.890 ⇒ 00:17:43.909 Miranda Wen: No, no, no, yeah, I stopped it, but yeah, it’s fine.
146 00:17:45.020 ⇒ 00:18:00.189 Miranda Wen: Yeah, I think, yeah, that’s totally valid. And I also think, like, yeah, I think benchmarking with cursor really makes sense. Like, what we need, like, we definitely want the median time to be, like, shorter than cursor, otherwise it doesn’t align with the faster time to value we are…
147 00:18:00.190 ⇒ 00:18:00.950 Greg Stoutenburg: Yeah.
148 00:18:01.080 ⇒ 00:18:02.600 Miranda Wen: Wanted to achieve, yeah.
149 00:18:06.390 ⇒ 00:18:09.899 Greg Stoutenburg: Okay, okay, is that helpful?
150 00:18:09.900 ⇒ 00:18:11.089 Miranda Wen: Yes, yes, that’s hard.
151 00:18:11.200 ⇒ 00:18:14.750 Greg Stoutenburg: Yeah, I’m just trying to think through, you know, the user experience of it.
152 00:18:15.530 ⇒ 00:18:23.510 Greg Stoutenburg: And then at least 6% of pilot users complete a first meaningful workflow without a live walkthrough.
153 00:18:25.120 ⇒ 00:18:26.170 Greg Stoutenburg: Yeah.
154 00:18:27.150 ⇒ 00:18:34.539 Greg Stoutenburg: if it were me, I’d probably just delete that part, because given that one of the goals is that…
155 00:18:35.080 ⇒ 00:18:52.050 Greg Stoutenburg: no formal introduction is needed, then, then the whole thing is just self-service. So, it should be that 100% of users complete a first meaningful workflow without a live walkthrough. But then that part would already be covered by the first success metric.
156 00:18:57.310 ⇒ 00:18:58.070 Miranda Wen: Let me move…
157 00:19:01.380 ⇒ 00:19:10.430 Miranda Wen: Okay, yeah, I think maybe I should frame the first one a bit differently, like, like… Like, likes, like…
158 00:19:11.210 ⇒ 00:19:12.440 Miranda Wen: Only.
159 00:19:12.610 ⇒ 00:19:21.060 Miranda Wen: So do you think that should be, like, 40% of Pile users complete a meaningful workflow successfully on first run, without, like, without a live walkthrough?
160 00:19:22.140 ⇒ 00:19:24.580 Greg Stoutenburg: Yeah, I mean, basically the idea is that, like.
161 00:19:25.890 ⇒ 00:19:29.640 Greg Stoutenburg: What you’re trying to do is quantify an answer to the question.
162 00:19:29.640 ⇒ 00:19:30.130 Miranda Wen: Yeah.
163 00:19:30.130 ⇒ 00:19:32.330 Greg Stoutenburg: Can someone log in and use this?
164 00:19:32.450 ⇒ 00:19:34.349 Greg Stoutenburg: Yes or no, right?
165 00:19:35.100 ⇒ 00:19:39.389 Greg Stoutenburg: And you’re trying to go, well, we want the answer to be yes.
166 00:19:39.690 ⇒ 00:19:42.559 Greg Stoutenburg: But out of a huge number of users.
167 00:19:43.280 ⇒ 00:19:50.420 Greg Stoutenburg: How high… how many of them would need to be able to do it in order for us to say, yeah, someone can do this on their own.
168 00:19:50.750 ⇒ 00:19:51.280 Miranda Wen: Mmm.
169 00:19:51.280 ⇒ 00:19:51.940 Greg Stoutenburg: Right?
170 00:19:54.330 ⇒ 00:19:55.819 Miranda Wen: Yes, yes, yes, yes.
171 00:19:59.170 ⇒ 00:20:02.309 Miranda Wen: Yes, yeah, yeah, I think… I think that’s super valid, yeah.
172 00:20:10.150 ⇒ 00:20:18.789 Greg Stoutenburg: Sorry, let me just check… oh yeah, okay, alright. My, my, calendar just beeped at me. Okay, so does that make sense?
173 00:20:18.790 ⇒ 00:20:19.930 Miranda Wen: Yes, totally, totally.
174 00:20:19.930 ⇒ 00:20:32.970 Greg Stoutenburg: Okay, and then time to first success is, yeah, this is a time-to-value metric common in SaaS. In general, you know, the shorter, the better. I think that the time to success should be…
175 00:20:33.150 ⇒ 00:20:36.619 Greg Stoutenburg: On average, just a couple of minutes, because
176 00:20:36.840 ⇒ 00:20:45.539 Greg Stoutenburg: The idea of running a skill is that it saves you the amount of time that would be required even to do a lot of prompting, because the skill just covers the prompting.
177 00:20:46.820 ⇒ 00:20:47.410 Greg Stoutenburg: Yeah.
178 00:20:48.960 ⇒ 00:20:51.400 Greg Stoutenburg: Yup.
179 00:20:51.660 ⇒ 00:20:57.800 Greg Stoutenburg: Alright, users report higher confidence in guided Forge Lab workflows than in generic chat for the same recurring text. Yes.
180 00:20:58.330 ⇒ 00:21:04.600 Greg Stoutenburg: What’s a good CSAT score? I’m not actually sure what the answer to that is.
181 00:21:08.200 ⇒ 00:21:14.260 Miranda Wen: Family, I… Yeah, also whether or not it’s, like, trackable, like…
182 00:21:14.260 ⇒ 00:21:19.699 Greg Stoutenburg: Yeah, Gemini says it’s, 70. So, I think you got it.
183 00:21:22.840 ⇒ 00:21:24.420 Greg Stoutenburg: Yeah, alright.
184 00:21:25.790 ⇒ 00:21:43.139 Greg Stoutenburg: All right, I’ll leave the third one alone. A small number of expert-built workflows are reused across multiple users or teams, showing the Forge lab can turn specialist knowledge into repeatable internal products. At least two are used by 3 or more teams. That one’s hard for me, because I don’t know how, I don’t know how big the team is. I mean, I think the go-to-market team here is only, like.
185 00:21:43.690 ⇒ 00:21:45.379 Greg Stoutenburg: It’s only, like, 5 people.
186 00:21:45.380 ⇒ 00:21:47.439 Miranda Wen: Yeah, 4 or 5 people, including.
187 00:21:47.440 ⇒ 00:21:48.240 Greg Stoutenburg: Yeah.
188 00:21:48.840 ⇒ 00:21:53.280 Greg Stoutenburg: So… Yeah.
189 00:21:54.640 ⇒ 00:21:59.469 Greg Stoutenburg: Yeah, that one’s tough. I mean, if this is just about the pilot, then I think leave that one as it is.
190 00:21:59.990 ⇒ 00:22:00.930 Miranda Wen: Okay.
191 00:22:01.130 ⇒ 00:22:10.120 Greg Stoutenburg: Early adoption is strongest in business user workflows with clear operational value in those usage patterns. Index. 30% a weekly active.
192 00:22:17.610 ⇒ 00:22:25.529 Greg Stoutenburg: Yeah, I’d suggest making that much higher, because, even on a small team.
193 00:22:25.860 ⇒ 00:22:26.270 Miranda Wen: Hmm.
194 00:22:26.270 ⇒ 00:22:32.479 Greg Stoutenburg: If Forge Lab is gonna do what Forge Lab is for, then we would see users active
195 00:22:32.940 ⇒ 00:22:36.530 Greg Stoutenburg: pretty steadily. Like, if this is something…
196 00:22:37.000 ⇒ 00:22:40.240 Greg Stoutenburg: Like, put differently, if this is something that someone comes…
197 00:22:40.370 ⇒ 00:22:52.590 Greg Stoutenburg: No. No one is gonna come and use this occasionally. Like, no one occasionally uses a cursor skill, you know what I mean? Like, it’s part of your workflow or it isn’t, provided that the right skills are built into the platform.
198 00:22:52.700 ⇒ 00:22:58.989 Greg Stoutenburg: So, I think… Yeah, I think for this one, I’d say more like 80%.
199 00:22:58.990 ⇒ 00:23:00.730 Miranda Wen: Excuse me.
200 00:23:01.120 ⇒ 00:23:02.500 Greg Stoutenburg: I know that’s a high number.
201 00:23:04.500 ⇒ 00:23:06.970 Miranda Wen: Yeah, especially for pilot, I feel, yeah.
202 00:23:06.970 ⇒ 00:23:09.600 Greg Stoutenburg: Yeah, I think for Pilot, I would make that much higher.
203 00:23:09.600 ⇒ 00:23:09.980 Miranda Wen: Yeah, totally.
204 00:23:09.980 ⇒ 00:23:10.580 Greg Stoutenburg: Yeah.
205 00:23:13.450 ⇒ 00:23:18.369 Greg Stoutenburg: Alright, what else is missing? Will users have the ability to build their own workflows?
206 00:23:19.390 ⇒ 00:23:25.490 Miranda Wen: I think, like, for the first internal user group, like, we are… we don’t want them to be the…
207 00:23:25.490 ⇒ 00:23:31.779 Greg Stoutenburg: Yeah, okay. Yeah, and it might not be important in, you know, even in an external sales context, because…
208 00:23:31.960 ⇒ 00:23:34.360 Greg Stoutenburg: Users who are on some team
209 00:23:34.620 ⇒ 00:23:38.229 Greg Stoutenburg: You know, if there’s a process that they’re supposed to follow.
210 00:23:38.350 ⇒ 00:23:45.899 Greg Stoutenburg: Then they should just follow the process, and then the skills will help them execute that, rather than, you know, encouraging
211 00:23:46.470 ⇒ 00:23:51.130 Greg Stoutenburg: alternative ways of trying to hit some goal. Alright, so we’ve got…
212 00:23:51.490 ⇒ 00:24:00.090 Greg Stoutenburg: We’ve got initial time to value covered, we’ve got first important action, we’ve got repeatable action, we’ve got,
213 00:24:00.400 ⇒ 00:24:05.500 Greg Stoutenburg: We’ve got consistent engagement in the form of return,
214 00:24:08.290 ⇒ 00:24:13.459 Greg Stoutenburg: Yeah, I mean, that’s what I would be looking for for a pilot, so… yeah.
215 00:24:16.520 ⇒ 00:24:27.559 Miranda Wen: Yeah, what do you think about, like, I think I was most unsure about, like, in terms of actually putting into implantation and tracking. I think the first one and the last one…
216 00:24:27.660 ⇒ 00:24:30.850 Miranda Wen: Like… Like, how should I track that? Do…
217 00:24:31.020 ⇒ 00:24:35.990 Miranda Wen: Like, in terms of methodology, or if it even…
218 00:24:36.120 ⇒ 00:24:42.599 Miranda Wen: Like, the most ideal, way of… the most ideal metric to track.
219 00:24:43.110 ⇒ 00:24:49.070 Greg Stoutenburg: Yeah, I mean, I think, I think for measuring any of these, like, I mean…
220 00:24:51.710 ⇒ 00:25:01.279 Greg Stoutenburg: Yeah, I mean, they’re definitely all measurable. Any product analytics tool, for example, would tell you this. As long as you’ve got users who have to be signed in.
221 00:25:01.720 ⇒ 00:25:09.359 Greg Stoutenburg: Then you’ll know which unique users are performing these actions, and can consolidate in some kind of report.
222 00:25:09.670 ⇒ 00:25:13.690 Greg Stoutenburg: Internally, we report on cursor usage. I know that.
223 00:25:13.830 ⇒ 00:25:18.330 Greg Stoutenburg: I feel like I’m not answering the question, but I’m trying to, though.
224 00:25:23.980 ⇒ 00:25:30.680 Miranda Wen: Yeah, like, we, right now it’s, like, raw to work, and I think when we roll to the external user, like, it will…
225 00:25:31.180 ⇒ 00:25:38.739 Miranda Wen: still be on… like… Open work, like, yeah, cause it doesn’t, like…
226 00:25:39.430 ⇒ 00:25:49.029 Miranda Wen: it’s not like, oh, it can actually be by, like, like, I don’t know, how many pull requests, it doesn’t… not into its, like, prob… it will mostly be, like…
227 00:25:51.310 ⇒ 00:25:56.639 Miranda Wen: I don’t know, like, would it be, like, running, like, more… Locally, and
228 00:25:57.240 ⇒ 00:26:07.299 Miranda Wen: And so, I’m not too worried about, like, the second one, because I feel like if the first-time user, I can literally just, like, have the zoom on, like, to see how they… how they want.
229 00:26:07.300 ⇒ 00:26:07.770 Greg Stoutenburg: Yeah.
230 00:26:07.770 ⇒ 00:26:11.030 Miranda Wen: First one… the first workflow, and also the third one…
231 00:26:11.260 ⇒ 00:26:13.490 Miranda Wen: I feel this is more so, like.
232 00:26:13.730 ⇒ 00:26:18.489 Miranda Wen: at, like, interview or let them feel inserted.
233 00:26:18.690 ⇒ 00:26:23.449 Miranda Wen: Yeah. Yeah. But when they are, like, mashing the…
234 00:26:24.100 ⇒ 00:26:29.369 Miranda Wen: yeah, the activity they’re using, and I think, I think probably, and also, like, Halt.
235 00:26:30.090 ⇒ 00:26:39.449 Miranda Wen: how many times they return for each one, like, those aren’t the things I’m not sure if it’s, like, this match is the most ideal. Like, directly, yes, it’s ideal, but…
236 00:26:40.550 ⇒ 00:26:44.890 Miranda Wen: Some, like, indirect measure can make, like, easier for me to measure.
237 00:26:45.170 ⇒ 00:26:53.289 Greg Stoutenburg: Well, I mean, I guess… I guess I don’t know how the thing is built, but if this is something that we’re using internally, I would think that, like.
238 00:26:54.640 ⇒ 00:26:57.750 Greg Stoutenburg: I mean, I would think that every one of these would be something that…
239 00:26:58.580 ⇒ 00:27:01.260 Greg Stoutenburg: We’d set up, that you’d be able to use.
240 00:27:01.780 ⇒ 00:27:06.670 Greg Stoutenburg: So, yeah, I mean, I guess I just don’t really know what the limitations are.
241 00:27:06.840 ⇒ 00:27:11.279 Miranda Wen: Mmm, okay, okay. Wait, I think, wait, limitation is a good point. I think that’s for, like…
242 00:27:11.710 ⇒ 00:27:16.270 Miranda Wen: Like, include this part as a part of the limitation, and .
243 00:27:16.270 ⇒ 00:27:21.890 Greg Stoutenburg: Yeah, I mean, yeah, I guess in… or maybe just, like, in risks and mitigations, you call out
244 00:27:22.130 ⇒ 00:27:28.480 Greg Stoutenburg: Like, hey, we have to have the tools in place to actually measure these things that we’re calling success metrics.
245 00:27:29.330 ⇒ 00:27:32.929 Greg Stoutenburg: They’re pretty basic, there’s nothing very complicated about these success metrics, so…
246 00:27:33.080 ⇒ 00:27:40.709 Greg Stoutenburg: we need to have a way to measure them, or we need to find alternative metrics. And if what that comes down to is, like, at the end of the day, it’s just…
247 00:27:40.990 ⇒ 00:27:44.859 Greg Stoutenburg: we want to measure these, but we’re going to do user interviews, then I guess what you have to do is, like.
248 00:27:45.290 ⇒ 00:27:54.829 Greg Stoutenburg: Turn it… turn the tool over to a client, email people every day and tell them to use it, and then say, we’re gonna do user interviews at the end of week two.
249 00:27:55.430 ⇒ 00:27:57.960 Greg Stoutenburg: And I’m gonna ask you these questions.
250 00:27:58.210 ⇒ 00:28:02.390 Greg Stoutenburg: How many, you know, did you complete a workflow on your first run?
251 00:28:02.690 ⇒ 00:28:03.220 Miranda Wen: Yeah.
252 00:28:03.220 ⇒ 00:28:05.250 Greg Stoutenburg: Did you come back for a second run?
253 00:28:05.680 ⇒ 00:28:09.190 Greg Stoutenburg: How long did it take you to get your first workflow executed?
254 00:28:09.740 ⇒ 00:28:11.520 Greg Stoutenburg: Did this speed you up?
255 00:28:13.810 ⇒ 00:28:30.760 Greg Stoutenburg: And then, you know, say more, okay, and then you ask the manager, alright, for people on your team, or you ask, you know, all the ICs, whatever, how many of you used, you know, workflow A? Okay, how many used workflow B? And you’re hoping to get two hands to go up, or, you know, three hands to go up. Yeah.
256 00:28:31.080 ⇒ 00:28:33.489 Greg Stoutenburg: Alright, I’m gonna have to hop in one minute.
257 00:28:33.490 ⇒ 00:28:36.729 Miranda Wen: Yes, yes, yeah, I agree. I think this is, like, super helpful, yeah, thank you.
258 00:28:36.730 ⇒ 00:28:39.200 Greg Stoutenburg: Okay, good. Alright, this was helpful? Okay, good, I’m glad.
259 00:28:39.540 ⇒ 00:28:41.100 Miranda Wen: Yes, yes. Cool.
260 00:28:41.100 ⇒ 00:28:53.299 Greg Stoutenburg: Yeah, alright. Great. Well, hey, great to meet you, and good luck on the project. Sounds… I mean, it sounds great. I think that we’ve done awesome with cursor skills, but, like, to go to the next step and have something that’s just, like, a little more user-friendly would be amazing. So…
261 00:28:53.300 ⇒ 00:28:54.720 Miranda Wen: Yes, yes.
262 00:28:55.200 ⇒ 00:28:55.780 Greg Stoutenburg: like it.
263 00:28:55.780 ⇒ 00:28:58.070 Miranda Wen: pretty well, yeah. Yeah.
264 00:28:58.070 ⇒ 00:29:01.039 Greg Stoutenburg: Alright, yeah, have a good weekend. Take care. Alright, bye, Miranda.