Meeting Title: ReadMe <> Brainforge Check-In Date: 2025-12-04 Meeting participants: Robert Tseng, Alicia Shin, Phoebe Miller, Mustafa Raja
WEBVTT
1 00:01:18.070 ⇒ 00:01:19.010 Alicia Shin: Bye!
2 00:01:20.230 ⇒ 00:01:21.140 Robert Tseng: Hey, Alicia.
3 00:01:22.020 ⇒ 00:01:23.470 Alicia Shin: You’re back from Hong Kong?
4 00:01:23.970 ⇒ 00:01:24.940 Robert Tseng: I am.
5 00:01:25.220 ⇒ 00:01:26.580 Alicia Shin: Travel back.
6 00:01:27.180 ⇒ 00:01:37.869 Robert Tseng: It was good. I’m actually still a little jet-lagged, but yeah, I mean, it’s much colder here in New York than there. The weather in Hong Kong is actually perfect at this time of year, so…
7 00:01:38.460 ⇒ 00:01:39.380 Alicia Shin: I bet.
8 00:01:39.470 ⇒ 00:01:42.709 Robert Tseng: Yeah Hey, Phoebe.
9 00:01:42.950 ⇒ 00:01:44.250 Phoebe Miller: Hey, how’s it going?
10 00:01:44.720 ⇒ 00:01:45.770 Robert Tseng: Good, how are you?
11 00:01:45.770 ⇒ 00:01:47.089 Phoebe Miller: Yeah, doing well, thanks.
12 00:01:48.080 ⇒ 00:01:52.419 Robert Tseng: I have Mustafa on my team, kind of just… he’s completely a fly on the wall, today.
13 00:01:52.830 ⇒ 00:01:53.430 Phoebe Miller: Sounds good.
14 00:01:58.080 ⇒ 00:02:02.800 Robert Tseng: I think that’s it from our side, so, I mean, happy to jump into it whenever, whenever you’re ready.
15 00:02:02.800 ⇒ 00:02:03.850 Phoebe Miller: Yeah, let’s do it.
16 00:02:04.180 ⇒ 00:02:05.180 Robert Tseng: Okay, great.
17 00:02:05.360 ⇒ 00:02:10.500 Robert Tseng: Yeah, so I know we just checked in on Tuesday, so, we got, like, kind of the…
18 00:02:10.770 ⇒ 00:02:13.079 Robert Tseng: The deck, from you, and…
19 00:02:13.220 ⇒ 00:02:22.769 Robert Tseng: we kind of just basically broke each of those experiments out. I think it’s gonna end up being, like, 3 dashboards, because we’re grouping all the pricing ones together.
20 00:02:22.810 ⇒ 00:02:31.599 Robert Tseng: And so we’ve already… yeah, we’re kind of… we’ll finish… we’ll finish them all by the end of the week, at least all the ones that have not… that have already been launched.
21 00:02:31.630 ⇒ 00:02:46.620 Robert Tseng: For today, we can kind of flash the pricing one, and just kind of use that as, like, to have the discussion of if there’s anything we want to change about how the dashboard is being presented, we can kind of just, like, copy-paste that across the other reports.
22 00:02:46.720 ⇒ 00:02:48.809 Robert Tseng: And then…
23 00:02:48.990 ⇒ 00:03:03.749 Robert Tseng: I think after that, like, if we want to spend more time discussing, yeah, just specifically the metrics for each experiment, like, happy to kind of spend time doing that, or if you wanted to kind of talk about anything else, like, you know, pretty much we have, like.
24 00:03:03.970 ⇒ 00:03:09.590 Robert Tseng: You know, up… once… I don’t know if the launch dates are changed, but
25 00:03:09.730 ⇒ 00:03:21.829 Robert Tseng: Yeah, there’s some stuff going out by tomorrow, and then also next week. Those are really kind of the next reports that we have kind of built out, and so I think we’re pretty open to trying to add other things to the roadmap as well.
26 00:03:22.040 ⇒ 00:03:24.179 Phoebe Miller: Cool. Yeah, let’s,
27 00:03:24.320 ⇒ 00:03:44.089 Phoebe Miller: So I want to get to a place where, an amplitude is just, like, super, super simple, because I think, you know, we’ve tried a bunch of things, and so now that self-serve conversion space has gotten a little messy. So, after this call, maybe we can just get to, like, the most simplistic view. It’s probably just, like.
28 00:03:44.090 ⇒ 00:04:00.599 Phoebe Miller: through some folder organization, or, you know… Sure. Like, the most simplistic view, so that when we go in, we know exactly what we’re looking at, because I think there’s, like, certain reports that the data is verified, and then certain ones that aren’t, and now that we feel like…
29 00:04:00.600 ⇒ 00:04:09.659 Phoebe Miller: we’re in the right place. I just want to be able to point to certain stuff at any point in time, and be like, this is what’s going on. So that’s just a global point. But yes, let’s.
30 00:04:09.660 ⇒ 00:04:10.480 Robert Tseng: Okay, sure.
31 00:04:10.480 ⇒ 00:04:22.859 Phoebe Miller: Let’s review, the dashboards or reports that correspond to the deck, and then we can talk about, the upcoming launches and make sure we’re prepared to track on those.
32 00:04:23.350 ⇒ 00:04:27.950 Robert Tseng: Okay, that sounds good. Alright, let me share my screen.
33 00:04:30.570 ⇒ 00:04:42.109 Phoebe Miller: And I think there’s probably, like, a million different ways to slice the data, and my general note is just, like, the most simple will probably work best for this group, at least initially.
34 00:04:42.110 ⇒ 00:04:46.230 Robert Tseng: Sure, sure, yeah. I know we have different, like, iterations of stuff.
35 00:04:46.230 ⇒ 00:04:46.620 Phoebe Miller: Yeah.
36 00:04:46.620 ⇒ 00:04:47.780 Robert Tseng: Floating around everywhere.
37 00:04:48.380 ⇒ 00:05:04.500 Robert Tseng: Okay, so, I mean, this is a bit small, you can kind of zoom in on your own screen, but, you know, we were… we were here, and so when I talk about pricing, you know, this one, the Super Hub, and, like, enterprise pricing will all kind of just be on, like, one dashboard.
38 00:05:04.590 ⇒ 00:05:23.360 Robert Tseng: I, yeah, so I think even just from, like, a file organization perspective, we started off with, like, notebook style, adding a lot more context. Once the reports are pretty fixed, like, we’ll just transition them to dashboards, and we’ll just kind of… we’ll do folders, like, one that’s, like, you know, kind of like a prod staging kind of situation. Cool.
39 00:05:23.650 ⇒ 00:05:42.399 Robert Tseng: Yeah, so I think the SuperHub, report is… is here. It’s like, we can see two weeks before, 2 weeks after launch, there is, like, an incremental lift in, kind of, the paid conversion rate going up after that from, you know, this percentage. These are low numbers, so it’s, you know, it is what it is. But I will…
40 00:05:42.400 ⇒ 00:05:45.400 Phoebe Miller: Would you call that incremental if it’s 4X?
41 00:05:45.560 ⇒ 00:05:49.579 Phoebe Miller: I would say that’s pretty… significant. I know the numbers are small, but…
42 00:05:49.990 ⇒ 00:06:00.169 Robert Tseng: Yeah, yeah. No, I mean, it’s, yeah, it’s incremental, yeah, I mean, I guess, I mean, if you… if you’re… the magnitude of it is… Yeah. Yeah, I guess, sure.
43 00:06:00.750 ⇒ 00:06:01.450 Robert Tseng: Okay.
44 00:06:01.450 ⇒ 00:06:02.600 Phoebe Miller: I think that’s great.
45 00:06:02.790 ⇒ 00:06:03.370 Robert Tseng: Yeah.
46 00:06:03.950 ⇒ 00:06:20.509 Robert Tseng: Yeah, I think the… but… so that’s… that’s just on the November 18th launch. I think, and we kind of were… we’re trying to try to make sense of, like, startup business upgrades before and afterwards, like, I think that’s kind of noisy, maybe we just need to track it over time a bit more.
47 00:06:20.800 ⇒ 00:06:27.750 Phoebe Miller: So, just to be clear, yeah, just to be clear on this initial dash, initial report, the top one.
48 00:06:27.750 ⇒ 00:06:29.339 Robert Tseng: Yes.
49 00:06:30.120 ⇒ 00:06:40.410 Phoebe Miller: And I think probably, Robert, I’m gonna do this on every report, just because I have to get into meetings and, like, justify these. So, can we open the, filters on this?
50 00:06:40.950 ⇒ 00:06:41.480 Robert Tseng: Yep.
51 00:06:42.860 ⇒ 00:06:52.400 Phoebe Miller: Okay, so… This is… Anyone who has completed onboarding.
52 00:06:53.750 ⇒ 00:06:57.669 Robert Tseng: Some of this didn’t save. Oh, no, that’s not what I wanted to do.
53 00:06:58.710 ⇒ 00:07:05.380 Robert Tseng: Yeah, this should really be 30 days, and then this should really just be active.
54 00:07:06.630 ⇒ 00:07:12.250 Robert Tseng: Oh, no, we had to… we have a limit on the cohort, so I took that off. Okay, whatever.
55 00:07:12.540 ⇒ 00:07:13.880 Robert Tseng: Yeah.
56 00:07:15.460 ⇒ 00:07:21.379 Phoebe Miller: So maybe you can just walk me through the, how the report is functioning, and then I can ask questions.
57 00:07:21.670 ⇒ 00:07:37.139 Robert Tseng: Yeah, so, I mean, this is just, like… it’s… it’s all users, and then, like, once they signed up to, like, when their subscription is a success, I mean, in the previous reports, we did this, kind of.
58 00:07:37.160 ⇒ 00:07:42.099 Robert Tseng: funnel step. We were doing completions within 30 days, but since we’re looking at, like, a…
59 00:07:42.160 ⇒ 00:07:51.680 Robert Tseng: less than 30-day window, it’s 2 weeks before, 2 weeks after, like, this is probably not the best way to look at it. So we had it set to 1, but maybe, like, maybe 7 days is probably a bit.
60 00:07:51.680 ⇒ 00:07:59.800 Phoebe Miller: 14… let’s do 4… let’s do 14, because we’re about to reintroduce the 14-day trial, and so there’s, like, a hypothesis that 14 days is the correct conversion window. Okay.
61 00:07:59.800 ⇒ 00:08:03.940 Robert Tseng: Okay, so yeah, that does change things a little bit, but yeah. So this… so…
62 00:08:03.940 ⇒ 00:08:09.030 Phoebe Miller: This is saying that people who… Completed onboarding.
63 00:08:09.830 ⇒ 00:08:14.530 Phoebe Miller: And then… Converted within that same 14-day period?
64 00:08:15.850 ⇒ 00:08:20.640 Robert Tseng: It’s not even complete onboarding, it’s, like, once they’ve signed up. So if we wanted to do onboarding, that would.
65 00:08:20.640 ⇒ 00:08:28.439 Phoebe Miller: No, no, no, that’s fine, that’s fine, that’s fine, it’s the same thing, so it’s alright. I just saw, I saw onboarding, okay, I see. Onboarding user sign-up success. Yep.
66 00:08:28.440 ⇒ 00:08:33.250 Alicia Shin: Is there any world where that first one, the sign-up success, would not be 100%?
67 00:08:35.720 ⇒ 00:08:41.290 Alicia Shin: Like, are you only looking at people who successfully sign up, and therefore do we only care about the right-hand side?
68 00:08:41.610 ⇒ 00:08:55.409 Robert Tseng: Yeah, yeah, you only care about the right-hand side. This is… this is always, you know, I mean, the number, like, the actual number is different, obviously, so it’s like, yeah, like, it’s out of… the denominator is 1400 versus 1900, but yeah.
69 00:08:56.090 ⇒ 00:08:56.860 Alicia Shin: Got it.
70 00:08:59.360 ⇒ 00:09:02.620 Phoebe Miller: And… and so you feel confident that…
71 00:09:06.180 ⇒ 00:09:13.040 Phoebe Miller: More… a percent… there’s a percent increase… In conversions, post…
72 00:09:13.870 ⇒ 00:09:17.670 Phoebe Miller: And this is on post-manage plan, what’s the date there?
73 00:09:17.830 ⇒ 00:09:19.140 Alicia Shin: November 18th.
74 00:09:19.140 ⇒ 00:09:24.010 Phoebe Miller: Yeah. Hmm.
75 00:09:25.000 ⇒ 00:09:30.849 Phoebe Miller: I mean, if that’s true, that’s great. I… I ha- I guess,
76 00:09:32.000 ⇒ 00:09:37.710 Phoebe Miller: Can we… can we double-click into the subscription success… People?
77 00:09:38.040 ⇒ 00:09:38.690 Robert Tseng: Yep.
78 00:09:38.690 ⇒ 00:09:39.260 Phoebe Miller: Okay.
79 00:09:44.700 ⇒ 00:09:47.370 Alicia Shin: How is subscription success defined?
80 00:09:52.640 ⇒ 00:09:58.489 Robert Tseng: That is the payment event. Oh, sorry, did I click on the right one? This is… no, this is the right one, yeah.
81 00:09:59.730 ⇒ 00:10:06.699 Alicia Shin: As in, include… it includes people who select free, or it filters out people who select.
82 00:10:06.700 ⇒ 00:10:10.290 Robert Tseng: No, this is, like, they’ve already paid, yeah. They’ve paid, okay, cool.
83 00:10:11.490 ⇒ 00:10:17.820 Phoebe Miller: And then, are we able to see… oh, okay, they’re on the left-hand side, okay. So we want to probably exclude…
84 00:10:18.590 ⇒ 00:10:19.790 Alicia Shin: Read me I.O.
85 00:10:19.790 ⇒ 00:10:21.210 Phoebe Miller: Males, yeah.
86 00:10:22.620 ⇒ 00:10:24.440 Phoebe Miller: Because there’s 5 of those.
87 00:10:26.560 ⇒ 00:10:32.589 Robert Tseng: Yeah, yeah, no, I… I… we had a…
88 00:10:32.930 ⇒ 00:10:36.820 Robert Tseng: cohort before that was automatically saluting them, but I… I guess…
89 00:10:37.200 ⇒ 00:10:40.929 Robert Tseng: Yeah, whatever, I guess we should have just added it to this report, but we didn’t.
90 00:10:41.350 ⇒ 00:10:41.930 Phoebe Miller: Okay.
91 00:10:41.930 ⇒ 00:10:43.760 Robert Tseng: I could quickly do that.
92 00:10:44.180 ⇒ 00:10:49.959 Phoebe Miller: Okay. And then I just want to make sure…
93 00:10:50.160 ⇒ 00:10:56.480 Phoebe Miller: Maybe what we should do, Alicia, is, like, for every report that we plan to bring into
94 00:10:56.650 ⇒ 00:11:12.849 Phoebe Miller: at least while we’re just making sure that things are true and correct, we should just run the actual users by… like, we should… because the volume’s low, especially for, like, subscription success, we should run the subscription success users by Stripe and just do, like, a double check.
95 00:11:12.980 ⇒ 00:11:24.219 Phoebe Miller: But I think… we don’t have to do it on the sign-up. Like, the sign-up, I now feel pretty confident, because Mark verified, but since subscription success, I don’t think we’ve done enough verification.
96 00:11:24.400 ⇒ 00:11:27.930 Alicia Shin: Yeah, that’s like a big QA until we feel good about this. That’s fine.
97 00:11:30.360 ⇒ 00:11:33.650 Alicia Shin: Do we want enterprise users?
98 00:11:33.650 ⇒ 00:11:36.529 Phoebe Miller: No, see, that’s the other thing, it’s like JFrog’s in there.
99 00:11:36.530 ⇒ 00:11:38.000 Alicia Shin: Lightcast.
100 00:11:40.150 ⇒ 00:11:45.479 Phoebe Miller: I guess, did JFrog, do you know if they were on a self-serve?
101 00:11:45.480 ⇒ 00:11:47.229 Alicia Shin: I don’t think they were, but I can double check.
102 00:11:47.230 ⇒ 00:11:47.880 Phoebe Miller: Okay.
103 00:11:48.410 ⇒ 00:11:50.859 Phoebe Miller: Because, like, if they were… if they did…
104 00:11:50.860 ⇒ 00:11:51.580 Alicia Shin: Of course, yeah.
105 00:11:51.580 ⇒ 00:11:56.620 Phoebe Miller: Yeah, but… but they signed before November 18th, I think, so that wouldn’t make sense anyway.
106 00:11:56.620 ⇒ 00:11:57.979 Alicia Shin: Okay, let me go look up right now.
107 00:11:58.200 ⇒ 00:12:01.409 Alicia Shin: I don’t… I don’t even see them in Stripe, so I don’t think so.
108 00:12:02.160 ⇒ 00:12:04.769 Phoebe Miller: So how would that be happening, Robert?
109 00:12:05.420 ⇒ 00:12:05.920 Alicia Shin: It’s…
110 00:12:09.720 ⇒ 00:12:11.579 Robert Tseng: I just filtered out the Enterprise.
111 00:12:12.890 ⇒ 00:12:15.759 Phoebe Miller: But do Enterprise have subscription success events?
112 00:12:16.680 ⇒ 00:12:18.180 Robert Tseng: No, I don’t think it changed the number.
113 00:12:20.830 ⇒ 00:12:25.559 Robert Tseng: Like, this 48 was still there, so even after I excluded it, so I don’t really think it…
114 00:12:26.160 ⇒ 00:12:26.750 Alicia Shin: Oh, weird.
115 00:12:26.750 ⇒ 00:12:31.319 Robert Tseng: Like, you may have seen… you may be… they showed up because they were probably captured at…
116 00:12:32.020 ⇒ 00:12:33.460 Phoebe Miller: As sign-ups?
117 00:12:33.830 ⇒ 00:12:36.570 Robert Tseng: But it’s the… even in the…
118 00:12:37.590 ⇒ 00:12:47.190 Robert Tseng: even in this, like, this is just, like, a random selection of 20 users that’s caught in, like, the full, list. They may not all have paperwork, but yeah.
119 00:12:47.500 ⇒ 00:12:52.479 Phoebe Miller: Oh, okay, is there a way for us to export the sign-up successes only?
120 00:12:53.000 ⇒ 00:12:53.570 Phoebe Miller: Sorry, B.
121 00:12:54.180 ⇒ 00:12:55.750 Phoebe Miller: The payment successes only?
122 00:12:55.750 ⇒ 00:12:56.950 Robert Tseng: Yeah…
123 00:12:57.190 ⇒ 00:13:03.390 Robert Tseng: I mean, I don’t know why this is not bringing me to that directly. This is only kind of highlighting this sign-up success.
124 00:13:05.800 ⇒ 00:13:07.770 Robert Tseng: Oh, it’s because I’m not clicking…
125 00:13:08.380 ⇒ 00:13:09.110 Phoebe Miller: That a little.
126 00:13:09.110 ⇒ 00:13:10.709 Robert Tseng: Yeah, yeah, that’s.
127 00:13:10.710 ⇒ 00:13:12.060 Phoebe Miller: Okay, there we go.
128 00:13:12.750 ⇒ 00:13:17.650 Phoebe Miller: Okay, so still, same note on the README stuff… And that’s right.
129 00:13:17.650 ⇒ 00:13:20.230 Robert Tseng: I already excluded them, but okay.
130 00:13:20.690 ⇒ 00:13:31.579 Phoebe Miller: And then… I’d be curious about, like, there’s rainforest times 4 or 5 here?
131 00:13:31.990 ⇒ 00:13:34.179 Alicia Shin: That feels like a service account or something.
132 00:13:36.520 ⇒ 00:13:37.540 Alicia Shin: Oh.
133 00:13:38.600 ⇒ 00:13:39.949 Phoebe Miller: They have multiple subscriptions?
134 00:13:40.740 ⇒ 00:13:41.910 Alicia Shin: Let me look.
135 00:13:41.910 ⇒ 00:13:42.650 Phoebe Miller: Okay.
136 00:13:50.570 ⇒ 00:13:51.980 Alicia Shin: Okay.
137 00:13:52.220 ⇒ 00:13:53.399 Phoebe Miller: 1, 2.
138 00:13:53.400 ⇒ 00:13:55.970 Alicia Shin: An active startup plan.
139 00:13:55.990 ⇒ 00:13:57.370 Phoebe Miller: 8, 9.
140 00:13:58.350 ⇒ 00:14:00.900 Alicia Shin: 9 of them are rainforest.
141 00:14:03.020 ⇒ 00:14:08.329 Alicia Shin: They have only one active startup plan.
142 00:14:09.580 ⇒ 00:14:12.809 Alicia Shin: Are these, like, testing accounts or something that, like, they’re…
143 00:14:12.810 ⇒ 00:14:17.840 Phoebe Miller: Oh, like, our rainforest? Like, our… because rainforest is a tool that we use.
144 00:14:17.840 ⇒ 00:14:24.180 Alicia Shin: Right, like, I get a ton of emails right now, but I don’t know what they mean. Are they doing something in our systems to, like, test?
145 00:14:25.320 ⇒ 00:14:29.349 Phoebe Miller: It might be running… running automated tasks.
146 00:14:29.630 ⇒ 00:14:33.910 Alicia Shin: Okay, then we might want to exclude those as well, if those… because I’m only.
147 00:14:33.910 ⇒ 00:14:40.069 Phoebe Miller: I don’t… I don’t know, though. I mean, it’s like a… it’s our QA tool, but I don’t know.
148 00:14:40.070 ⇒ 00:14:43.000 Alicia Shin: So I only see one active…
149 00:14:43.280 ⇒ 00:14:49.609 Alicia Shin: subscription that’s been active for a while, and I don’t see any recent signups that… or recent subscriptions that
150 00:14:49.810 ⇒ 00:14:52.949 Alicia Shin: would be… That should be included here.
151 00:14:52.950 ⇒ 00:14:57.130 Phoebe Miller: Yeah, so it’s probably… it’s probably just our QA running through the sign-up flow.
152 00:14:57.130 ⇒ 00:14:58.520 Alicia Shin: So, yeah, I think so.
153 00:14:59.420 ⇒ 00:15:06.030 Alicia Shin: Okay, so we’ll… this is, like, a good… this is great for us to be able to, like, QA this and then know what to filter out, so this is good.
154 00:15:07.620 ⇒ 00:15:08.330 Robert Tseng: Yeah.
155 00:15:10.740 ⇒ 00:15:12.030 Robert Tseng: Why are they still in there?
156 00:15:13.260 ⇒ 00:15:16.209 Robert Tseng: Zmail is not equivalent.
157 00:15:19.910 ⇒ 00:15:21.370 Robert Tseng: I already did this look.
158 00:15:21.720 ⇒ 00:15:25.800 Robert Tseng: No worries. I need to, like, refine this later, yeah. Yeah, yeah, I can refine it later.
159 00:15:28.370 ⇒ 00:15:38.210 Phoebe Miller: Okay, well, so that would… once we get that data, Correct. That would…
160 00:15:38.530 ⇒ 00:15:46.090 Phoebe Miller: go for manage… we did percent of signups to paid conversions. Yes, we would hope to see an increase, and then…
161 00:15:46.690 ⇒ 00:15:49.339 Phoebe Miller: We would hope to see an increase in…
162 00:15:50.980 ⇒ 00:15:55.809 Phoebe Miller: what’s basic, like a free plan? Free to start up, and start up to business? Is that…
163 00:15:57.010 ⇒ 00:15:59.970 Phoebe Miller: Okay. Do we… and do we have reports on that?
164 00:16:06.140 ⇒ 00:16:13.770 Robert Tseng: Not in this one yet. I mean, we have, like, plan breakdowns and other reports, but it’s not, like, not by pricing,
165 00:16:14.070 ⇒ 00:16:15.770 Robert Tseng: Yet. So…
166 00:16:17.750 ⇒ 00:16:19.759 Phoebe Miller: Great. What are… what are these ones?
167 00:16:19.760 ⇒ 00:16:22.949 Robert Tseng: I guess, like, we have this as, like,
168 00:16:30.100 ⇒ 00:16:35.160 Robert Tseng: They were already startup before… .
169 00:16:37.050 ⇒ 00:16:49.339 Mustafa Raja: So this is, so the, so the plan here would be free, which I think would be basic, and then, they, they would shift it to, startup.
170 00:16:50.050 ⇒ 00:16:52.479 Phoebe Miller: Okay, so that is what we’re hoping to see.
171 00:16:52.910 ⇒ 00:16:53.800 Mustafa Raja: Yeah.
172 00:16:56.130 ⇒ 00:16:57.280 Phoebe Miller: And…
173 00:16:59.360 ⇒ 00:17:01.400 Alicia Shin: But what’s before and after?
174 00:17:01.920 ⇒ 00:17:02.310 Phoebe Miller: Yeah.
175 00:17:02.310 ⇒ 00:17:03.090 Alicia Shin: hurts.
176 00:17:04.040 ⇒ 00:17:10.050 Robert Tseng: Before, like, the plan change, or before the Super Hub and after Super Hub, right?
177 00:17:11.119 ⇒ 00:17:11.759 Mustafa Raja: Yep.
178 00:17:15.380 ⇒ 00:17:18.300 Phoebe Miller: Got it. Okay.
179 00:17:19.060 ⇒ 00:17:22.429 Phoebe Miller: Can we do the same thing here on QAing the data?
180 00:17:23.400 ⇒ 00:17:24.410 Robert Tseng: Yeah…
181 00:17:24.630 ⇒ 00:17:27.280 Alicia Shin: Sorry, why would we do before and after SuperHub?
182 00:17:27.510 ⇒ 00:17:34.659 Phoebe Miller: Sorry, it’s bringing Managed Plan into SuperHub, so we’re… the idea is, like, all the little upgrade triggers.
183 00:17:35.660 ⇒ 00:17:40.560 Alicia Shin: Okay. So, so before is really just before November 18th?
184 00:17:40.560 ⇒ 00:17:41.250 Phoebe Miller: That’s right.
185 00:17:41.570 ⇒ 00:17:42.190 Mustafa Raja: Yeah.
186 00:17:42.700 ⇒ 00:17:44.200 Alicia Shin: Okay, got it.
187 00:17:45.220 ⇒ 00:17:48.140 Robert Tseng: Yeah, we should update the titles here.
188 00:17:48.270 ⇒ 00:17:51.230 Robert Tseng: Making it clear, like, what… what it’s before and after.
189 00:17:51.410 ⇒ 00:17:59.880 Robert Tseng: And then, like, the same set of filters you supply across all the reports. I know that we already… we deleted the cohort, but we just have to manually put it in.
190 00:18:01.250 ⇒ 00:18:09.820 Alicia Shin: Cool. And then, is there a way to consolidate the views so we can see, like, the actual uplift, in…
191 00:18:10.010 ⇒ 00:18:17.070 Alicia Shin: before and after the 18th, similar to how you had, like, the comparison bar charts. Is there, like, a different way we can sort of consolidate?
192 00:18:17.340 ⇒ 00:18:27.359 Robert Tseng: Yeah, we can. Yeah, it would just be, like, a second, like, basically you duplicate this event, and then you can kind of, I mean, maybe I could just do it right now.
193 00:18:27.570 ⇒ 00:18:33.739 Alicia Shin: Okay, no worries. I mean, I think that will be the most useful, visual for conversations.
194 00:18:34.000 ⇒ 00:18:34.840 Robert Tseng: Yeah.
195 00:18:35.540 ⇒ 00:18:37.280 Alicia Shin: Do you agree, Phoebe? I don’t… I know this.
196 00:18:37.280 ⇒ 00:18:45.799 Phoebe Miller: Yeah, I’m just… no, no, I 100% agree. I’m just trying to figure out what the… because it’s… it’s less of, like, a funnel,
197 00:18:45.980 ⇒ 00:18:50.220 Phoebe Miller: Because in theory, it’s, like, all startup customers, right? And then…
198 00:18:50.540 ⇒ 00:18:54.840 Phoebe Miller: So, I’m just trying to figure out what the conversion is based off of.
199 00:18:54.840 ⇒ 00:18:55.230 Alicia Shin: Yeah.
200 00:18:55.230 ⇒ 00:18:56.040 Phoebe Miller: Awesome.
201 00:18:56.040 ⇒ 00:19:00.759 Alicia Shin: Maybe even just comparing, like, the past 4 weeks, or, like, the time period before.
202 00:19:00.760 ⇒ 00:19:05.809 Phoebe Miller: I would, yeah, I would do, like, just number of upgrades, right? Yeah.
203 00:19:07.380 ⇒ 00:19:09.759 Phoebe Miller: Okay, cool, this is great.
204 00:19:09.940 ⇒ 00:19:12.260 Phoebe Miller: Cause the, the blue is after.
205 00:19:13.290 ⇒ 00:19:19.549 Robert Tseng: Yeah, yeah, I mean, I could… I can add another filter to basically hide the blue line before, but yeah, you just have to…
206 00:19:20.750 ⇒ 00:19:23.040 Alicia Shin: And then why are we still seeing the…
207 00:19:23.140 ⇒ 00:19:25.360 Phoebe Miller: green line after.
208 00:19:26.240 ⇒ 00:19:28.800 Robert Tseng: I guess not everybody is on…
209 00:19:29.480 ⇒ 00:19:31.169 Alicia Shin: Oh, I guess we, like, phased rolled it out.
210 00:19:31.170 ⇒ 00:19:31.670 Robert Tseng: Yeah.
211 00:19:31.670 ⇒ 00:19:33.370 Alicia Shin: Okay, okay, that makes sense, cool.
212 00:19:33.370 ⇒ 00:19:33.960 Phoebe Miller: So I.
213 00:19:33.960 ⇒ 00:19:34.849 Robert Tseng: It’s not a hard cutover.
214 00:19:34.850 ⇒ 00:19:41.460 Phoebe Miller: Also, Alicia, there’s, like, what, how much? 10% of self-serve customers still on… Legacy.
215 00:19:42.290 ⇒ 00:19:46.089 Alicia Shin: Yeah, it’s very low, so, but that makes sense, okay.
216 00:19:47.600 ⇒ 00:19:48.910 Alicia Shin: Cool, okay.
217 00:19:49.180 ⇒ 00:20:00.220 Phoebe Miller: Yeah, can we dig into… Start, okay, manage plan change, plan change startup… is SuperHub. Wait, is SuperHub, though.
218 00:20:03.370 ⇒ 00:20:06.270 Phoebe Miller: How’s the blue pulling through if it’s in SuperHope?
219 00:20:07.280 ⇒ 00:20:10.009 Phoebe Miller: Sorry, how’s the green pulling through if it is SuperHub?
220 00:20:12.660 ⇒ 00:20:15.169 Mustafa Raja: It’s with the SuperHub false filter.
221 00:20:15.370 ⇒ 00:20:16.400 Mustafa Raja: Oh…
222 00:20:16.400 ⇒ 00:20:21.939 Robert Tseng: It… I mean, maybe it was, like, turned on for internal users first, we didn’t exclude the README, so…
223 00:20:29.530 ⇒ 00:20:38.570 Phoebe Miller: I guess what I’m asking is if the green is people on the previous experience, but the filter is…
224 00:20:39.050 ⇒ 00:20:43.310 Phoebe Miller: is SuperHub? How is it even on the report?
225 00:20:47.030 ⇒ 00:20:56.980 Phoebe Miller: Oh, it’s a second. I see it’s a second. Okay. Yeah. Got it. It’s false. You wanted to see them both in the same… Yeah, yeah, yeah, yeah, got it, got it, got it. Okay.
226 00:20:59.530 ⇒ 00:21:00.340 Phoebe Miller: Cool.
227 00:21:00.580 ⇒ 00:21:03.610 Phoebe Miller: Startup, true, free, free launch.
228 00:21:04.630 ⇒ 00:21:13.110 Phoebe Miller: Okay, sweet. Can we do some… because, like, again, if this is true, that huge uptick the week of November 23rd is incredible.
229 00:21:13.510 ⇒ 00:21:16.399 Phoebe Miller: Could we do some QA?
230 00:21:18.050 ⇒ 00:21:18.690 Phoebe Miller: On the…
231 00:21:18.690 ⇒ 00:21:20.090 Robert Tseng: Yeah, let’s just take a look.
232 00:21:20.090 ⇒ 00:21:20.680 Phoebe Miller: Yeah.
233 00:21:23.000 ⇒ 00:21:25.600 Alicia Shin: And this is… the filters are not applied yet, right?
234 00:21:25.600 ⇒ 00:21:26.630 Phoebe Miller: Brain farms to read.
235 00:21:26.630 ⇒ 00:21:32.719 Robert Tseng: Yeah, the Rainforest… we should just exclude Rainforest and README from everything. Yes. Yeah.
236 00:21:33.400 ⇒ 00:21:35.059 Alicia Shin: Oof, that’s most of it.
237 00:21:36.880 ⇒ 00:21:38.450 Phoebe Miller: Right, so it was all our testing.
238 00:21:43.930 ⇒ 00:21:44.790 Phoebe Miller: Manage people.
239 00:21:45.080 ⇒ 00:21:45.820 Phoebe Miller: Jane.
240 00:21:47.400 ⇒ 00:21:55.720 Alicia Shin: Once the filters are applied, if you just let me know the best way for me to pull these down, I’m happy to just review it after, and just make sure we feel good about it.
241 00:21:56.170 ⇒ 00:21:56.720 Robert Tseng: Yeah.
242 00:21:57.960 ⇒ 00:22:09.220 Alicia Shin: Like, can we also add, I see above where we can do, like, a description of, like, the question that this chart answers. I think that would be really helpful for us. Do you mind adding that in as well, so we…
243 00:22:09.920 ⇒ 00:22:13.900 Alicia Shin: Don’t forget, as we’re looking at these.
244 00:22:14.350 ⇒ 00:22:14.940 Robert Tseng: Yeah.
245 00:22:15.280 ⇒ 00:22:16.340 Alicia Shin: Cool, thanks.
246 00:22:16.900 ⇒ 00:22:24.389 Phoebe Miller: Okay, so we have percent of signups to paid conversions, that’s across all of the recently launched projects. Then we’ll have…
247 00:22:24.530 ⇒ 00:22:28.309 Phoebe Miller: number of upgrades.
248 00:22:28.570 ⇒ 00:22:35.289 Phoebe Miller: startup, or free to start up, and then, I guess, just apply the same, filters for the startup to business.
249 00:22:35.680 ⇒ 00:22:36.210 Robert Tseng: Yeah.
250 00:22:36.210 ⇒ 00:22:40.839 Phoebe Miller: Okay, cool. Let’s move on to AI.
251 00:22:46.410 ⇒ 00:22:50.670 Robert Tseng: I mean, this is kind of more something we’ve already… .
252 00:22:51.210 ⇒ 00:22:55.810 Phoebe Miller: Oh, I’m, I’m particular, specifically, like, the number of AI boosters added.
253 00:22:58.370 ⇒ 00:23:01.720 Robert Tseng: Yeah, I… I don’t think we’ve added…
254 00:23:02.040 ⇒ 00:23:04.340 Robert Tseng: I don’t think we’ve added that yet.
255 00:23:04.960 ⇒ 00:23:10.890 Alicia Shin: So, if I recall, you guys had asked us to send over, like, a booster checkbox or something?
256 00:23:10.890 ⇒ 00:23:11.909 Robert Tseng: Is that… Yeah.
257 00:23:12.580 ⇒ 00:23:22.040 Alicia Shin: I don’t know… I, I, yeah, I’m not sure if that’s possible, but… Because if they have…
258 00:23:23.320 ⇒ 00:23:24.810 Alicia Shin: Okay, hold on, let me think.
259 00:23:26.180 ⇒ 00:23:29.840 Alicia Shin: Wouldn’t that just be, like, another plan that’s listed, though?
260 00:23:30.190 ⇒ 00:23:33.200 Alicia Shin: like, AI Booster as, like, a add-on play.
261 00:23:33.200 ⇒ 00:23:34.890 Phoebe Miller: A strike plan, yeah.
262 00:23:38.940 ⇒ 00:23:39.580 Alicia Shin: So…
263 00:23:46.910 ⇒ 00:23:47.530 Phoebe Miller: Hmm.
264 00:23:53.460 ⇒ 00:23:58.980 Robert Tseng: I can’t edit your… I can’t edit, your cohort, Mustafa.
265 00:24:00.050 ⇒ 00:24:00.810 Mustafa Raja: I can…
266 00:24:04.980 ⇒ 00:24:10.109 Robert Tseng: I mean, this is the one that replaced the old cohort, right? Because we were trying to keep it under 5, so…
267 00:24:10.440 ⇒ 00:24:17.280 Robert Tseng: this one, like, there’s… it’s not a… it’s not a plant type. We can… we can basically just…
268 00:24:17.730 ⇒ 00:24:23.900 Robert Tseng: We’re counting how many times this event fired per user, and so it’s… it’s like a proxy,
269 00:24:24.670 ⇒ 00:24:27.350 Robert Tseng: Or, like, it’s not a true feature flag, it’s like…
270 00:24:28.000 ⇒ 00:24:47.730 Robert Tseng: we just, like, created… I mean, which is… which is fine, like, but it’s… this is not a real tag on a user. This is, like, looking at users, how many of them performed this event, like, if they did it more than one time, then consider them someone who had used AI Booster. So that’s… that was our workaround.
271 00:24:47.740 ⇒ 00:24:53.940 Robert Tseng: But we don’t act… there is not actually a plan, or there’s not actually a type to select on a user.
272 00:24:54.090 ⇒ 00:24:55.540 Robert Tseng: Alright, yeah, yeah.
273 00:24:55.540 ⇒ 00:24:59.289 Phoebe Miller: Let’s just add on our side, at least we can just ask Davin to…
274 00:24:59.560 ⇒ 00:25:14.940 Phoebe Miller: send over an event. Because it’s like… or maybe we just have to create, like, a back-end checkbox or something, but… because it’s a… it’s an upgrade added in Stripe, Robert, so I guess that… since you all don’t have Stripe events, right, you just have…
275 00:25:15.740 ⇒ 00:25:24.089 Phoebe Miller: Mongo, like, payment success, which comes from Stripe, but I don’t think that they have the equivalent for, expansion.
276 00:25:24.250 ⇒ 00:25:30.160 Phoebe Miller: But we should be able to add an event. That’s, like, super crucial to us. So if you go to slide 3 in the deck.
277 00:25:30.440 ⇒ 00:25:36.979 Phoebe Miller: Sorry, I’m sure I wasn’t clear, but, like, these are the three things that we want to look at every single week.
278 00:25:37.210 ⇒ 00:25:43.500 Phoebe Miller: Because we have this, our, our, our…
279 00:25:43.590 ⇒ 00:26:03.469 Phoebe Miller: assumption for the business growing is that people want AI, and so we need to know how many people are adding the AI booster to, justify, like, that business decision. So that’s, like, super important. And then we just saw conversion by plan type and upgrade by plan type, so I feel good about 2 and 3.
280 00:26:03.610 ⇒ 00:26:08.470 Phoebe Miller: But one, we need to get in place, like, very, very quickly.
281 00:26:09.020 ⇒ 00:26:11.770 Phoebe Miller: Okay, so on… can you go back to 4 now?
282 00:26:13.130 ⇒ 00:26:14.859 Phoebe Miller: Okay, so we have…
283 00:26:17.150 ⇒ 00:26:22.070 Phoebe Miller: Oh, do we have percent onboarding complete? That’s an old one, but do we just have somewhere where that lives?
284 00:26:23.430 ⇒ 00:26:25.989 Robert Tseng: Yeah, that’s in one of the old,
285 00:26:26.480 ⇒ 00:26:36.550 Robert Tseng: reports… I mean, we want to bring it into the same format that we’re kind of showing you on the payment conversion side, so it’s just not done yet. It’s probably sitting in sign-up funnel or something.
286 00:26:36.790 ⇒ 00:26:42.699 Robert Tseng: Okay. Yeah, like, there… we had some of this old… this is Ashley’s old one.
287 00:26:51.210 ⇒ 00:26:52.630 Phoebe Miller: Okay, so what I would…
288 00:26:52.630 ⇒ 00:26:53.230 Robert Tseng: Yeah.
289 00:26:53.230 ⇒ 00:27:01.389 Phoebe Miller: What I would ask, then, is a singular dashboard, once we’ve done the work to verify the reports, a singular dashboard that has…
290 00:27:01.500 ⇒ 00:27:13.979 Phoebe Miller: Really only 4 reports, and that would be… or maybe it’s 5… the sign-up funnel to conversion.
291 00:27:14.120 ⇒ 00:27:17.459 Phoebe Miller: So, that would be number of signups.
292 00:27:17.650 ⇒ 00:27:22.520 Phoebe Miller: Number of onboarding completes, and number of payment successes.
293 00:27:22.790 ⇒ 00:27:33.040 Phoebe Miller: And then the next would be the number of upgrades from startup to bus… or from free to startup, and then another one of the number of upgrades from startup to business.
294 00:27:33.240 ⇒ 00:27:41.809 Phoebe Miller: And then the final one would be number of AI boosters added. So it’s… it’s… Yeah, go ahead.
295 00:27:41.810 ⇒ 00:27:46.130 Alicia Shin: I’ll message, Davin about that. Cool. Monday.
296 00:27:46.840 ⇒ 00:27:50.500 Phoebe Miller: Does that all make sense to you all on your side?
297 00:27:51.820 ⇒ 00:27:53.560 Robert Tseng: Yeah, you want that for every experiment.
298 00:27:54.620 ⇒ 00:27:56.630 Phoebe Miller: Well, I, I just,
299 00:27:56.920 ⇒ 00:28:08.119 Phoebe Miller: I don’t know, unless you have a suggestion, like, I’m… is there an additional data point that we should be looking at at this point? Like, it just feels like we’re boiling the ocean, and those are really the only the four things that I care about.
300 00:28:08.950 ⇒ 00:28:09.390 Alicia Shin: Do we care about…
301 00:28:09.390 ⇒ 00:28:10.459 Robert Tseng: Yeah, no, that makes sense.
302 00:28:10.460 ⇒ 00:28:12.410 Alicia Shin: Do we care about downgrades at all, Phoebe?
303 00:28:12.970 ⇒ 00:28:16.389 Phoebe Miller: Potentially, but not for this.
304 00:28:17.270 ⇒ 00:28:21.619 Phoebe Miller: I mean, yes, obviously, but… but not for growth.
305 00:28:21.620 ⇒ 00:28:23.510 Alicia Shin: The opposite of growth.
306 00:28:24.890 ⇒ 00:28:35.019 Phoebe Miller: Okay, can we take a quick look at 7, slide 7? Because I want to make sure we have tracking in place for all the things that are coming. Okay, percent of signups converted, same thing.
307 00:28:35.230 ⇒ 00:28:41.059 Phoebe Miller: number of days to launch. Is that something that you all have in Amplitude? Is that easy?
308 00:28:44.410 ⇒ 00:28:49.549 Robert Tseng: Yeah, we can. We can… you can count the number of days before the… before another event fires.
309 00:28:50.500 ⇒ 00:28:55.779 Phoebe Miller: Cool. So between, sign up and payments, etc.
310 00:28:55.780 ⇒ 00:28:56.800 Robert Tseng: Between two events, yeah.
311 00:28:56.800 ⇒ 00:29:02.190 Phoebe Miller: Cool. Okay, so we need… and we have percent of signups converted. So…
312 00:29:02.360 ⇒ 00:29:08.489 Phoebe Miller: This is gonna go… this is, Alicia confirmed, gonna go live tomorrow, so we just need to have that in place pre and post.
313 00:29:08.910 ⇒ 00:29:18.879 Phoebe Miller: Before tomorrow. And then… number of boosters added, same thing, number of boosters added, and then we need to get DevDash in. Okay.
314 00:29:20.220 ⇒ 00:29:30.200 Phoebe Miller: I think, in general, I just want to oversimplify all of the work we’re doing. Like, I don’t even want to look at anything that doesn’t fit into one of those five
315 00:29:30.580 ⇒ 00:29:31.650 Phoebe Miller: Reports.
316 00:29:32.080 ⇒ 00:29:35.149 Phoebe Miller: The five reports being percent of signups converted.
317 00:29:35.190 ⇒ 00:29:52.960 Phoebe Miller: percent of signups completing onboarding, upgrades from free to startup, upgrades from startup to business, and number of boosters added, and then the… I guess the sixth one now is, like, number of days between events, would be, like, the only things I want to look at. So can we clean up the space so that that’s the case?
318 00:29:53.660 ⇒ 00:29:54.250 Robert Tseng: Yeah.
319 00:29:54.530 ⇒ 00:29:55.980 Phoebe Miller: Okay, great.
320 00:29:56.190 ⇒ 00:30:05.149 Phoebe Miller: And just so you all know, like, we’re getting a ton of pressure now for this, like, so much so that…
321 00:30:05.210 ⇒ 00:30:23.750 Phoebe Miller: we’re having to, like, really justify any single person that’s working on it, and, like, how long we’ve been working on it, how we’re working on it. And I know we’ve been the blocker on a lot of stuff, but we have resources now, so anything that you all need, like, we need to sprint towards something here.
322 00:30:23.920 ⇒ 00:30:30.469 Phoebe Miller: So we’re happy to do that. If there are things that are unclear, like, let us know. We’re around. This is my number one priority.
323 00:30:31.620 ⇒ 00:30:35.609 Robert Tseng: Okay, great. Yeah, I mean, I think the AI booster,
324 00:30:35.980 ⇒ 00:30:41.800 Robert Tseng: Slag is probably… like, I mean, I showed you how we’re doing it right now, it seems like it’s not gonna be good enough, so…
325 00:30:41.920 ⇒ 00:30:43.779 Robert Tseng: We probably would need that.
326 00:30:43.930 ⇒ 00:30:49.419 Robert Tseng: But, we can… yeah, the other… the other stuff, we can just… we can just clean up the reporting there.
327 00:30:49.790 ⇒ 00:30:58.010 Phoebe Miller: Okay, and Robert, if there’s another way you can think through a proxy for… because it’s like, you have all of the data on people using the AI features.
328 00:30:58.010 ⇒ 00:30:58.350 Robert Tseng: Yeah.
329 00:30:58.350 ⇒ 00:31:04.170 Phoebe Miller: Right? So, that might be a proxy for that.
330 00:31:04.170 ⇒ 00:31:07.839 Robert Tseng: We have a cohort of, like, people who are using AI features, so if that’s, like.
331 00:31:08.170 ⇒ 00:31:13.040 Robert Tseng: sufficient. We can… we can use that instead. Instead of just, like, whether or not they toggled it by
332 00:31:13.540 ⇒ 00:31:17.279 Robert Tseng: You know, firing that talk… that event from the pricing page.
333 00:31:17.770 ⇒ 00:31:27.969 Alicia Shin: Actually, Robert, our engineer is following up right now. She’s asking if there’s more specifics on when to trigger that event, or where to add that metadata. Do you have a perspective on that?
334 00:31:28.650 ⇒ 00:31:37.440 Robert Tseng: Yeah, well, so I think they actually are two different things, right? One is, like, I guess what Mustafa had shown in the report was, like, when they literally hit that, like.
335 00:31:39.230 ⇒ 00:31:50.509 Robert Tseng: toggle, like, on the pricing page. They haven’t actually used any AI features yet, they’ve only just hit the toggle. That’s a different milestone to me than, like, when they actually used an AI feature. So, I guess, like, which one do you care about?
336 00:31:50.510 ⇒ 00:31:55.259 Alicia Shin: Well, we’re talking about the actual subscription and purchasing of a $150.
337 00:31:55.260 ⇒ 00:31:58.699 Robert Tseng: Okay, sure, then in that case, this should be the pricing, yeah.
338 00:31:58.700 ⇒ 00:31:59.390 Alicia Shin: Okay.
339 00:31:59.620 ⇒ 00:32:00.050 Robert Tseng: Yeah.
340 00:32:00.050 ⇒ 00:32:06.480 Alicia Shin: the AI booster is paid for.
341 00:32:08.010 ⇒ 00:32:10.569 Alicia Shin: And then where to add that metadata.
342 00:32:11.980 ⇒ 00:32:19.180 Robert Tseng: I think it can just be… I mean, it should be just… .
343 00:32:19.810 ⇒ 00:32:22.939 Mustafa Raja: Maybe in the AI usage, events?
344 00:32:23.710 ⇒ 00:32:28.420 Robert Tseng: That, or another identify event, so it just adds on as, like, a user property.
345 00:32:28.880 ⇒ 00:32:30.370 Alicia Shin: Identify event.
346 00:32:30.730 ⇒ 00:32:49.009 Robert Tseng: Yeah, so, like, when you, like, when a customer signs up, it fires an identify event, it adds their email, so anytime you add, like, a user property that, like, just kind of persists across all the events, like, you just want to know whether or not they, like, toggled it. So, once it fired, they can basically just add it to…
347 00:32:49.010 ⇒ 00:32:50.170 Alicia Shin: The user.
348 00:32:50.740 ⇒ 00:32:51.310 Alicia Shin: Bye.
349 00:32:51.910 ⇒ 00:32:56.749 Alicia Shin: Hopefully this can get done. She’s traveling right now, which is why I think it will take until next week, but,
350 00:32:56.750 ⇒ 00:32:57.150 Robert Tseng: Okay.
351 00:32:57.150 ⇒ 00:32:58.509 Alicia Shin: Like, she’s responsive, so…
352 00:32:59.160 ⇒ 00:33:02.309 Robert Tseng: Can we… Okay. Do you guys have a hard stop, by the way?
353 00:33:02.550 ⇒ 00:33:07.580 Robert Tseng: I… I can go a little bit more over. I just wanna take… I do have another call, yeah.
354 00:33:07.580 ⇒ 00:33:14.430 Phoebe Miller: Okay, I just want to take 2 minutes to just quickly… Yeah. I think part of what’s difficult, and maybe I’m just projecting, but it’s like…
355 00:33:14.880 ⇒ 00:33:16.150 Phoebe Miller: the…
356 00:33:16.400 ⇒ 00:33:25.569 Phoebe Miller: You all aren’t in our business, and so you don’t, like, deeply understand how the stuff is being deployed, and so then for you then to
357 00:33:25.980 ⇒ 00:33:38.869 Phoebe Miller: create the reports is difficult, which I… I acknowledge. Like, it’s… it’s difficult for myself and Alicia, and we work here, and, like, sometimes we don’t know. So, I just… maybe before we dive into report creation, we can, like.
358 00:33:39.340 ⇒ 00:33:47.340 Phoebe Miller: go through exactly what’s happening, the changes that are happening, and then you guys don’t have to do the guesswork. So, like, to be clear.
359 00:33:48.090 ⇒ 00:34:01.099 Phoebe Miller: add AI Booster, which I… I’m on our free plan currently, so you can’t do that, but if I were on startup or business, you could click this button, add AI Booster, and then you would… that would add to your Stripe plan $150 a month.
360 00:34:01.120 ⇒ 00:34:11.240 Phoebe Miller: And then you get access to all of our AI features. This is, like, a big growth lever for us, which is why we care about it. And maybe, Mustavo, this is helpful for you too, like.
361 00:34:11.469 ⇒ 00:34:29.719 Phoebe Miller: if you’re on our free plan, we’re hoping that the AI booster is a carrot to get onto our startup or business plans, because you can only add the AI booster now. Now, personally, I think that’s silly, but that’s how it currently functions, right? So it’s like, that’s why I’m curious about upgrades from free to startup and from startup to business, because
362 00:34:30.139 ⇒ 00:34:40.050 Phoebe Miller: the idea is, like, the AI booster is encouraging people to do that. And having this new, pretty managed plan in SuperHub
363 00:34:40.159 ⇒ 00:34:45.259 Phoebe Miller: README is SuperHub. Like, it’s… that’s the README 2.0.
364 00:34:45.610 ⇒ 00:34:48.470 Phoebe Miller: I don’t actually agree with this, but, like.
365 00:34:48.780 ⇒ 00:35:11.519 Phoebe Miller: it living inside SuperHub may encourage more conversions, just because there’s one less step for, like, this used to open, you know, in a separate tab. I think if people want to pay, they’re probably going to do it, but hopefully, like, this is, like, a slightly nicer experience. Does that explain all of, like, is it… are the experiments clear, I guess, is my… because I don’t want to…
366 00:35:12.240 ⇒ 00:35:15.669 Phoebe Miller: Like, hopefully we can have a very easy handoff of, like.
367 00:35:16.000 ⇒ 00:35:23.370 Phoebe Miller: end product hands off to me to Alicia, we hand off to you all, reports are created, and then we go back, and it’s just like, that’s the cycle.
368 00:35:25.160 ⇒ 00:35:27.609 Robert Tseng: Yeah, yeah, I understand the experiments.
369 00:35:27.610 ⇒ 00:35:28.240 Phoebe Miller: Okay.
370 00:35:28.620 ⇒ 00:35:35.190 Phoebe Miller: Or, I guess, do we under… do you understand the like?
371 00:35:35.760 ⇒ 00:35:46.960 Phoebe Miller: reason we’ve used these metrics to justify the experiments, or are there… is there confusion or suggestions on, like, better ways to track them?
372 00:35:48.960 ⇒ 00:36:05.790 Robert Tseng: I mean, I think what I told you last time was you need to have, like, a common metric across the experiments that you can actually know, like, which ones are moving the same needle, and it’s how you benchmark. And then… then there are secondary, tertiary, like, metrics within the experiments that are unique to that experiment.
373 00:36:05.940 ⇒ 00:36:14.030 Robert Tseng: I mean, I think we can iterate on those things. I don’t really think… I think, yeah, we’re trying to keep it simple, so I’m not, like, adding
374 00:36:14.160 ⇒ 00:36:18.810 Robert Tseng: Adding much more to it, but… Yeah, so… I guess, like.
375 00:36:18.960 ⇒ 00:36:25.220 Robert Tseng: I… I mean, I think we were looking at the same, same, same thing, so I don’t… I don’t think there’s any confusion around it.
376 00:36:25.220 ⇒ 00:36:43.020 Phoebe Miller: Yeah, as long as we’re aligned on that. And for me, it’s like, percent of signups converted is, as we talked about, the through line, and then there’s experiment-specific metrics, like the percent of upgrades, like the number of boosters added, etc.
377 00:36:43.350 ⇒ 00:36:55.559 Phoebe Miller: importantly, like, this Friday, we’re adding back the trial, and so the… again, the through line is, like, percent of signups converted, but the event… the experiment-specific metric would be, like, days to pay.
378 00:36:55.860 ⇒ 00:36:59.460 Phoebe Miller: and I… so I just want to make sure that…
379 00:36:59.610 ⇒ 00:37:05.329 Phoebe Miller: And, like, you all understand the trial being reintroduced, what that means, how that functions.
380 00:37:06.220 ⇒ 00:37:06.840 Robert Tseng: Yeah.
381 00:37:07.100 ⇒ 00:37:07.910 Phoebe Miller: Okay.
382 00:37:08.020 ⇒ 00:37:12.780 Phoebe Miller: Alrighties, well, we’ll just keep chugging along, but…
383 00:37:12.930 ⇒ 00:37:21.430 Phoebe Miller: please let us know when we should QA, when the reports are ready, and when we should QA, and then we’ll let you know when the AI booster, is added.
384 00:37:22.280 ⇒ 00:37:23.509 Robert Tseng: Okay, sounds good.
385 00:37:23.830 ⇒ 00:37:37.439 Phoebe Miller: Sweet. And if you all could, once… so, just in sequence order, once we QA the reports, then let’s get that consolidated dash of, like, the 5 or 6 metrics that we’re looking at for growth.
386 00:37:37.670 ⇒ 00:37:52.989 Phoebe Miller: And then everything else, if we could put it just, like, in an archive folder or something, I just don’t want to confuse people, because oftentimes also, like, we’ll get internal team members being like, hey, I saw this AI dash, can I look at this? I’m like, no, don’t look at it, you know?
387 00:37:53.220 ⇒ 00:37:53.790 Robert Tseng: Okay.
388 00:37:54.480 ⇒ 00:37:56.500 Phoebe Miller: Okay, sweet. Thanks all.
389 00:37:56.570 ⇒ 00:37:57.200 Alicia Shin: Thank you.
390 00:37:57.200 ⇒ 00:37:57.820 Robert Tseng: Thank you.
391 00:37:57.980 ⇒ 00:37:58.810 Robert Tseng: Bye. Bye.