Meeting Title: ReadMe <> Brainforge Check-In Date: 2025-10-16 Meeting participants: Robert Tseng, Alicia Shin, Henry Zhao
WEBVTT
1 00:00:20.210 ⇒ 00:00:21.630 Robert Tseng: Hey, Alicia.
2 00:00:21.630 ⇒ 00:00:23.030 Alicia Shin: Hi, how are you?
3 00:00:23.210 ⇒ 00:00:24.199 Robert Tseng: Good, how are you?
4 00:00:25.120 ⇒ 00:00:29.379 Alicia Shin: I’m good. We’re done with the board meeting, and I’m, like, I’m able to breathe.
5 00:00:31.370 ⇒ 00:00:35.030 Robert Tseng: Is it… you mean, it’s like a monthly thing, or… I’m assuming?
6 00:00:35.030 ⇒ 00:00:39.249 Alicia Shin: Quarterly thing, so it’s like a lot of substance in it.
7 00:00:39.250 ⇒ 00:00:40.180 Robert Tseng: Okay.
8 00:00:40.380 ⇒ 00:00:47.200 Alicia Shin: It’s just like a… you know, it’s an executive audience, it’s… a lot of prep goes into it, and then, like.
9 00:00:47.200 ⇒ 00:00:47.740 Robert Tseng: Yeah.
10 00:00:47.740 ⇒ 00:00:49.740 Alicia Shin: You say all the right words. It’s like…
11 00:00:49.740 ⇒ 00:00:50.190 Robert Tseng: Yeah.
12 00:00:50.190 ⇒ 00:00:52.070 Alicia Shin: month.
13 00:00:52.610 ⇒ 00:01:02.649 Robert Tseng: I… I had to prepare a lot of stuff at my last in-house role for board meetings that were monthly, so I can… I can understand.
14 00:01:02.650 ⇒ 00:01:04.920 Alicia Shin: Hi, Henry.
15 00:01:06.320 ⇒ 00:01:07.550 Henry Zhao: Hello, guys.
16 00:01:10.900 ⇒ 00:01:12.649 Robert Tseng: It’s just us today, right?
17 00:01:12.920 ⇒ 00:01:14.310 Alicia Shin: From our end, yeah.
18 00:01:14.310 ⇒ 00:01:15.400 Robert Tseng: Okay, cool.
19 00:01:16.110 ⇒ 00:01:19.460 Robert Tseng: Okay, well, yeah, I guess…
20 00:01:19.600 ⇒ 00:01:24.149 Robert Tseng: you know, we didn’t get a chance to connect on Tuesday, but we were kind of sharing updates,
21 00:01:24.400 ⇒ 00:01:27.800 Robert Tseng: Yeah, I guess… I can basically
22 00:01:27.900 ⇒ 00:01:43.119 Robert Tseng: do a voiceover, some of the links that I shared out as a starting point, and then we can maybe kind of go back to the original requirements doc. I know we’ve kind of taken things in different directions. I think we have a much better sense of, like.
23 00:01:43.290 ⇒ 00:01:47.270 Robert Tseng: what we can and can’t do at this point, and then I know you had a…
24 00:01:47.600 ⇒ 00:01:54.980 Robert Tseng: question about, like, well, what… why do we, like… it is basically talking through is, you know, do we… do we need to,
25 00:01:54.980 ⇒ 00:02:13.659 Robert Tseng: bump amplitude plan or not. I think Henry will also be able to chime in there. We discussed this earlier today, and he’s already gotten the chance to kind of poke around more in Mongo. So, yeah, we definitely have some thoughts on, like, kind of what direction, from a tooling or systems perspective, we could go in as well.
26 00:02:14.090 ⇒ 00:02:14.570 Robert Tseng: Okay.
27 00:02:14.880 ⇒ 00:02:20.139 Alicia Shin: That sounds great. I guess I can share, like, one other piece of context coming out of the board meeting.
28 00:02:20.140 ⇒ 00:02:21.580 Robert Tseng: Yes, please, yeah.
29 00:02:21.580 ⇒ 00:02:24.809 Alicia Shin: to this work. So,
30 00:02:25.120 ⇒ 00:02:39.190 Alicia Shin: there’s going to be, like, a really big focus on self-serve conversion this quarter. There’s actually, like, a little task force kicking off next week. Our COO is involved, and our head of product, who I mentioned, Ashley, is also involved.
31 00:02:39.190 ⇒ 00:02:39.590 Robert Tseng: Okay.
32 00:02:39.590 ⇒ 00:02:53.560 Alicia Shin: it’s work that people have been chipping away at in different forums, but I think there’s going to be a more concerted effort. So, obviously, I have not yet circulated, this work here, because I know we’re QAing and getting it to, like, a decent place, but I think… Yeah.
33 00:02:53.560 ⇒ 00:03:01.969 Alicia Shin: a fast follow, and I can share updates once we actually meet, to see, like, what… what people are thinking, but I imagine that this…
34 00:03:02.440 ⇒ 00:03:15.949 Alicia Shin: this measurement and tracking is going to be a really critical input for us, and so I’d love to get to the point where we can start pulling in other stakeholders who will be, like, the true consumers of this, and get their feedback as well, but I think we’re.
35 00:03:15.950 ⇒ 00:03:16.300 Robert Tseng: Yup.
36 00:03:16.300 ⇒ 00:03:25.510 Alicia Shin: In my mind, I was playing Phoebe, I was like, I think we’re, like, about 80% there, we just need to, like, feel really good about the underlying numbers and the foundations, and then from there, we can tweak things as people have feedback.
37 00:03:25.510 ⇒ 00:03:30.510 Robert Tseng: Yep, totally. Okay, great, great context, thank you for that.
38 00:03:30.520 ⇒ 00:03:39.690 Robert Tseng: Yeah, okay, well then that… in that case, I mean, I did add a couple, like, small things to the original notebook that I shared with you, but, I mean, I think…
39 00:03:39.690 ⇒ 00:03:53.410 Robert Tseng: the… one of the deliverables to the head of product would be the dashboard. So I think I would want to really just kind of focus on that first, get your feedback on it, what other elements we need to pull into there, and then kind of get that ready to be shared. So…
40 00:03:53.410 ⇒ 00:03:56.349 Robert Tseng: I’ll share my screen, and then see…
41 00:03:57.330 ⇒ 00:04:09.979 Robert Tseng: Okay, so here’s a dashboard. A lot of it is just the same charts that we had in the notebook, but I kind of stripped out a lot of, like, the text there. So, obviously, the main funnel, so kind of just remembering,
42 00:04:10.810 ⇒ 00:04:17.030 Robert Tseng: Yeah, yeah, these are the… the core, kind of, paid conversion steps, like the…
43 00:04:17.029 ⇒ 00:04:32.570 Robert Tseng: the paid conversion funnel steps. With the Mongo access, we’re still validating whether or not, like, these numbers are actually right, but, you can set here at the top of the dashboard, like, what you want to set the default parameters to. I’ve just done daily over 30 days.
44 00:04:32.570 ⇒ 00:04:35.990 Robert Tseng: Rather than kind of prefixing that it was going to be
45 00:04:35.990 ⇒ 00:04:47.570 Robert Tseng: September versus, like, August, whatever. So, I think this is just, like, up to user discretion to be able to change this as they want. Yeah, I think the subscription success element, or, like, kind of
46 00:04:50.350 ⇒ 00:05:05.130 Robert Tseng: events, I think, will probably be able to, you know, if… assuming that we get the transactions for Mongo, like, if there’s any engineering work that I want to make sure is right, obviously, the revenue… the revenue conversion needs to be right, so,
47 00:05:05.130 ⇒ 00:05:14.040 Robert Tseng: I think that’s one… I foresee that being one engineering piece that we would need to push into amplitude, as soon as we can.
48 00:05:14.070 ⇒ 00:05:20.269 Robert Tseng: Because this is not actually tied to Stripe, so I don’t actually believe that this is the case. Yeah, so I’ll just kind of…
49 00:05:21.220 ⇒ 00:05:34.120 Robert Tseng: couch expectations there. We do have some version of an enterprise versus free. Obviously, we have, like, non-free and free launch. Was not sure how we would consolidate those. I think we were looking into
50 00:05:34.120 ⇒ 00:05:51.100 Robert Tseng: do we even trust the plant type labeling and amplitude from initial poking around in Mongo? It seems like not really. Like, I think the enterprise numbers are very different, and I imagine the free numbers will be different, too. So, I think there’s probably some data cleanup work,
51 00:05:51.100 ⇒ 00:06:04.529 Robert Tseng: that’s possibly achievable just by using amplitude filters with no additional engineering work, but it’s not something that we would be able to finalize until, I guess, we… we get to that recon… until we finish that one on the Mongo side.
52 00:06:04.840 ⇒ 00:06:05.710 Robert Tseng: I’m sorry.
53 00:06:05.710 ⇒ 00:06:13.059 Alicia Shin: What do you mean by, data cleanup work? Our plan types are correct in Mongo, so what is the cleanup required?
54 00:06:13.060 ⇒ 00:06:15.820 Robert Tseng: Like, they’re not correct in amplitude, so…
55 00:06:15.820 ⇒ 00:06:18.170 Alicia Shin: So just making sure that we’re pulling them correctly. Okay.
56 00:06:18.170 ⇒ 00:06:27.609 Robert Tseng: Yeah. Okay. I don’t believe these are coming from Mongo, so they’re not coming from Mongo, it’s Amplitude, so that could be something that we would need to, like.
57 00:06:28.510 ⇒ 00:06:47.439 Robert Tseng: we… you know, if we’re already going to be pushing data into Amplitude for Mongo for the payment event, then I would say this is also within scope, but if we only had limited engineering resources and we could only do one thing, I would say this is the most important, and we may be able to just, like,
58 00:06:47.680 ⇒ 00:06:52.850 Robert Tseng: figure this out without, a direct Mongo integration.
59 00:06:52.850 ⇒ 00:06:56.580 Alicia Shin: So how are you… how are you pulling it right now, then, if it’s not for Mongo again?
60 00:06:56.880 ⇒ 00:07:05.780 Robert Tseng: I… however this was set up with the user profiles, I mean, I don’t really know, like, the telemetry of how it’s flowing in directly.
61 00:07:05.780 ⇒ 00:07:17.330 Robert Tseng: But, I mean, Henry, you can kind of chime in here, but we were checking enterprise numbers earlier, and this… these enterprise… enterprise user numbers are off from what we see in Mongo, so… Oh, Henry, yeah.
62 00:07:17.330 ⇒ 00:07:22.120 Henry Zhao: I haven’t confirmed that yet, so I’m looking at what Mark sent me, and I’ll just double-check that.
63 00:07:22.520 ⇒ 00:07:23.030 Robert Tseng: Okay.
64 00:07:23.030 ⇒ 00:07:38.839 Alicia Shin: We probably filter out Enterprise for what it’s worth, because this group’s gonna be solely focused on self-serve, and Enterprise, we, like, almost have a separate funnel, like, our sales team’s involved, it, like, it’s a totally different beast. So, feel free to de-scope them if that simplifies things.
65 00:07:38.840 ⇒ 00:07:49.780 Robert Tseng: Okay, yeah, I mean, generally, I did that across all the other reports, but I don’t exactly remember why, but I remember we… I don’t know, wanted to keep… we kept Enterprise on one. I can… I can remove that.
66 00:07:49.940 ⇒ 00:08:07.729 Robert Tseng: But yeah, I think I care most about, like, re… kind of actually… of accurately categorizing these none types, because they’re the majority of the users. So if they should all be starting off as free or some other label, like, I don’t know, I just… it just doesn’t sit right with me to have
67 00:08:07.850 ⇒ 00:08:10.260 Robert Tseng: None be your biggest, you know, group.
68 00:08:11.720 ⇒ 00:08:12.450 Alicia Shin: Okay.
69 00:08:12.760 ⇒ 00:08:20.430 Robert Tseng: So, this is maybe something that we can get your, kind of, take on, like, after the call, just, like, how you want to categorize these, but yeah.
70 00:08:21.230 ⇒ 00:08:33.929 Alicia Shin: Yeah, I guess, have you documented anywhere, just, like, where the data is coming from today? Because that might be something we just run by our head of product, so that she can tell us, like, definitively.
71 00:08:34.650 ⇒ 00:08:38.490 Robert Tseng: Yeah, I mean, I haven’t documented it, I could just look here, you can kind of just…
72 00:08:38.490 ⇒ 00:08:42.139 Alicia Shin: Even just, like, rough bullets, I think it’ll be good to just, like, vet.
73 00:08:43.159 ⇒ 00:08:53.760 Robert Tseng: Yeah, so… Node.js SDK, this is not Bunko, this is just, yeah, I mean, I think these are just, like, client-side,
74 00:08:55.330 ⇒ 00:09:01.700 Robert Tseng: sources. So… Yeah, like, I… I… this makes me believe this is not coming from Mako.
75 00:09:01.700 ⇒ 00:09:12.719 Alicia Shin: Yeah. Yeah. It’s like, my read of where we are right now, let me know if you agree with my summary, is, like, we have the foundation in place for the reporting, but we now need to, like, validate the data sources to make sure, like.
76 00:09:13.250 ⇒ 00:09:17.210 Alicia Shin: that is pulling in correctly. Is that fair?
77 00:09:17.840 ⇒ 00:09:33.099 Robert Tseng: Yeah. Yeah, I mean, like, the foundations are there to be able to, like, look at activity and get some directionality, but if we actually want to… especially if we’re doing anything revenue reporting-wise, we want to make sure we’re using, like, the… we want to make sure that we’re using validated data.
78 00:09:33.290 ⇒ 00:09:34.480 Alicia Shin: And I don’t know…
79 00:09:34.700 ⇒ 00:09:45.139 Alicia Shin: I mean, we’re not going to do revenue reporting out of amplitude, we have others, like, we use Stripe for that and whatnot, but I… obviously, we do want to, like, correctly capture when someone is paying. Yep.
80 00:09:45.250 ⇒ 00:09:52.460 Alicia Shin: So, hopefully that simplifies the scope, but I appreciate that there’s no integration today, so it’s probably same effort.
81 00:09:52.700 ⇒ 00:10:00.790 Robert Tseng: Yeah. Okay. So… pulling that out. Then we were talking about, basically, trial plans, so,
82 00:10:01.390 ⇒ 00:10:19.829 Robert Tseng: Yeah, I think we kind of made some versioning of, like, trial versus non-trial users. It’s not really a label, so the filtering that’s used here is just whether or not a user took action within the first 14 days, and if they were taking more actions after 14 days and still on a paid plan. It’s… yeah, it’s like a…
83 00:10:19.830 ⇒ 00:10:29.800 Robert Tseng: I… that’s kind of how we… I… yeah, anyway, I wasn’t sure if, looking at trial was still something that was important, but I just kind of put something here just in case we wanted to actually.
84 00:10:29.800 ⇒ 00:10:30.410 Alicia Shin: Yeah.
85 00:10:30.410 ⇒ 00:10:31.219 Robert Tseng: When we look at that cut.
86 00:10:31.220 ⇒ 00:10:42.410 Alicia Shin: This… this is going to be important. I don’t believe we’ve turned the trial back on, so this makes sense that you’re, like, this framework makes sense. Henry, when you’re talking to Mark, can you just ask him.
87 00:10:43.170 ⇒ 00:10:48.329 Alicia Shin: how we would identify trial users once the child’s gone again. He’ll be able to tell you explicitly.
88 00:10:48.530 ⇒ 00:10:50.280 Henry Zhao: I did want to clarify on that also.
89 00:10:51.340 ⇒ 00:10:52.120 Henry Zhao: Yeah.
90 00:10:55.620 ⇒ 00:11:00.879 Robert Tseng: I’m sure we have an offer or something. And I know FreeLaunch is tied to trial users as well, by the way, so…
91 00:11:02.670 ⇒ 00:11:07.170 Henry Zhao: Wait, but just to confirm, right now, everyone that joins is getting a 14-day free trial, or no?
92 00:11:08.250 ⇒ 00:11:10.820 Alicia Shin: I don’t think we’ve turned the trial back on right now.
93 00:11:11.130 ⇒ 00:11:14.199 Henry Zhao: So, Robert, what you told me yesterday is basically that we haven’t turned that on yet.
94 00:11:14.920 ⇒ 00:11:15.350 Alicia Shin: Is that what he.
95 00:11:15.350 ⇒ 00:11:23.309 Robert Tseng: Well, no, I was telling Henry that everybody gets 14 days before they get forced to move into a plan.
96 00:11:23.430 ⇒ 00:11:24.380 Robert Tseng: change, which.
97 00:11:24.380 ⇒ 00:11:24.930 Alicia Shin: It’s basically…
98 00:11:24.930 ⇒ 00:11:25.560 Robert Tseng: leave that.
99 00:11:26.080 ⇒ 00:11:29.529 Alicia Shin: That is the plan when we turn the trial back on.
100 00:11:29.530 ⇒ 00:11:31.190 Henry Zhao: Okay, so right now it’s not happening.
101 00:11:31.800 ⇒ 00:11:40.649 Alicia Shin: It was before November 2024, and then we got rid of it, and now we’re bringing it back. Sorry, that’s… it’s very confusing to follow what’s happening here.
102 00:11:40.650 ⇒ 00:11:44.019 Henry Zhao: It was before 9th November 2024, you said?
103 00:11:44.020 ⇒ 00:11:49.680 Alicia Shin: Yes, that’s when we, like, did a really big product launch, and then the… removed the trial.
104 00:11:51.290 ⇒ 00:11:57.479 Robert Tseng: Okay, well then, yeah, the takeaways here are definitely not accurate then, but I did do the 14-day bucket.
105 00:11:57.480 ⇒ 00:12:03.180 Alicia Shin: I like this view, so once we have it, you know, we can look at it correctly.
106 00:12:03.410 ⇒ 00:12:05.799 Robert Tseng: Okay, cool, good to know.
107 00:12:05.930 ⇒ 00:12:17.529 Robert Tseng: Yeah, and then obviously we have these, like, kind of metrics based on the plan, so basically, based on plan type, your… yeah, but what the benchmarks are for the different, funnel stages, from sign-up to project creation.
108 00:12:17.700 ⇒ 00:12:20.259 Robert Tseng: You’re safe. Anyway, so when…
109 00:12:20.270 ⇒ 00:12:38.719 Robert Tseng: We’ll clear up trial, kind of how trial and plan types kind of mix in, and then, yeah, obviously making sure that plan types are accurately labeled. So, that’s definitely a data quality thing we need to get figured out. Sign up to projects, obviously there’s a lot of daily volatility here, kind of, yeah, I think this is…
110 00:12:38.750 ⇒ 00:12:54.709 Robert Tseng: really just based off of, like, different plant types. Volume’s pretty low, other than the… than the main one, which is none. So, I mean, that’s why this chart looks kind of crazy. Yeah, kind of similar deal here. The next step, project created to launch attempt.
111 00:12:57.800 ⇒ 00:13:06.550 Robert Tseng: Yeah, and then we kind of looked at some flavor of this last… last week, how many people are going… what share of users are going from
112 00:13:06.820 ⇒ 00:13:14.520 Robert Tseng: project… created to launch attempts within the first day, within 7 days, 14 days, within 30 days. So,
113 00:13:14.700 ⇒ 00:13:15.890 Robert Tseng: Yeah, I mean…
114 00:13:16.000 ⇒ 00:13:25.320 Robert Tseng: I mean, from what… the data that I’ve looked at, I don’t think launch attempts is really a good proxy for whether or not people are really ready to convert to plan… like, a paid plan.
115 00:13:25.640 ⇒ 00:13:42.000 Robert Tseng: probably need something else, and so I think there’s some room for additional scope to, like, do some discovery on, like, what that activation event truly is. I don’t think attempted launches should be it. Like, it just doesn’t really seem like there’s anything that, definitive from there.
116 00:13:42.250 ⇒ 00:13:54.350 Alicia Shin: Okay. Yeah, other things that we’ve, like, hypothesized were around, like, the uploading of OAS files and usage of, like, different, modules that we have, like API reference and whatnot, so we could play around with that as we…
117 00:13:54.490 ⇒ 00:13:56.060 Alicia Shin: Get this tighter.
118 00:13:56.060 ⇒ 00:14:00.649 Robert Tseng: Yeah, so all of those features are… I guess it’s probably…
119 00:14:05.610 ⇒ 00:14:07.610 Robert Tseng: To the old notebook.
120 00:14:14.090 ⇒ 00:14:17.500 Robert Tseng: Yeah, so… I did kind of, like…
121 00:14:21.120 ⇒ 00:14:21.910 Robert Tseng: Dude.
122 00:14:22.910 ⇒ 00:14:41.069 Robert Tseng: Yeah, I guess we could come up with a list of what you think are the core… I mean, I kind of generally know what the core features are now, like, in terms of, like, what people are using, but yeah, so if it’s OAS upload, or, you know, saving, like, saving a guide, having a custom page, change logs, like, whatever these core features are, like, I think we can…
123 00:14:41.710 ⇒ 00:14:50.649 Robert Tseng: I mean, I could take this and, you know, blow it up into more of, like, a trend view and add all the other segments. I think this, by default, only allows me to go up to 10.
124 00:14:50.650 ⇒ 00:15:09.150 Robert Tseng: But, and you have more than 10 features. So anyway, it’s just gonna maybe take a couple more tries of, like, showing you all the different feature usages, like, all the different features, and how their usage ties to, kind of, like, revenue, and, like, we can then decide, like, you know, which one we want to use for the activation event.
125 00:15:09.160 ⇒ 00:15:25.670 Robert Tseng: I would say that it’s less about isolating a single feature. We want to be able to kind of see, like, what feature sets are users kind of, like, performing together, because it’s probably not just, like, they do one thing and then they end up, like.
126 00:15:26.260 ⇒ 00:15:44.720 Robert Tseng: converting or whatever. So I think that complexity, we still don’t understand, like, the sequence of, like, are they… which… which ones… which… which features are users going through in order, and, like, when do they actually convert to, you know, when do they actually convert to a paid plan? I think there’s some more discovery to be around
127 00:15:44.720 ⇒ 00:15:46.990 Robert Tseng: Around that, to be done around there.
128 00:15:48.200 ⇒ 00:15:48.870 Alicia Shin: Cool.
129 00:15:49.100 ⇒ 00:15:58.729 Robert Tseng: Yeah, so I kind of mentioned that as, like, a next step for how I think the feature usage-based, like, kind of
130 00:15:58.730 ⇒ 00:16:09.670 Robert Tseng: reporting can be enhanced here. Like, I think I just, yeah, pretty much just showed you what this was. Oh, wait, I did actually blow it up into a trend view, so we can kind of see some of that here.
131 00:16:09.910 ⇒ 00:16:11.640 Robert Tseng: But… yeah.
132 00:16:11.640 ⇒ 00:16:16.359 Robert Tseng: Okay, so I think those are kind of the… to summarize some of the adjustments that I want to make here.
133 00:16:16.360 ⇒ 00:16:34.399 Robert Tseng: One is, obviously, validating the data here, making sure that we’re using a better event than the subscription success, something that’s coming from Mongo directly, so I think we can hopefully work with Mark to be able to get that in. And then we want to make sure that the plan types are correctly labeled, we trust the labeling for every user.
134 00:16:34.660 ⇒ 00:16:44.040 Robert Tseng: Obviously, we should understand how the trial process will work when it turns back on. And then, I mean, as far as these, like, benchmarks for the different
135 00:16:44.760 ⇒ 00:16:51.480 Robert Tseng: funnels stages by plan type. I think this is pretty much how it’s going to be set up, so…
136 00:16:52.190 ⇒ 00:16:53.829 Robert Tseng: Yeah, we just have to…
137 00:16:54.050 ⇒ 00:17:03.659 Robert Tseng: maybe do a little bit more here on the… on the feature usage side to figure out, like, what’s a better activation feature to use than, launch attempt.
138 00:17:04.000 ⇒ 00:17:09.580 Alicia Shin: For the, subscription success, what’s the work that our engineer would need to do?
139 00:17:10.560 ⇒ 00:17:16.630 Robert Tseng: It’s… I… they might even have, like, an out-of-the-box…
140 00:17:16.810 ⇒ 00:17:23.150 Robert Tseng: Force? Let’s see… I should be pretty lightweight, like, I literally think you could just look for Bonco.
141 00:17:24.030 ⇒ 00:17:25.819 Robert Tseng: Nope, okay, maybe they don’t.
142 00:17:25.930 ⇒ 00:17:38.989 Robert Tseng: Well, I think what’s typical is, like, you know, I would use some middleware, like a segment, to pass events from, like, a production or analytics database into Amplitude.
143 00:17:39.260 ⇒ 00:17:56.350 Robert Tseng: Yeah, instead of just using, like, that SDK or whatever, which is what you’re using. But Amplitude is platformed directly on Snowflake, so this is the cleanest integration. If you do, which I don’t think you have Amazon or… I don’t think you have AWS or BigQuery, but if you did have…
144 00:17:56.350 ⇒ 00:17:57.430 Alicia Shin: AWS.
145 00:17:57.740 ⇒ 00:18:04.739 Robert Tseng: Oh, you do? Okay. Well, I don’t think an S3 bucket is the way to go anyway, but, do they have an AWS now?
146 00:18:08.810 ⇒ 00:18:11.210 Robert Tseng: And that’s just Amazon Marketplace.
147 00:18:11.520 ⇒ 00:18:21.909 Robert Tseng: Yeah, okay, well, if you have AWS… okay, well, I’m… that kind of confuses me. So you already do have… yeah, I guess… is it possible that we could go poke around on what’s in AWS?
148 00:18:23.610 ⇒ 00:18:28.960 Robert Tseng: Yeah, I guess the decision is basically, do you want to pull from AWS, or do you want to pull from Mongo? So…
149 00:18:29.800 ⇒ 00:18:39.820 Robert Tseng: I think it’d be helpful to get Mark’s input on, like, what’s possible. He doesn’t have context into what we’re doing, by the way, like, he’s just trying to be helpful, and he’s a nice guy, but I think.
150 00:18:39.820 ⇒ 00:18:53.479 Alicia Shin: definitely offer, like, technical, like, guidance for us. He’s, he’s our, most tenured engineer, so… Okay. He’s the right person to sort of do that. And then, I think the next step, I want to bring Ashley into
151 00:18:53.530 ⇒ 00:19:12.860 Alicia Shin: the next meeting here, so she can start making sure, like, we are building towards what she ultimately needs to consume, and as our product, she can, she can also help validate, like… because if engineers have to do any work at all, she’ll have to put it, like, assign it out. Totally. So she’ll be able to… she’ll be, like, a great person to, like, offer, like, advice here.
152 00:19:13.100 ⇒ 00:19:30.560 Robert Tseng: Okay, yeah, well, I mean, so my point is that it’s just about setting up a new source, you have to decide where it’s going to come from, and then, like, yeah, we’re just identifying what… it will be an event that we need to pass in, which is, like, that true conversion event, or that purchase event.
153 00:19:30.560 ⇒ 00:19:36.870 Alicia Shin: But then also, I’m assuming for the plan types, since they’re… I don’t… we will confirm this, but…
154 00:19:36.870 ⇒ 00:19:49.489 Robert Tseng: Most likely, they are not reliable, because most of them are none. And if we’re going to have to update the plant types, that will also have to come in, through the same source, but through a different… it’s not going to be…
155 00:19:49.670 ⇒ 00:19:54.240 Robert Tseng: In event, we’re gonna have to, like, batch send a bunch of user data into
156 00:19:54.580 ⇒ 00:19:59.369 Robert Tseng: Into amplitude. So, that’s kind of how I envision things going.
157 00:20:01.070 ⇒ 00:20:01.680 Alicia Shin: Cool.
158 00:20:02.630 ⇒ 00:20:13.439 Robert Tseng: Okay, so that’s that. I think another thing I wanted to quickly highlight, well, it got really dark all of a sudden. But yeah, it’s just kind of…
159 00:20:13.530 ⇒ 00:20:38.500 Robert Tseng: Yeah, with the AI feature exploration, so now that you kind of gave me the ability to ask some cohorts, I was able to go and push this a little bit further. The idea is here is, like, okay, you know, here’s a set of AI features that we released, I’m able to show you things like all the unique users, based on the different, features, the event totals, kind of the share of active users that have adopted it. Obviously, the percentage is quite low, I have some callouts here that you can see.
160 00:20:39.180 ⇒ 00:20:51.259 Robert Tseng: It’s, like, only 1% of your active user base that’s really using the AI features. Seems like, week over week, that number is increasing, but there was, like, a spike in feature usage when it first launched that kind of just, like.
161 00:20:51.520 ⇒ 00:20:59.730 Robert Tseng: dipped, and has not really kind of come up since then. So, seems like the prompting is the most popular feature, obviously.
162 00:21:00.060 ⇒ 00:21:13.949 Alicia Shin: Okay, so that’s, like, our Ask AI, I guess? Yeah. Cool. Yeah, and you can go in Amp… in Amplitude, you can actually see all the different prompts that people are asking, so, no, we have, like, a… we have a Slack stream of them, it’s pretty interesting.
163 00:21:13.950 ⇒ 00:21:18.170 Robert Tseng: Yeah, but it’s just, like, a soup of random things that people are saying. Yeah, it’s pretty cool.
164 00:21:18.170 ⇒ 00:21:21.289 Alicia Shin: concerning if someone’s looking at my GPT stuff.
165 00:21:21.290 ⇒ 00:21:25.259 Robert Tseng: Yeah, I was like, wow, I should be careful about using Ask AI.
166 00:21:25.260 ⇒ 00:21:27.930 Alicia Shin: And we definitely should, yeah.
167 00:21:28.690 ⇒ 00:21:34.900 Alicia Shin: My, my husband works at Perplexity, and he was like, I have access to everything. I was like.
168 00:21:34.900 ⇒ 00:21:35.750 Robert Tseng: Oh, wow.
169 00:21:37.760 ⇒ 00:21:45.190 Robert Tseng: Oh, no, I definitely, say a lot to perplexity, so I should be careful now.
170 00:21:47.090 ⇒ 00:21:49.750 Alicia Shin: Okay. Cool.
171 00:21:49.750 ⇒ 00:21:58.409 Robert Tseng: Yeah, and then I just kind of teed up a couple questions that I felt like, you know, pretty universal, and, you know, part of the features we’re launching, driving meaningful user actions, like correlated revenue, so…
172 00:21:58.450 ⇒ 00:22:13.959 Robert Tseng: whatever, like, plan changes, you know, are there transactions and stuff like that. So, I mean, this is kind of the same chart that we had before of, like, the paid conversion funnel, but I swapped out the different cohorts, so what we can see is that
173 00:22:14.050 ⇒ 00:22:30.080 Robert Tseng: you know, active AI users, so users that have used AI features in the past 30 days, you know, they are obviously more… I mean, it makes sense that they are obviously paying, or they’re more prone to,
174 00:22:30.400 ⇒ 00:22:32.790 Robert Tseng: like, changed their plan, I suppose, and…
175 00:22:32.810 ⇒ 00:22:47.889 Robert Tseng: that would make sense, because in order to use these active AI features, you needed to be on a paid plan anyway, so I don’t really know if that’s that insightful in this particular case, but, like, this is, like, kind of an example of what I meant by, like.
176 00:22:47.890 ⇒ 00:23:08.639 Robert Tseng: I can now create different custom cohorts where I can say users that did XYZ things in the past, you know, whatever period, like, that’s… that’s why I’m able to… able to stitch together now. And I think that’s, you know, in order to maximize what your usage and amplitude, we need to be able to… we need to be doing this a lot more often.
177 00:23:09.010 ⇒ 00:23:10.290 Alicia Shin: Okay. Yep.
178 00:23:10.290 ⇒ 00:23:11.010 Robert Tseng: Yeah.
179 00:23:11.130 ⇒ 00:23:29.519 Robert Tseng: And then obviously retention, fewer… few users, so, you know, user volume’s too low to… low to really make this any substantial call-out, but sure, people who are using the AI, you know, feature are… that are, are, you know, coming back to the product more frequently.
180 00:23:31.010 ⇒ 00:23:39.840 Robert Tseng: Huh, there was one chart that didn’t end up showing up here, maybe it got stuck in the other space, but, yeah, and then I just had, like, a couple other charts that were…
181 00:23:40.290 ⇒ 00:23:48.579 Robert Tseng: You know, engagement metrics, they’re always gonna… we’re always gonna wanna cut it up by… by plan, by role, company size, so we’re…
182 00:23:48.580 ⇒ 00:24:03.749 Robert Tseng: not just looking at feature usage of all users, but we’re trying to, like, drill down into, like, which segments are really using these features the most, using features the most. And so, kind of defining, like, what are those, like, standardized cuts that we always want to be making.
183 00:24:04.230 ⇒ 00:24:09.620 Alicia Shin: Okay, this is great. Yeah, this is… this is awesome. It’s cool that we’re able to start tracking this.
184 00:24:09.870 ⇒ 00:24:10.500 Robert Tseng: Cool.
185 00:24:10.630 ⇒ 00:24:21.970 Robert Tseng: So the last thing I want to kind of tee up here is kind of transitioning to the, okay, well, what do we need more amplitude features for? Well, I think one thing is, you know, from a cohort perspective, I mean, I only built…
186 00:24:21.970 ⇒ 00:24:31.830 Robert Tseng: one additional one yesterday, so we can build up to three more, but, you know, I would be… you know, that’s not… it’s not that many. We could… we could definitely build a lot more. I could do active…
187 00:24:31.840 ⇒ 00:24:36.639 Robert Tseng: guide users. I could do active, whatever, and just kind of, like, start to bucket
188 00:24:37.420 ⇒ 00:24:50.530 Robert Tseng: every workflow, and adding all the different activities for that workflow, and creating, you know, workflow-based segments, like, that’s probably what I would want to do in order to better, kind of, do the,
189 00:24:50.690 ⇒ 00:24:53.969 Robert Tseng: Like, feature usage exploration.
190 00:24:54.170 ⇒ 00:25:00.300 Robert Tseng: So that’s kind of, like, one call-out. And then, being able to create, like.
191 00:25:00.930 ⇒ 00:25:14.810 Robert Tseng: proxy events, I keep referring to them as proxy events, but they’re basically, like, how do I use existing data that we’re already capturing to basically create events without… without adding more telemetry, or, like, creating more, like.
192 00:25:14.810 ⇒ 00:25:29.889 Robert Tseng: without relying on engineering, is basically the English version of that. So, like, for example, I’ve mentioned before, I can’t really create the downgrade event, because the way that the downgrade event is currently tracked.
193 00:25:29.940 ⇒ 00:25:31.960 Robert Tseng: Oops.
194 00:25:33.530 ⇒ 00:25:38.049 Robert Tseng: You would need to do something kind of, like, complicated, like,
195 00:25:38.340 ⇒ 00:25:51.020 Robert Tseng: Whatever, elements clicked, and filter by text, like… Great question mark, land.
196 00:25:51.170 ⇒ 00:26:00.629 Robert Tseng: Change? Yeah, whatever, like, it’s… you know, there’s, like, a whole sequence where I’m basically isolating the button within the…
197 00:26:00.760 ⇒ 00:26:07.880 Robert Tseng: within the UI for when you actually hit the downgrade, and that’s, like… it’s not… It…
198 00:26:08.660 ⇒ 00:26:17.039 Robert Tseng: that’s, like, not really… that… but that’s only really, like, the user pressing that button. It’s not, like, off of the plan changing.
199 00:26:17.370 ⇒ 00:26:31.190 Robert Tseng: we do capture plan type, and so in the events, you can see, like, a user was on Enterprise before, and they down… or, sorry, that’s not the best one, but, like, they were a business, and they downgraded to startup. Right.
200 00:26:31.400 ⇒ 00:26:42.450 Robert Tseng: And if I had computations, I could capture, you know, when plan changes, and… From… from that particular sequence.
201 00:26:42.730 ⇒ 00:26:51.609 Robert Tseng: create another kind of, like, property that says that this was… that logs it as a downgrade. Like, something like that. I… I’m not…
202 00:26:51.610 ⇒ 00:26:56.970 Alicia Shin: It’s creating a new column, basically, based on, like, the values that… like, the logic that we define.
203 00:26:56.970 ⇒ 00:26:58.649 Robert Tseng: Correct, yeah.
204 00:26:58.650 ⇒ 00:26:59.469 Alicia Shin: Got it.
205 00:26:59.650 ⇒ 00:27:07.260 Robert Tseng: I don’t know why Amplitude puts that behind a paywall now for the next year, because that seems like a very basic function for a product analytics tool, but .
206 00:27:07.260 ⇒ 00:27:14.739 Alicia Shin: It’s such a jump in the pricing, I was like, wow, what is this, like, magical feature that they’re, like, keeping from me?
207 00:27:15.080 ⇒ 00:27:17.159 Robert Tseng: What’s the difference? I’m curious.
208 00:27:17.160 ⇒ 00:27:19.050 Alicia Shin: So we pay, like, $500 a month or something.
209 00:27:19.050 ⇒ 00:27:19.630 Robert Tseng: Yeah.
210 00:27:19.630 ⇒ 00:27:22.800 Alicia Shin: And then they’re quoting us… $25,000 a year.
211 00:27:23.120 ⇒ 00:27:24.440 Robert Tseng: Oh, my goodness.
212 00:27:24.440 ⇒ 00:27:26.530 Henry Zhao: Yeah, that’s, like, usually what I’ve seen.
213 00:27:26.780 ⇒ 00:27:27.190 Alicia Shin: Yeah.
214 00:27:27.190 ⇒ 00:27:30.970 Robert Tseng: Enterprise is always, like… okay, yeah.
215 00:27:30.970 ⇒ 00:27:45.380 Alicia Shin: Yeah, so, like, my perspective on this is I can always make a case for it, but I think I’d rather focus… if the real value is the plan changes, let’s get the self-serve conversion stuff to, like, a really good place with all of the different
216 00:27:45.380 ⇒ 00:27:55.309 Alicia Shin: sources and data elements, and when we feel good about that, and if the team still believes, like, that next business question is the one to answer, we can add it in then, but I…
217 00:27:55.310 ⇒ 00:27:55.670 Robert Tseng: Okay.
218 00:27:55.670 ⇒ 00:28:04.549 Alicia Shin: to, like, upgrade, and, like, we’re not at a stable place with, like, the base, like, the V1. That feels like a V2 of, like, iteration.
219 00:28:04.690 ⇒ 00:28:05.470 Robert Tseng: Sure.
220 00:28:05.470 ⇒ 00:28:06.350 Alicia Shin: Cool. Yep.
221 00:28:06.600 ⇒ 00:28:19.910 Alicia Shin: Cool. Yeah, and it’ll be good for, ultimately, like, I want someone else to own that decision if they believe that it’s important. They can own the signing off of the budget that I’m gonna spend, so, we can… we can bring it up when… when we feel good about it.
222 00:28:20.220 ⇒ 00:28:27.129 Henry Zhao: But Robert, Ujuman mentioned, do we want to propose maybe using, like, what is that tool called to do the analysis for now?
223 00:28:27.810 ⇒ 00:28:30.090 Henry Zhao: Monkey something?
224 00:28:30.310 ⇒ 00:28:31.210 Robert Tseng: FuckDB.
225 00:28:31.210 ⇒ 00:28:32.220 Henry Zhao: DuckDB, yeah.
226 00:28:32.220 ⇒ 00:28:38.710 Robert Tseng: Oh, yeah, I guess,
227 00:28:40.460 ⇒ 00:28:43.240 Robert Tseng: Yeah, well, that would… it would… so, I guess…
228 00:28:44.740 ⇒ 00:28:57.670 Robert Tseng: Yeah, what we could do to add… to do that, well, I don’t want to… I want to see if we can look at AWS first, because if they already have an AWS and it’s already modeled data, we don’t need that to be… we could just run…
229 00:28:57.670 ⇒ 00:29:00.289 Henry Zhao: We’ll get AWS first, then we’ll talk tomorrow.
230 00:29:00.290 ⇒ 00:29:01.050 Robert Tseng: Yeah.
231 00:29:01.050 ⇒ 00:29:02.690 Henry Zhao: Let’s kind of work on what… yeah.
232 00:29:02.820 ⇒ 00:29:03.370 Henry Zhao: Well, that number.
233 00:29:03.370 ⇒ 00:29:03.750 Robert Tseng: 25.
234 00:29:03.750 ⇒ 00:29:05.979 Henry Zhao: That’s good to know, so we can look for alternatives.
235 00:29:06.170 ⇒ 00:29:08.700 Alicia Shin: Yeah, I’m pretty sure that’s what she quoted me. I was like.
236 00:29:10.190 ⇒ 00:29:12.650 Robert Tseng: Yeah, yeah, no, that’s pretty standard, so…
237 00:29:12.900 ⇒ 00:29:21.960 Alicia Shin: Cool. Okay, this sounds great. I… I pinged Ashley to see what her availability’s like next week, and I don’t know…
238 00:29:22.300 ⇒ 00:29:28.450 Alicia Shin: If she’s able to make this, but can we tentatively… any chance you’re available at 11.30 on Tuesday?
239 00:29:29.230 ⇒ 00:29:30.560 Robert Tseng: This is Eastern, right?
240 00:29:30.560 ⇒ 00:29:34.470 Alicia Shin: Yes. And I know today’s Thursday, so if we want to look at.
241 00:29:35.340 ⇒ 00:29:35.720 Henry Zhao: Damn.
242 00:29:35.720 ⇒ 00:29:39.629 Alicia Shin: Or 12.30 on Thursday next week, if you prefer, like, a week.
243 00:29:40.050 ⇒ 00:29:41.290 Henry Zhao: I’m also free that time.
244 00:29:41.760 ⇒ 00:29:45.010 Robert Tseng: Yeah, I mean, I, I, I mean, I guess…
245 00:29:46.660 ⇒ 00:29:48.820 Robert Tseng: Yeah, I think Thursday would be better for me.
246 00:29:48.820 ⇒ 00:29:54.520 Alicia Shin: Okay, okay. Let’s move ours to then, and then I’ll add Ashley to that one.
247 00:29:54.520 ⇒ 00:29:58.219 Robert Tseng: Okay, I’ll move it to 12.30 Thursday, kind of.
248 00:29:58.580 ⇒ 00:29:59.690 Robert Tseng: That’s great.
249 00:29:59.690 ⇒ 00:30:00.150 Alicia Shin: Okay.
250 00:30:00.300 ⇒ 00:30:01.090 Robert Tseng: Yeah.
251 00:30:01.860 ⇒ 00:30:10.850 Alicia Shin: Awesome. Hopefully she can join. I feel like we’re at the point where her input will be very valuable to get us, moving even further. But this is looking great so far, so…
252 00:30:11.170 ⇒ 00:30:24.200 Robert Tseng: Okay, great. Yeah, so we have, we have a couple things to follow up on, so hopefully we’ll be able to handle the data, like, quality stuff async, and then, yeah, if there’s anything, any new requirements, you can just, you know, send them our way, yeah.
253 00:30:24.200 ⇒ 00:30:33.930 Alicia Shin: For sure, yeah. Ashley will be joining the meeting, so we’ll be able to sort of clarify, but I imagine, like, everything we have today, we’d still want to track regardless of what sort of new requirements are added.
254 00:30:34.110 ⇒ 00:30:34.840 Robert Tseng: Sure.
255 00:30:36.000 ⇒ 00:30:36.750 Alicia Shin: Okay.
256 00:30:37.020 ⇒ 00:30:38.719 Robert Tseng: Okay, sounds good. Alright.
257 00:30:38.720 ⇒ 00:30:39.360 Alicia Shin: Okay, bye.
258 00:30:39.940 ⇒ 00:30:40.640 Robert Tseng: Right.