Meeting Title: Data Analysis Office Hours Date: 2025-10-28 Meeting participants: Robert Tseng, Uttam Kumaran, Amber Lin, Henry Zhao, Mustafa Raja
WEBVTT
1 00:01:20.810 ⇒ 00:01:21.750 Uttam Kumaran: Hello!
2 00:01:23.310 ⇒ 00:01:24.380 Robert Tseng: Hello?
3 00:01:26.580 ⇒ 00:01:27.490 Uttam Kumaran: Okay.
4 00:01:29.630 ⇒ 00:01:34.350 Uttam Kumaran: this day was going good, and then I didn’t get the PM support I needed, and I did a bunch of yelling.
5 00:01:37.600 ⇒ 00:01:41.510 Robert Tseng: Sorry, what was the last, like, 10 seconds? My mic was switching.
6 00:01:41.510 ⇒ 00:01:42.770 Uttam Kumaran: I said I,
7 00:01:43.410 ⇒ 00:01:48.440 Uttam Kumaran: I didn’t get the PM support. I was good, and then I didn’t get the PM support I needed.
8 00:01:48.760 ⇒ 00:01:52.040 Uttam Kumaran: And I started yelling, and then things started getting better.
9 00:01:53.840 ⇒ 00:01:55.440 Robert Tseng: For… for what is… for what?
10 00:01:56.210 ⇒ 00:01:57.359 Uttam Kumaran: No, dude, it’s like…
11 00:01:57.480 ⇒ 00:02:01.100 Uttam Kumaran: Look, I came in, I’m running all the meetings, but I need Rico and Amber to, like.
12 00:02:01.390 ⇒ 00:02:03.260 Robert Tseng: Oh, yeah. Rough out the edges, like…
13 00:02:04.190 ⇒ 00:02:09.019 Uttam Kumaran: listen to the follow-ups, like, all… I’m a follow-up machine, like, all I’m doing is getting updates.
14 00:02:09.220 ⇒ 00:02:10.849 Uttam Kumaran: Every single person.
15 00:02:11.170 ⇒ 00:02:15.180 Uttam Kumaran: You know, and the least you can do is, like, note those down, follow up on them.
16 00:02:15.720 ⇒ 00:02:19.020 Uttam Kumaran: Right? Like, do I have to say that out loud, I guess.
17 00:02:19.390 ⇒ 00:02:20.160 Uttam Kumaran: Bye.
18 00:02:22.840 ⇒ 00:02:26.960 Uttam Kumaran: I’m just, like, just telling them to be more mindful, put some more thought into it, so…
19 00:02:27.240 ⇒ 00:02:29.330 Uttam Kumaran: Just teaching everybody slowly.
20 00:02:39.150 ⇒ 00:02:44.769 Robert Tseng: Let me see if I can add you to a channel with Cantor.
21 00:02:45.150 ⇒ 00:02:46.719 Uttam Kumaran: Okay. Yeah, that’d be great.
22 00:02:48.390 ⇒ 00:02:51.209 Uttam Kumaran: If not, I can… I’ll start a group DM.
23 00:02:53.370 ⇒ 00:02:54.710 Robert Tseng: I think we can.
24 00:03:37.670 ⇒ 00:03:47.239 Robert Tseng: Okay, I just… through Unislack with just him, I think is probably the better approach.
25 00:03:48.490 ⇒ 00:03:51.470 Robert Tseng: Okay, I gotta, I gotta stay away from Slack, there’s just too much going on there.
26 00:03:51.470 ⇒ 00:03:52.050 Uttam Kumaran: Okay.
27 00:03:52.050 ⇒ 00:03:55.909 Robert Tseng: Alright, yeah, so read me stuff, okay.
28 00:03:56.840 ⇒ 00:04:00.840 Robert Tseng: Oh, no, it’s actually… Do it,
29 00:04:04.250 ⇒ 00:04:05.690 Robert Tseng: I’m gonna share my screen.
30 00:04:10.460 ⇒ 00:04:16.549 Robert Tseng: So, basically, what I was trying to do here, again, was once again, roll everything up into a few
31 00:04:17.260 ⇒ 00:04:20.899 Robert Tseng: Kind of just… Types of analysis.
32 00:04:21.110 ⇒ 00:04:29.709 Robert Tseng: I’ve read through her docs, there’s not that much here, it’s just, like, her asking questions about, like, what this data is. So, looking through that, looking through the notes,
33 00:04:29.910 ⇒ 00:04:40.350 Robert Tseng: Yeah, right, there’s, like, one bucket of work, and these could be really the projects on linear. I think linear projects maybe are kind of maybe labeled more, like, project by project, less about
34 00:04:40.650 ⇒ 00:04:50.809 Robert Tseng: focus areas, but one is, like, you know, conversion funnel analytics, right? So that’s pretty much this work of… is pretty much what we’ve been doing for the first
35 00:04:51.070 ⇒ 00:05:04.020 Robert Tseng: sets of amplitude reports. Then there’s this kind of, like, pricing adoption. Well, it’s more like… Feature, pricing, experimentation.
36 00:05:04.930 ⇒ 00:05:08.310 Robert Tseng: Yeah, I think that’s a more precise way to call this.
37 00:05:08.570 ⇒ 00:05:14.259 Robert Tseng: Conversion, funnel… I’m just gonna call it optimization. Analytics means nothing.
38 00:05:14.600 ⇒ 00:05:19.159 Robert Tseng: Data infrastructure… I think it’s more, like, product data.
39 00:05:19.520 ⇒ 00:05:26.440 Robert Tseng: Infrastructure… Hmm…
40 00:05:31.250 ⇒ 00:05:32.010 Robert Tseng: Sure.
41 00:05:32.260 ⇒ 00:05:35.750 Robert Tseng: And then this is more like,
42 00:05:39.310 ⇒ 00:05:40.520 Robert Tseng: Patient…
43 00:05:43.680 ⇒ 00:05:44.600 Robert Tseng: Sure.
44 00:05:47.820 ⇒ 00:05:49.620 Robert Tseng: Okay, so…
45 00:05:49.880 ⇒ 00:06:03.280 Robert Tseng: I think there’s a next step on this conversion funnel stuff. There’s… she gave some tactical feedback on, like, adjustments to our existing reports. We haven’t validated the funnel steps against Mongo yet, so I think that’s still really the blocker there.
46 00:06:04.850 ⇒ 00:06:19.599 Robert Tseng: Then with, like, feature and pricing, this is less urgent. They haven’t… well, I guess they have their new AI features out, and so there is a follow-on analysis from what you shared that’s less about AI feature usage exploration, it’s more like,
47 00:06:26.880 ⇒ 00:06:32.270 Robert Tseng: Well, actually, it’s probably still the same thing, actually. I just gotta… Add more details there.
48 00:06:32.380 ⇒ 00:06:41.070 Robert Tseng: The tracking effort is also not that urgent, because we still have to go and find the right engineer to implement. We’re just pretty much trying to use what we have.
49 00:06:41.280 ⇒ 00:06:43.300 Robert Tseng: And then building up a…
50 00:06:43.740 ⇒ 00:06:49.269 Robert Tseng: like, a backlog of, like, capabilities that we need. This seems like what we kind of…
51 00:06:49.450 ⇒ 00:06:55.980 Robert Tseng: pushed for so far is not enough to push them into the next tier of amplitude, which is fine. We can just exhaust
52 00:06:56.310 ⇒ 00:07:00.320 Robert Tseng: Its usage before we… we asked for it.
53 00:07:02.930 ⇒ 00:07:10.889 Robert Tseng: And then we have the activation and retention. So I think this is really the net new work that needs to happen this week, which is…
54 00:07:11.320 ⇒ 00:07:22.619 Robert Tseng: just like, she gave some direction around, okay, I would expect a activated user to use the,
55 00:07:24.410 ⇒ 00:07:30.200 Robert Tseng: what was it? Like, the guides product? Was it guides? Guides? Question mark?
56 00:07:31.750 ⇒ 00:07:35.150 Robert Tseng: What was she saying?
57 00:07:40.780 ⇒ 00:07:41.340 Robert Tseng: Mmm…
58 00:07:54.530 ⇒ 00:07:58.440 Robert Tseng: What did the head of product say?
59 00:07:59.660 ⇒ 00:08:04.350 Robert Tseng: about… Future groups and activation.
60 00:08:13.700 ⇒ 00:08:16.860 Robert Tseng: Okay, something about, like, references…
61 00:08:18.430 ⇒ 00:08:27.220 Robert Tseng: you know, references to guides, and so, like, I’d probably just go and test a couple of these cohorts to really see if this is… these actually lead to meaningful engagement.
62 00:08:27.380 ⇒ 00:08:42.489 Robert Tseng: And then I want to do my own exploration that’s not influenced by her, of, like, what are the early features that they’re using within, like, their first week, first two weeks. We kind of did that lightly. It was just, like, any feature, but I just want to be able to say what specific features.
63 00:08:42.590 ⇒ 00:08:44.819 Robert Tseng: So I think that’s what’s really gonna flush out.
64 00:08:44.820 ⇒ 00:08:47.600 Uttam Kumaran: And so, in terms of that, like, what do you… your…
65 00:08:47.850 ⇒ 00:08:51.149 Uttam Kumaran: In Amplitude, though, do you have the user properties, like, created at?
66 00:08:51.530 ⇒ 00:08:53.430 Robert Tseng: Yeah, so… a lot.
67 00:08:53.430 ⇒ 00:08:53.910 Uttam Kumaran: Oh, okay.
68 00:08:53.910 ⇒ 00:09:00.099 Robert Tseng: So… I have most of what I need in amplitude.
69 00:09:07.450 ⇒ 00:09:08.870 Robert Tseng: So…
70 00:09:13.850 ⇒ 00:09:17.550 Robert Tseng: But honestly, probably just duplicate this.
71 00:09:21.520 ⇒ 00:09:27.199 Robert Tseng: And I’ll haul it, hold up for now.
72 00:09:36.150 ⇒ 00:09:37.440 Robert Tseng: Oh, no.
73 00:09:43.920 ⇒ 00:09:51.129 Uttam Kumaran: So all that’s in here is plan type, And the project, they’re like, Related to.
74 00:09:51.670 ⇒ 00:09:56.370 Uttam Kumaran: Like, you can answer… that’s all that’s in Mongo, like, you don’t need to… you don’t need that for a lot of stuff.
75 00:09:58.770 ⇒ 00:10:04.630 Robert Tseng: Yeah, I think the… my question is, like, is the plan type even, like…
76 00:10:05.150 ⇒ 00:10:07.660 Robert Tseng: Do we have it… do we have it?
77 00:10:08.010 ⇒ 00:10:11.069 Robert Tseng: is it split correctly? Like,
78 00:10:17.250 ⇒ 00:10:18.420 Robert Tseng: Go back…
79 00:10:21.710 ⇒ 00:10:25.389 Robert Tseng: And I go by type.
80 00:10:26.180 ⇒ 00:10:27.080 Robert Tseng: No, bulk.
81 00:10:27.560 ⇒ 00:10:28.960 Robert Tseng: Motion funnel.
82 00:10:29.410 ⇒ 00:10:30.380 Robert Tseng: Whoa.
83 00:10:40.320 ⇒ 00:10:48.350 Robert Tseng: Yeah, so if I, like, just… I don’t know, let’s just pick this for… Simplicity, and…
84 00:10:48.770 ⇒ 00:10:54.770 Robert Tseng: Project created, already broken out by… Plan…
85 00:11:00.750 ⇒ 00:11:08.109 Robert Tseng: Right, these are all the different plan types. This doesn’t really make sense. Just look at weekly over the past 12 weeks. Great.
86 00:11:08.380 ⇒ 00:11:22.470 Robert Tseng: Yeah, it’s like, most of the users don’t have a plan tight. So, like, this is… this is what I… this is what I wanted from the QA, where, in their mind, it should just be free, startup, business, and enterprise. There’s all these extra stuff that’s, like.
87 00:11:22.600 ⇒ 00:11:28.749 Robert Tseng: I mean, I’m assuming that all the none are also free, and also whatever, so I can go and just, like.
88 00:11:31.060 ⇒ 00:11:40.830 Robert Tseng: do that, but, like, and then just, does this volume even make sense? If I just took it, like, the past month, and I, like, I don’t know, did KPI, and I just did monthly…
89 00:11:41.360 ⇒ 00:11:44.750 Robert Tseng: And I did this task.
90 00:11:45.530 ⇒ 00:11:51.779 Robert Tseng: Mods… Like, would these numbers match up to what we see in Mongo?
91 00:11:55.450 ⇒ 00:12:02.899 Robert Tseng: And then she’s also saying, oh, there should be trend… like, payment success should be coming from
92 00:12:03.280 ⇒ 00:12:13.740 Robert Tseng: should be tied to Stripe, which, if it is, it must be coming through Mongo, because it’s not coming directly into Amplitude. I don’t really believe the transactions that I’m seeing here, either.
93 00:12:14.620 ⇒ 00:12:18.559 Robert Tseng: Like, these, the subscription event, success event.
94 00:12:19.330 ⇒ 00:12:20.240 Uttam Kumaran: Building events.
95 00:12:20.240 ⇒ 00:12:21.569 Robert Tseng: Yeah, the billing events.
96 00:12:23.850 ⇒ 00:12:32.219 Robert Tseng: Yeah. So, I mean, there’s not that much I want in Mongo, I just wanted to validate that this funnel actually makes sense. She’s fine with using this as a proxy, so…
97 00:12:32.220 ⇒ 00:12:33.330 Uttam Kumaran: Okay, okay, cool.
98 00:12:38.270 ⇒ 00:12:38.950 Robert Tseng: Yep.
99 00:12:39.490 ⇒ 00:12:40.260 Uttam Kumaran: Okay.
100 00:12:40.700 ⇒ 00:12:44.069 Robert Tseng: So, I mean, I figured I would just, like, stay on and just start to…
101 00:12:44.070 ⇒ 00:12:44.709 Uttam Kumaran: Yeah, let’s do it.
102 00:12:44.710 ⇒ 00:12:46.200 Robert Tseng: Crank through this, yeah, but .
103 00:12:46.200 ⇒ 00:12:50.969 Uttam Kumaran: So, I guess maybe… maybe one thing I’ll just ask you is I, I wrote out, like.
104 00:12:51.310 ⇒ 00:12:55.710 Uttam Kumaran: what we need from Mongo, You can use that if you want to.
105 00:12:55.890 ⇒ 00:12:56.340 Robert Tseng: Okay.
106 00:12:56.340 ⇒ 00:12:59.709 Uttam Kumaran: I… I don’t, like… Yeah, I mean…
107 00:13:01.860 ⇒ 00:13:09.639 Uttam Kumaran: Yeah. If you give me another shot, I can close this out, but, like, if it’s sensitive, you know, we could do it later, whatever, but
108 00:13:09.900 ⇒ 00:13:11.769 Uttam Kumaran: I’ll send it, I’ll send it in,
109 00:13:13.950 ⇒ 00:13:15.510 Uttam Kumaran: I’ll send in the REA channel.
110 00:13:16.820 ⇒ 00:13:17.560 Robert Tseng: Okay.
111 00:13:17.790 ⇒ 00:13:25.590 Robert Tseng: I also had a draft of things that we kinda… Like, project management stuff, so…
112 00:13:25.950 ⇒ 00:13:29.199 Robert Tseng: There’s project management stuff, not really…
113 00:13:41.310 ⇒ 00:13:43.790 Robert Tseng: Peace.
114 00:13:45.820 ⇒ 00:13:46.520 Uttam Kumaran: Yeah.
115 00:13:47.650 ⇒ 00:13:49.179 Robert Tseng: You want me to shoot this over to them?
116 00:13:49.580 ⇒ 00:13:52.370 Uttam Kumaran: It’s up to you, like…
117 00:13:53.320 ⇒ 00:13:57.639 Uttam Kumaran: I… I… I just want to make sure that it’s necessary for what you need immediately.
118 00:13:58.220 ⇒ 00:14:07.160 Robert Tseng: Okay. Well, I’m thinking I will probably ask them for the Mongo stuff along the way, but I want to just… I want to share something with them today, so…
119 00:14:08.810 ⇒ 00:14:13.370 Robert Tseng: Yeah, I’m gonna share with them, kind of, this…
120 00:14:15.030 ⇒ 00:14:21.910 Robert Tseng: doc that I put together. Obviously, there’s some blanks in here, but, like, I just want them to know how I’m thinking about it. So I’m gonna send over a loom and…
121 00:14:22.030 ⇒ 00:14:25.839 Robert Tseng: I mean, I’m gonna just tick… tick one and go for it.
122 00:14:26.430 ⇒ 00:14:34.690 Robert Tseng: So, anyway, going back to what I was… Thinking with this… Okay, so…
123 00:14:34.900 ⇒ 00:14:38.039 Robert Tseng: I’m gonna take some time to just structure my thinking here.
124 00:14:39.000 ⇒ 00:14:51.900 Robert Tseng: So… What do I want to know? Well, I’m gonna wanna know… Do… Read me… is the…
125 00:14:56.220 ⇒ 00:15:02.789 Robert Tseng: is the activated… What is inactivated?
126 00:15:08.010 ⇒ 00:15:08.760 Robert Tseng: Excuse me.
127 00:15:10.130 ⇒ 00:15:31.860 Robert Tseng: Did it go to… The news references… And then… guides… What other… features… Comfort.
128 00:15:33.900 ⇒ 00:15:41.780 Robert Tseng: Yeah, and then… what are the… Earliest.
129 00:15:44.180 ⇒ 00:15:53.480 Robert Tseng: What are the features… what are the features that users, used during… There…
130 00:15:53.770 ⇒ 00:15:55.379 Uttam Kumaran: First few weeks.
131 00:15:58.420 ⇒ 00:16:06.800 Robert Tseng: Yeah, and then that’s gonna be able to tell her what…
132 00:16:15.560 ⇒ 00:16:24.610 Robert Tseng: Identify… Combinations, like, future combinations that have the highest… That’s correlate.
133 00:16:29.150 ⇒ 00:16:30.920 Robert Tseng: Next, for sure.
134 00:16:31.840 ⇒ 00:16:33.750 Robert Tseng: with paid versions.
135 00:16:35.280 ⇒ 00:16:44.290 Robert Tseng: And then… do the AI features in, Drive.
136 00:16:45.540 ⇒ 00:16:51.269 Robert Tseng: more usage across to AI features, across products.
137 00:16:54.950 ⇒ 00:17:13.949 Robert Tseng: Yeah, identify… serious… Early behavioral signals that, new behaviors that signal.
138 00:17:14.349 ⇒ 00:17:24.329 Robert Tseng: I purchase intent… Identify… Friction!
139 00:17:25.150 ⇒ 00:17:26.799 Robert Tseng: points, and…
140 00:17:31.360 ⇒ 00:17:32.300 Robert Tseng: Yes.
141 00:17:34.290 ⇒ 00:17:37.729 Robert Tseng: Invest here with customers for youth.
142 00:17:43.590 ⇒ 00:17:48.110 Robert Tseng: Okay, I think that’s enough for me to start drafting this.
143 00:17:54.770 ⇒ 00:17:55.750 Robert Tseng: Okay.
144 00:17:56.160 ⇒ 00:18:06.240 Robert Tseng: So… Great. Now I’m here… I believe there were a couple useful charts.
145 00:18:18.180 ⇒ 00:18:19.139 Robert Tseng: Oh my god.
146 00:18:36.170 ⇒ 00:18:38.620 Robert Tseng: Okay, I’ll keep those.
147 00:18:47.540 ⇒ 00:18:49.920 Robert Tseng: Yeah, I’ll keep this.
148 00:18:53.120 ⇒ 00:18:59.359 Robert Tseng: Project and subscription, official usage… It allows my plan type.
149 00:19:01.850 ⇒ 00:19:02.720 Robert Tseng: Okay.
150 00:19:04.440 ⇒ 00:19:05.929 Robert Tseng: And then…
151 00:19:09.260 ⇒ 00:19:14.669 Robert Tseng: Yeah, I mean, some of this stuff is… helpful, I guess.
152 00:19:24.140 ⇒ 00:19:26.829 Robert Tseng: I don’t need that.
153 00:19:28.350 ⇒ 00:19:29.200 Robert Tseng: Gosh.
154 00:19:29.520 ⇒ 00:19:30.310 Robert Tseng: Okay.
155 00:19:32.860 ⇒ 00:19:42.850 Robert Tseng: So, activated user… Let’s go with… Mmm…
156 00:19:48.380 ⇒ 00:19:53.460 Robert Tseng: Weekly… Completed signups.
157 00:19:54.340 ⇒ 00:19:56.600 Robert Tseng: No, any active.
158 00:19:58.270 ⇒ 00:19:59.320 Robert Tseng: Thanks.
159 00:19:59.650 ⇒ 00:20:15.700 Robert Tseng: And… We’re going to… I don’t think this’ll change… Unique users… 7 a week…
160 00:20:35.510 ⇒ 00:20:39.650 Robert Tseng: Oh… Not nearly.
161 00:20:39.650 ⇒ 00:20:41.039 Uttam Kumaran: That’s just any event.
162 00:20:41.970 ⇒ 00:20:43.400 Robert Tseng: Yeah, definitely.
163 00:20:43.760 ⇒ 00:20:45.090 Robert Tseng: Yeah.
164 00:20:47.340 ⇒ 00:20:48.989 Uttam Kumaran: What the fuck is an active event?
165 00:20:49.500 ⇒ 00:20:51.790 Robert Tseng: So you can…
166 00:20:51.790 ⇒ 00:20:52.220 Uttam Kumaran: Oh.
167 00:20:52.220 ⇒ 00:20:57.289 Robert Tseng: You can mark events as, like, not active, so, like, an app open or whatever.
168 00:20:57.290 ⇒ 00:20:58.250 Uttam Kumaran: Filter it out, yeah, yeah.
169 00:20:58.250 ⇒ 00:21:04.669 Robert Tseng: Yeah, yeah. I mean, I will probably make it more narrow later, but I just wanted something to start with.
170 00:21:04.870 ⇒ 00:21:10.999 Robert Tseng: So, like, I usually just have, like, something, and then I’m… then I’m, like, starting to think about how I’m gonna construct this cohort.
171 00:21:11.120 ⇒ 00:21:15.779 Robert Tseng: So they have to have used references, so I’m, like, trying to create a segment around that.
172 00:21:16.350 ⇒ 00:21:20.309 Robert Tseng: So, I kind of had this AI feature.
173 00:21:20.420 ⇒ 00:21:24.729 Robert Tseng: you had this segment that’s performed AI features.
174 00:21:24.860 ⇒ 00:21:28.199 Robert Tseng: But I’m probably gonna do something else.
175 00:21:29.190 ⇒ 00:21:35.350 Robert Tseng: And… trial users… okay, I did that already.
176 00:21:35.790 ⇒ 00:21:46.760 Robert Tseng: Okay, so all users that have performed… References…
177 00:21:48.710 ⇒ 00:21:52.119 Robert Tseng: I don’t recall off the top of my head, so…
178 00:21:57.930 ⇒ 00:21:59.290 Robert Tseng: read me…
179 00:22:27.860 ⇒ 00:22:37.550 Robert Tseng: Okay… References… Okay, so I believe that this is once…
180 00:22:40.240 ⇒ 00:22:45.810 Robert Tseng: They have created an API definition, and I think that’s what references is.
181 00:22:47.200 ⇒ 00:22:47.970 Robert Tseng: And what.
182 00:22:47.970 ⇒ 00:22:49.740 Uttam Kumaran: What are you using to log into Amplitude?
183 00:22:51.370 ⇒ 00:22:55.690 Robert Tseng: Into this amplitude? Oh, I can add you as a user, but I just…
184 00:22:56.440 ⇒ 00:22:57.530 Uttam Kumaran: Yeah, can you add me?
185 00:22:57.530 ⇒ 00:22:58.100 Robert Tseng: Yeah.
186 00:23:03.040 ⇒ 00:23:06.740 Uttam Kumaran: Okay, yeah, this I can do. I’ll go play around later today.
187 00:23:10.200 ⇒ 00:23:15.759 Uttam Kumaran: So this is saying that there’s just 58 people that are active, like, in the last week?
188 00:23:21.470 ⇒ 00:23:23.060 Uttam Kumaran: Isn’t that, like, fucking horrible?
189 00:23:30.380 ⇒ 00:23:32.219 Uttam Kumaran: Oh, okay, I was gonna say.
190 00:23:32.220 ⇒ 00:23:33.870 Robert Tseng: I was like, that’s pretty low.
191 00:23:34.410 ⇒ 00:23:36.189 Uttam Kumaran: Yeah, okay, okay, okay.
192 00:23:36.190 ⇒ 00:23:37.670 Robert Tseng: Okay.
193 00:23:41.560 ⇒ 00:23:43.950 Robert Tseng: Okay, yeah, that was just, like, I clicked the wrong one.
194 00:23:44.190 ⇒ 00:23:44.730 Uttam Kumaran: Okay.
195 00:23:45.240 ⇒ 00:23:45.870 Robert Tseng: Okay.
196 00:23:49.090 ⇒ 00:23:51.310 Robert Tseng: sure.
197 00:23:52.460 ⇒ 00:23:53.260 Robert Tseng: Okay.
198 00:23:53.970 ⇒ 00:23:54.630 Uttam Kumaran: Thanks.
199 00:23:56.900 ⇒ 00:23:57.530 Robert Tseng: Nope.
200 00:24:07.340 ⇒ 00:24:08.260 Robert Tseng: posts…
201 00:24:15.570 ⇒ 00:24:17.910 Robert Tseng: Oof. You have a valid URL.
202 00:24:18.690 ⇒ 00:24:21.159 Robert Tseng: Okay, I’m just gonna say, very first memory.
203 00:24:26.760 ⇒ 00:24:27.739 Robert Tseng: Is that not a…
204 00:24:27.740 ⇒ 00:24:29.370 Uttam Kumaran: We’ll do the HTTPS.
205 00:24:29.550 ⇒ 00:24:30.300 Robert Tseng: Okay.
206 00:24:31.870 ⇒ 00:24:33.699 Robert Tseng: Aha, there you go.
207 00:24:43.400 ⇒ 00:24:45.560 Uttam Kumaran: Oh, they’re also way over their MTUs.
208 00:24:46.530 ⇒ 00:24:47.389 Robert Tseng: Yeah, they are.
209 00:24:53.100 ⇒ 00:25:00.060 Robert Tseng: So, reference, form submitted… all of these steps to this,
210 00:25:07.430 ⇒ 00:25:13.790 Robert Tseng: Okay, well… It needs to have performed 3 forms. I did.
211 00:25:19.200 ⇒ 00:25:21.000 Robert Tseng: pre-filter…
212 00:25:30.650 ⇒ 00:25:39.839 Robert Tseng: Your eye definition… And then there’s some… Is that it?
213 00:25:40.360 ⇒ 00:25:49.350 Robert Tseng: I can add another one. Getting started, new edit… Quicks.
214 00:25:50.850 ⇒ 00:25:52.569 Robert Tseng: I don’t even have to submit that.
215 00:25:54.460 ⇒ 00:25:57.250 Robert Tseng: I don’t care about form started, I care about form.
216 00:25:58.080 ⇒ 00:25:59.420 Robert Tseng: Submitted.
217 00:25:59.980 ⇒ 00:26:04.060 Robert Tseng: With… the same property.
218 00:26:13.290 ⇒ 00:26:22.930 Robert Tseng: Okay, so if they’ve submitted this… Definitely fewer users now.
219 00:26:24.530 ⇒ 00:26:29.609 Robert Tseng: So they don’t have to have done it in the past week, they just have to have done it in the past 30 days.
220 00:26:31.630 ⇒ 00:26:40.250 Robert Tseng: I’m going to… look at…
221 00:26:51.090 ⇒ 00:26:52.280 Robert Tseng: Last step.
222 00:26:54.900 ⇒ 00:26:57.389 Robert Tseng: Let’s just see…
223 00:26:57.390 ⇒ 00:26:59.980 Uttam Kumaran: Okay, I’m hopping to a Looker meeting. I’ll be back.
224 00:26:59.980 ⇒ 00:27:00.310 Robert Tseng: Yeah, yeah.
225 00:27:00.310 ⇒ 00:27:01.339 Uttam Kumaran: is done quickly.
226 00:27:02.330 ⇒ 00:27:03.310 Robert Tseng: No worries.
227 00:27:03.670 ⇒ 00:27:10.150 Uttam Kumaran: Alright, I’ll just be here until I’m done. Yeah, hang out here! No, I want to come back, and I want to do this, I’m just like, yeah, yes, I’ll be back.
228 00:27:10.150 ⇒ 00:27:10.799 Robert Tseng: Yeah, yeah.
229 00:27:13.060 ⇒ 00:27:13.910 Robert Tseng: Okay.
230 00:27:17.310 ⇒ 00:27:24.510 Robert Tseng: Hmm… I should probably limit.
231 00:27:25.940 ⇒ 00:27:30.020 Robert Tseng: To users that are new.
232 00:27:51.020 ⇒ 00:27:53.529 Robert Tseng: Let’s just look at new users.
233 00:28:01.920 ⇒ 00:28:04.160 Robert Tseng: Launches that have 6 months.
234 00:28:08.840 ⇒ 00:28:10.580 Robert Tseng: Did anything…
235 00:28:31.360 ⇒ 00:28:33.109 Robert Tseng: Let me narrow it down.
236 00:28:35.360 ⇒ 00:28:36.130 Robert Tseng: Yeah.
237 00:28:47.150 ⇒ 00:28:49.390 Robert Tseng: These numbers didn’t get updated.
238 00:28:51.630 ⇒ 00:28:52.320 Robert Tseng: Oop.
239 00:28:54.270 ⇒ 00:28:58.730 Robert Tseng: Okay, let me save this first. Something is not loading correctly.
240 00:29:00.870 ⇒ 00:29:04.449 Robert Tseng: The heck? How’s this still 21? I don’t think it is.
241 00:29:05.670 ⇒ 00:29:08.770 Robert Tseng: 21,000 people.
242 00:29:10.230 ⇒ 00:29:12.900 Robert Tseng: Has not done this.
243 00:29:15.910 ⇒ 00:29:19.340 Robert Tseng: Let me first… Bye.
244 00:29:21.440 ⇒ 00:29:22.540 Robert Tseng: Bye.
245 00:29:27.940 ⇒ 00:29:28.820 Robert Tseng: Meh.
246 00:29:37.880 ⇒ 00:29:40.570 Robert Tseng: Okay, we probably will just have to create that cohort.
247 00:29:41.720 ⇒ 00:29:47.690 Robert Tseng: 30 day…
248 00:30:34.820 ⇒ 00:30:37.110 Robert Tseng: Hmm. Hmm.
249 00:30:53.700 ⇒ 00:30:54.380 Robert Tseng: Oh my gosh.
250 00:30:54.710 ⇒ 00:30:58.739 Robert Tseng: What am I getting stuck on?
251 00:31:08.960 ⇒ 00:31:10.970 Robert Tseng: I don’t cover new users.
252 00:31:11.370 ⇒ 00:31:12.410 Robert Tseng: Darn.
253 00:31:12.920 ⇒ 00:31:14.550 Robert Tseng: New plan.
254 00:31:15.970 ⇒ 00:31:18.840 Robert Tseng: It’s not really what I want.
255 00:31:22.910 ⇒ 00:31:23.780 Robert Tseng: Okay.
256 00:31:34.600 ⇒ 00:31:38.500 Robert Tseng: Where do I create a new customer, a new user?
257 00:31:39.030 ⇒ 00:31:41.750 Robert Tseng: cohort from House 30.
258 00:31:43.280 ⇒ 00:31:46.540 Robert Tseng: With only new active events.
259 00:32:08.340 ⇒ 00:32:11.919 Robert Tseng: I don’t think I could do… I did not perform, where…
260 00:32:20.020 ⇒ 00:32:20.870 Robert Tseng: Oh.
261 00:32:23.650 ⇒ 00:32:24.550 Robert Tseng: Okay.
262 00:32:26.180 ⇒ 00:32:28.710 Robert Tseng: Wait, is that all you? That’s all I have to do?
263 00:32:37.890 ⇒ 00:32:40.980 Robert Tseng: Okay, well… Such as life.
264 00:32:48.130 ⇒ 00:32:49.750 Robert Tseng: Go ahead, the new…
265 00:33:17.060 ⇒ 00:33:17.840 Robert Tseng: Ayah.
266 00:33:35.020 ⇒ 00:33:38.730 Robert Tseng: Okay, well, I guess that more or less gets closer.
267 00:34:16.219 ⇒ 00:34:21.029 Robert Tseng: Well, that still means that most people did not use this product.
268 00:34:26.090 ⇒ 00:34:33.530 Robert Tseng: Okay, that’s… That’s okay, I’ll do that. So…
269 00:34:36.659 ⇒ 00:34:38.340 Robert Tseng: New users…
270 00:34:58.640 ⇒ 00:35:00.750 Robert Tseng: first time API.
271 00:35:01.590 ⇒ 00:35:02.680 Robert Tseng: Reference…
272 00:35:09.230 ⇒ 00:35:10.080 Robert Tseng: Okay.
273 00:35:47.340 ⇒ 00:35:50.319 Robert Tseng: That’s really not helpful.
274 00:35:51.620 ⇒ 00:35:53.339 Robert Tseng: I just don’t think she’s right, then.
275 00:35:54.130 ⇒ 00:35:59.420 Robert Tseng: Oh, I add more filters here, that would be helpful.
276 00:36:01.410 ⇒ 00:36:04.610 Robert Tseng: Shouldn’t really be any active event, right?
277 00:36:04.820 ⇒ 00:36:05.660 Robert Tseng: Well…
278 00:36:14.290 ⇒ 00:36:16.520 Robert Tseng: Okay, well, they can’t be Enterprise.
279 00:36:38.130 ⇒ 00:36:40.460 Robert Tseng: Moves the number slightly.
280 00:36:41.920 ⇒ 00:36:43.449 Robert Tseng: Still not that helpful.
281 00:36:48.810 ⇒ 00:36:59.049 Robert Tseng: Okay, okay. Let’s see. So… References, and then guides. Okay, well, let’s just drill into… People that are using…
282 00:37:02.680 ⇒ 00:37:04.820 Robert Tseng: references, right?
283 00:37:05.830 ⇒ 00:37:16.120 Robert Tseng: If we apply the same thing here, And… Instead, I’m looking at… Same cohorts.
284 00:37:16.380 ⇒ 00:37:34.440 Robert Tseng: Any active events? Well, let’s do… Specifically… Project creation… And… Attempted launch. What’s that?
285 00:37:34.620 ⇒ 00:37:35.929 Robert Tseng: fair proxy.
286 00:37:37.930 ⇒ 00:37:49.280 Robert Tseng: Then… Not people who are… Okay, that use this feature are attempting to launch way more. That’s…
287 00:37:50.690 ⇒ 00:37:52.110 Robert Tseng: That makes sense to me.
288 00:37:53.200 ⇒ 00:37:54.570 Robert Tseng: 50%.
289 00:37:55.140 ⇒ 00:37:59.370 Robert Tseng: Okay, so… Project…
290 00:38:03.880 ⇒ 00:38:11.880 Robert Tseng: New project to… Attempted launch… Notion.
291 00:38:18.300 ⇒ 00:38:23.249 Robert Tseng: Great. So, we’re gonna have to do references and then guides.
292 00:39:08.390 ⇒ 00:39:10.150 Henry Zhao: Where did, Utsam and Ember go?
293 00:39:11.410 ⇒ 00:39:15.700 Robert Tseng: Oh, I don’t know. I think they probably had other things that they were doing.
294 00:39:23.570 ⇒ 00:39:33.579 Henry Zhao: Are you recording this? Because when you… when I do have time, I do kind of want to see your… your… how you do analysis and amplitude. I already saw the other 25-minute video you recorded, but it’s always good to.
295 00:39:33.580 ⇒ 00:39:37.000 Robert Tseng: But yeah, I mean, this is… this is recorded. This is how I do it.
296 00:39:43.930 ⇒ 00:39:46.760 Robert Tseng: Dang. I have that much data? Okay.
297 00:39:47.060 ⇒ 00:39:54.420 Robert Tseng: Well… So… Low volume of users.
298 00:39:55.770 ⇒ 00:40:03.010 Robert Tseng: reference users… Api definition… question marks.
299 00:40:06.990 ⇒ 00:40:14.539 Henry Zhao: Yeah, I think I need, like, a lot of practice on coming up with questions and analysis with, like, little to no
300 00:40:14.820 ⇒ 00:40:19.299 Henry Zhao: context slash, this is what we talked about earlier, right? It’s like…
301 00:40:19.300 ⇒ 00:40:19.890 Robert Tseng: Yeah.
302 00:40:20.380 ⇒ 00:40:22.770 Henry Zhao: Being able to explore and find questions on my own.
303 00:40:23.670 ⇒ 00:40:25.490 Robert Tseng: Yeah, that’s.
304 00:40:27.060 ⇒ 00:40:28.359 Robert Tseng: That’s what it is.
305 00:40:29.040 ⇒ 00:40:34.549 Henry Zhao: Yeah. I never thought I needed to develop this skill so badly until, like, README kind of exposed that.
306 00:40:36.550 ⇒ 00:40:41.760 Robert Tseng: That’s… it’s not really an exercise you would do in a more mature company, is all I would say.
307 00:40:42.170 ⇒ 00:40:43.479 Henry Zhao: Yeah, exactly.
308 00:40:43.480 ⇒ 00:40:44.310 Robert Tseng: Yeah.
309 00:40:46.250 ⇒ 00:40:50.830 Robert Tseng: So, up to you how much you actually want to learn it, but I don’t know.
310 00:40:54.890 ⇒ 00:41:05.590 Robert Tseng: A PPI definition form… Submission… References, and you say equals.
311 00:41:09.760 ⇒ 00:41:16.560 Robert Tseng: I’m just writing down, kind of, like, the assumptions that I’m making as I’m exploring these, so I can go back and add it to a notebook later.
312 00:41:18.210 ⇒ 00:41:18.790 Henry Zhao: Boom.
313 00:41:24.570 ⇒ 00:41:25.760 Robert Tseng: Huh.
314 00:41:26.830 ⇒ 00:41:33.979 Robert Tseng: Okay, I should rename these. So this is, new users that have,
315 00:41:52.090 ⇒ 00:41:57.410 Robert Tseng: As far as… New users…
316 00:42:10.280 ⇒ 00:42:11.219 Robert Tseng: What the heck?
317 00:42:15.490 ⇒ 00:42:17.280 Robert Tseng: Right, definition form.
318 00:42:42.960 ⇒ 00:42:52.810 Robert Tseng: This one is… New users… Use references and updates.
319 00:42:54.700 ⇒ 00:42:55.550 Robert Tseng: Great.
320 00:42:55.710 ⇒ 00:43:01.750 Robert Tseng: Okay, so… That’s… Basically, what this is.
321 00:43:02.000 ⇒ 00:43:02.730 Robert Tseng: Cool.
322 00:43:06.550 ⇒ 00:43:12.110 Robert Tseng: what other features they use to convert? Well, now it’s kind of a volume question. It’s like…
323 00:43:15.530 ⇒ 00:43:20.710 Robert Tseng: only 8 out of 77. It’s like 10% of users are actually…
324 00:43:21.330 ⇒ 00:43:24.819 Robert Tseng: Well, that’s if they’re attempting to launch, so that’s not fair to say.
325 00:43:25.800 ⇒ 00:43:28.000 Robert Tseng: So, I’ll keep those same segments.
326 00:43:29.790 ⇒ 00:43:34.360 Robert Tseng: I’ll talk about… Obviously, they’re creating projects.
327 00:43:35.450 ⇒ 00:43:38.190 Robert Tseng: And I’m gonna go blasts.
328 00:43:38.310 ⇒ 00:43:39.660 Robert Tseng: Links…
329 00:43:45.160 ⇒ 00:43:46.549 Robert Tseng: I look at dairy.
330 00:43:50.300 ⇒ 00:43:57.770 Robert Tseng: Makes sense… References and guides… Volume’s low.
331 00:43:58.250 ⇒ 00:44:00.930 Robert Tseng: I guess they’re not creating new projects.
332 00:44:02.160 ⇒ 00:44:07.520 Robert Tseng: So maybe… let’s do any active events.
333 00:44:11.150 ⇒ 00:44:15.690 Robert Tseng: No, there are just too many random… just too noisy.
334 00:44:25.600 ⇒ 00:44:30.100 Robert Tseng: So… References… nope?
335 00:44:31.700 ⇒ 00:44:33.400 Robert Tseng: References…
336 00:44:56.110 ⇒ 00:44:57.060 Robert Tseng: Oh.
337 00:44:58.350 ⇒ 00:45:00.689 Robert Tseng: Okay, that could be interesting.
338 00:45:09.500 ⇒ 00:45:10.590 Robert Tseng: Yeah.
339 00:45:10.820 ⇒ 00:45:11.540 Robert Tseng: Sure.
340 00:45:11.740 ⇒ 00:45:12.969 Robert Tseng: I can try that.
341 00:45:52.230 ⇒ 00:45:54.330 Robert Tseng: Daily is probably too much.
342 00:46:03.190 ⇒ 00:46:03.860 Robert Tseng: Oof.
343 00:46:04.340 ⇒ 00:46:06.630 Robert Tseng: Then I have to expand all these definitions.
344 00:46:07.630 ⇒ 00:46:11.659 Robert Tseng: Hmm… Does it seem that helpful?
345 00:46:13.300 ⇒ 00:46:17.150 Robert Tseng: I looked at it daily, on or after daily…
346 00:46:18.830 ⇒ 00:46:21.219 Robert Tseng: I mean, the retention’s so low anyway.
347 00:46:21.760 ⇒ 00:46:24.999 Robert Tseng: Sure, they’re retained more if they come back.
348 00:46:26.090 ⇒ 00:46:30.540 Robert Tseng: But… Where do they change over time?
349 00:46:31.860 ⇒ 00:46:43.609 Robert Tseng: Day 1, Day 7… Interesting. So… This is telling me… Why is it so volatile?
350 00:46:44.970 ⇒ 00:46:45.440 Robert Tseng: What?
351 00:46:45.440 ⇒ 00:46:50.619 Henry Zhao: I often see graphs like this, and I, like, have no idea how to proceed, like, it just seems…
352 00:46:51.030 ⇒ 00:46:55.120 Henry Zhao: Either no insights can be drawn, or too small of a sample size, you know?
353 00:46:55.470 ⇒ 00:46:56.140 Robert Tseng: Yeah.
354 00:46:56.510 ⇒ 00:46:58.019 Henry Zhao: You see that kind of graph all the time?
355 00:46:58.540 ⇒ 00:47:02.370 Henry Zhao: And that’s where, like, I was talking earlier that, like, SQL comes in handy, because I can…
356 00:47:02.770 ⇒ 00:47:07.870 Henry Zhao: Kind of group by, or maybe, like, see breakdowns a little bit more easily.
357 00:47:11.390 ⇒ 00:47:12.200 Robert Tseng: Yeah.
358 00:47:12.200 ⇒ 00:47:14.130 Henry Zhao: And so I kind of just get stuck.
359 00:47:19.490 ⇒ 00:47:20.790 Robert Tseng: So…
360 00:47:29.190 ⇒ 00:47:32.250 Robert Tseng: Okay, yeah, this is… there’s too much going on here.
361 00:47:32.810 ⇒ 00:47:36.549 Robert Tseng: I don’t care about 1 day. I’ll do 7-day…
362 00:47:37.290 ⇒ 00:47:39.180 Robert Tseng: Just look at all the 7 days.
363 00:47:41.880 ⇒ 00:47:46.940 Henry Zhao: How would you manually do this if there was, like, Way more rows.
364 00:47:48.060 ⇒ 00:47:52.449 Henry Zhao: And I know there’s Export CSV here, but I think in Mixpanel, for example, there’s no export CSV.
365 00:47:52.780 ⇒ 00:47:53.510 Robert Tseng: Yeah.
366 00:47:54.900 ⇒ 00:47:59.959 Robert Tseng: I mean, you just gotta limit your search, yeah.
367 00:47:59.960 ⇒ 00:48:01.020 Henry Zhao: Biggest ones, yeah.
368 00:48:01.550 ⇒ 00:48:06.179 Robert Tseng: Yeah. I mean, I’m just, like, intuitively thinking about, well, I don’t really care about, like.
369 00:48:08.240 ⇒ 00:48:15.279 Robert Tseng: day, like, one day. 7 day, I don’t… there’s not… there’s not enough interesting data there. Probably 14 days.
370 00:48:15.650 ⇒ 00:48:20.810 Robert Tseng: more realistic. I’ll take a look at 14. I’m assuming the names will be higher.
371 00:48:21.750 ⇒ 00:48:29.119 Robert Tseng: No, this is still… What the heck is going on here?
372 00:48:29.260 ⇒ 00:48:31.569 Robert Tseng: Why is it not by 7-day windows?
373 00:48:31.800 ⇒ 00:48:34.219 Robert Tseng: Oh, it’s a sale by daily.
374 00:48:35.860 ⇒ 00:48:36.830 Robert Tseng: Oof.
375 00:48:40.750 ⇒ 00:48:44.250 Robert Tseng: I might have to just create another cohort.
376 00:48:56.940 ⇒ 00:49:00.609 Robert Tseng: Product bunches… The last 6 months.
377 00:49:03.130 ⇒ 00:49:06.200 Robert Tseng: Okay, I’m… I’m okay with that.
378 00:49:08.510 ⇒ 00:49:12.360 Robert Tseng: Okay, I guess amplitude’s not okay with that. Let’s see…
379 00:49:18.260 ⇒ 00:49:25.520 Robert Tseng: Oh, well… That’s just silly. I’m deleting your cohort, dude.
380 00:49:26.290 ⇒ 00:49:27.130 Robert Tseng: Alright.
381 00:49:29.430 ⇒ 00:49:38.430 Robert Tseng: Obviously, we’re working within constraints here. I only have the ability to make 5 cohorts, unless they bump up to a new plan. I think it’s fine, no one’s really looking at this.
382 00:49:39.250 ⇒ 00:49:43.260 Robert Tseng: Except for me, so…
383 00:49:44.420 ⇒ 00:49:51.670 Robert Tseng: Yeah, I still need an ability to limit new users. 30-day, I’ll do 90-day new users, that’s fine.
384 00:49:52.620 ⇒ 00:50:04.100 Robert Tseng: I can… Copying this cohort… I’m gonna do… Yeah.
385 00:50:06.230 ⇒ 00:50:10.700 Robert Tseng: Let’s see… 90…
386 00:50:42.070 ⇒ 00:50:45.430 Robert Tseng: So, 90 days, and…
387 00:50:48.700 ⇒ 00:50:54.389 Robert Tseng: Yeah, they could have done this. I’ll just… I’ll just expand the window 90 days for everything.
388 00:51:09.240 ⇒ 00:51:10.120 Robert Tseng: Oops.
389 00:51:10.600 ⇒ 00:51:12.370 Robert Tseng: Missed this one.
390 00:51:15.080 ⇒ 00:51:17.229 Robert Tseng: Okay, much more interesting.
391 00:51:18.640 ⇒ 00:51:25.480 Robert Tseng: So, yeah, I could do a weekly…
392 00:51:26.030 ⇒ 00:51:28.449 Robert Tseng: Let’s look at all the weak ones…
393 00:51:34.040 ⇒ 00:51:38.320 Robert Tseng: Okay, not much difference there. Maybe the week twos…
394 00:51:43.180 ⇒ 00:51:46.549 Robert Tseng: Starting to see some separation.
395 00:51:49.280 ⇒ 00:51:50.440 Robert Tseng: That’s so…
396 00:51:55.190 ⇒ 00:51:57.919 Robert Tseng: New users that use the reference.
397 00:51:58.520 ⇒ 00:52:03.850 Robert Tseng: There’s not… it’s not a clear winner here. It’s just, like, it’s all about the same.
398 00:52:06.210 ⇒ 00:52:08.700 Robert Tseng: Week 1, retention.
399 00:52:10.320 ⇒ 00:52:12.640 Robert Tseng: Does that tell a more interesting story?
400 00:52:17.270 ⇒ 00:52:19.550 Robert Tseng: No, it’s not the same.
401 00:52:20.830 ⇒ 00:52:26.779 Robert Tseng: Okay, I’ll do a week one, and I’ll do a week 4. I think that should be… that would be fine.
402 00:52:29.640 ⇒ 00:52:38.160 Robert Tseng: Okay, so what I’m saying here is… Dude, yeah, these are…
403 00:52:38.390 ⇒ 00:52:44.099 Robert Tseng: doesn’t really make a difference. I don’t think this is really a true activation. It’s not like it’s better than the average.
404 00:52:44.380 ⇒ 00:52:46.599 Robert Tseng: Yeah, I think she’s wrong. Okay.
405 00:52:47.130 ⇒ 00:52:48.940 Robert Tseng: That could just be the takeaway here.
406 00:52:49.570 ⇒ 00:52:54.710 Robert Tseng: Huh. I’m gonna go back.
407 00:52:55.010 ⇒ 00:53:02.140 Robert Tseng: So… Did you use reference guides, or da-da-da.
408 00:53:12.550 ⇒ 00:53:21.710 Robert Tseng: Yeah, what I can say is… preferences… And guide usage…
409 00:53:25.950 ⇒ 00:53:26.959 Robert Tseng: That was it.
410 00:53:28.160 ⇒ 00:53:34.300 Robert Tseng: Significantly… Drive… more paid.
411 00:53:38.900 ⇒ 00:53:40.130 Robert Tseng: Conversions.
412 00:53:40.410 ⇒ 00:53:49.519 Robert Tseng: That’s… my position… I’ll say this is week 1 and Week 4.
413 00:53:50.260 ⇒ 00:53:58.769 Robert Tseng: Week 1 and 4 retention… Or… Pride and references.
414 00:54:01.020 ⇒ 00:54:03.720 Robert Tseng: References, users…
415 00:54:15.540 ⇒ 00:54:21.319 Robert Tseng: But I will just run that by… Subscription success…
416 00:54:22.920 ⇒ 00:54:26.250 Robert Tseng: Ugh, it’s a very different story now.
417 00:54:27.350 ⇒ 00:54:33.290 Robert Tseng: I think the volume’s low, though, so I doubt this will be that significant.
418 00:54:34.970 ⇒ 00:54:36.350 Robert Tseng: Still…
419 00:54:40.690 ⇒ 00:54:41.690 Robert Tseng: Yeah.
420 00:54:47.970 ⇒ 00:54:52.240 Robert Tseng: Actually… I should change this definition.
421 00:54:52.440 ⇒ 00:54:58.719 Robert Tseng: don’t really care about any active event. They need to have created a project other in the past 90 days.
422 00:55:00.030 ⇒ 00:55:04.529 Robert Tseng: new user that has created a project. Yeah, that’s what I should be.
423 00:55:05.930 ⇒ 00:55:11.260 Robert Tseng: If they haven’t created a project, it doesn’t really matter what they do. They’re probably just not a real user. Okay.
424 00:55:11.790 ⇒ 00:55:15.420 Robert Tseng: I should update those cohorts.
425 00:55:18.500 ⇒ 00:55:31.930 Robert Tseng: cohort, or… New users need to have created a project… project in past 3… Okay, sure.
426 00:55:37.770 ⇒ 00:55:39.450 Robert Tseng: And…
427 00:55:42.600 ⇒ 00:55:51.500 Robert Tseng: Who had been new, And… Project created. Yes.
428 00:55:52.080 ⇒ 00:55:53.530 Robert Tseng: In the past.
429 00:55:54.210 ⇒ 00:55:56.460 Robert Tseng: 90 days.
430 00:55:57.740 ⇒ 00:55:58.480 Robert Tseng: Oops.
431 00:56:00.620 ⇒ 00:56:02.559 Robert Tseng: Same thing with this one.
432 00:56:08.830 ⇒ 00:56:09.990 Robert Tseng: Oops.
433 00:56:13.540 ⇒ 00:56:19.150 Robert Tseng: Okay, let’s see…
434 00:56:20.630 ⇒ 00:56:23.130 Robert Tseng: Does that change my story on the other one?
435 00:56:23.330 ⇒ 00:56:25.459 Robert Tseng: Oops, not the right one.
436 00:56:27.320 ⇒ 00:56:31.259 Robert Tseng: No. Doesn’t really change the story. Slight difference.
437 00:56:32.510 ⇒ 00:56:35.110 Robert Tseng: Sure. Use references.
438 00:56:36.830 ⇒ 00:56:44.330 Robert Tseng: But yeah, references and guides, not really anything that significant. But cool. This is a big one.
439 00:56:47.830 ⇒ 00:56:51.439 Henry Zhao: For each of these, do you think you need to test for, like, statistical significance?
440 00:56:51.660 ⇒ 00:56:57.149 Henry Zhao: Like, where do we come to a point where we figure out whether we’re cherry-picking data, or we’re actually…
441 00:56:57.440 ⇒ 00:57:00.320 Henry Zhao: finding things that are statistically significant, you know what I mean?
442 00:57:01.280 ⇒ 00:57:07.189 Robert Tseng: Yeah, well, I would say, okay, let’s say we draw this conclusion, they buy my argument that
443 00:57:07.560 ⇒ 00:57:13.180 Robert Tseng: Okay. New users that have used references are converting at a higher clip than their average.
444 00:57:13.520 ⇒ 00:57:18.279 Robert Tseng: well… then I think we can run a statistically significant test.
445 00:57:18.280 ⇒ 00:57:18.880 Henry Zhao: Okay.
446 00:57:19.140 ⇒ 00:57:24.510 Robert Tseng: Against that cohort. But, like, we would have to measure… we have to track them over time for longer.
447 00:57:24.510 ⇒ 00:57:32.590 Henry Zhao: Yeah, so these… so basically these notes you’re putting are basically, like, data-backed hypotheses that we can then go and test what we think are the strongest bets, basically.
448 00:57:32.990 ⇒ 00:57:33.750 Robert Tseng: Correct.
449 00:57:33.750 ⇒ 00:57:35.090 Henry Zhao: Okay, that makes sense.
450 00:57:35.470 ⇒ 00:57:36.100 Robert Tseng: Yeah.
451 00:57:36.900 ⇒ 00:57:42.640 Robert Tseng: Which is cool, because then you have different levers, like, I can make recommendations to the head of product, and I could say, okay, great, like.
452 00:57:42.790 ⇒ 00:57:56.800 Robert Tseng: you know, there aren’t that many users here. Maybe, like, I think I… you weren’t… you maybe were on the call, but what I realized is, okay, 10% of active users are using this references guide. She’s saying this is a core activation feature in their product.
453 00:57:56.870 ⇒ 00:58:13.989 Robert Tseng: And she thinks that, you know, if users aren’t using this feature, I don’t know why they’re using README at all. So, okay, well, if that narrows down your audience, that only 10% of them are doing it, then yes, I would… I would expect to see that they need to be converting at a higher clip, which they are. Volume may be low.
454 00:58:14.080 ⇒ 00:58:21.339 Robert Tseng: some volatility, seems like it’s like, huh, this purchasing cycle is interesting. It looks like it’s, like, on a monthly cadence, so…
455 00:58:21.630 ⇒ 00:58:22.340 Robert Tseng: whoever is
456 00:58:22.760 ⇒ 00:58:34.240 Robert Tseng: a new user there. There, like, this is something to note. Maybe it’s not just about behavior, there’s also a billing cycle that we need to keep in mind. And so, I think we can brainstorm, like, what are the different
457 00:58:34.430 ⇒ 00:58:39.180 Robert Tseng: things that we would test, knowing that, okay, seems like early to mid
458 00:58:39.460 ⇒ 00:58:56.139 Robert Tseng: week of the month, if we run, like, a promotion against users, and we just… and we only expose new users to the references product to test this hypothesis, you know, we can let that run long enough so that there is this… we can run… have a statistically significant test that.
459 00:58:56.540 ⇒ 00:59:04.620 Robert Tseng: okay, we’re… yeah, we’re now able to convert new users at a, you know, at a higher rate if we’re able to prove that, right? So I’m just, like, kind of…
460 00:59:04.950 ⇒ 00:59:12.899 Robert Tseng: going… yeah, like, that’s how I’m interpreting what I’m reading on the chart, to making a recommendation to the head of product on what she should actually run as her experiment.
461 00:59:12.900 ⇒ 00:59:15.390 Henry Zhao: Oh, okay, makes sense. Yeah, that makes a lot of sense.
462 00:59:16.060 ⇒ 00:59:29.360 Robert Tseng: Yeah. So, I mean, I know that it’s not, like, super data-backed, like, you guys can tell, there’s not that much data, like, I think it’s… but I think that’s… if that turns out, that could… for a startup, like, that could be a…
463 00:59:29.990 ⇒ 00:59:33.180 Robert Tseng: That could be a very significant, like.
464 00:59:33.370 ⇒ 00:59:35.999 Robert Tseng: tests that they would run, you know? Yeah. And
465 00:59:36.140 ⇒ 00:59:42.520 Robert Tseng: Yeah, so… and it didn’t take that much effort. I just had to, like, look at some stuff and try to make some assumptions.
466 00:59:42.520 ⇒ 00:59:46.560 Henry Zhao: Yeah, and I think you probably need to over… not overthink things so much in the, like, discovery phase.
467 00:59:46.950 ⇒ 00:59:47.550 Robert Tseng: Yeah.
468 00:59:47.550 ⇒ 00:59:48.130 Henry Zhao: Yes.
469 00:59:50.270 ⇒ 00:59:59.459 Robert Tseng: Yeah, I mean, they’re looking at this and think it’s spaghetti as well, they don’t really know what the heck is going on. So, I think anything that you can pick up that I think is… is worth… that’s worth calling out, you can start to tell that story.
470 00:59:59.460 ⇒ 01:00:00.609 Henry Zhao: Okay.
471 01:00:00.610 ⇒ 01:00:01.170 Robert Tseng: Yeah.
472 01:00:03.530 ⇒ 01:00:15.000 Robert Tseng: Okay, cool. So, I think that’s, that’s worth, kind of… that’s packaging up, and that’s all the time I got for today, so I’m gonna pretty much wrap this up and shoot that over.
473 01:00:15.620 ⇒ 01:00:16.330 Robert Tseng: Right.
474 01:00:16.330 ⇒ 01:00:18.070 Henry Zhao: Thank you so much, this was really helpful.
475 01:00:18.470 ⇒ 01:00:20.440 Robert Tseng: Yep. Alright, see you guys.
476 01:00:20.440 ⇒ 01:00:21.230 Henry Zhao: See you guys.