Meeting Title: ReadMe <> Brainforge Check-In Date: 2025-10-23 Meeting participants: Parliament Conference Room, Robert Tseng, Ashley Samay, Henry Zhao, Uttam Kumaran, Alicia Shin
WEBVTT
1 00:00:22.840 ⇒ 00:00:33.060 Parliament Conference Room: How does this work? No? TV? Oh, weird. It must have been the wrong input. Go to, go all the way left to inputs. Yeah.
2 00:00:35.190 ⇒ 00:00:36.590 Parliament Conference Room: She might get a baby.
3 00:00:39.450 ⇒ 00:00:40.800 Parliament Conference Room: It’s good to speak.
4 00:00:41.060 ⇒ 00:00:43.160 Parliament Conference Room: I just saw a face, right?
5 00:00:43.710 ⇒ 00:00:45.000 Robert Tseng: I’m here. Hello.
6 00:00:45.000 ⇒ 00:00:47.420 Parliament Conference Room: Oh, oh my god.
7 00:00:47.950 ⇒ 00:00:48.580 Parliament Conference Room: Hello!
8 00:00:48.580 ⇒ 00:00:52.560 Robert Tseng: I can’t see you, but I… I saw… .
9 00:00:52.560 ⇒ 00:00:53.759 Parliament Conference Room: We’ll see us momentarily.
10 00:00:53.760 ⇒ 00:00:54.460 Robert Tseng: Yeah.
11 00:00:55.030 ⇒ 00:00:57.139 Robert Tseng: Oh, okay, there you are.
12 00:00:57.410 ⇒ 00:01:05.110 Parliament Conference Room: We were, like, staring at an Apple TV configuration, and we were like, where’s… where’s our Zoom?
13 00:01:05.660 ⇒ 00:01:08.879 Parliament Conference Room: It feels nice having the heat on. Yeah. It’s cold out there.
14 00:01:08.880 ⇒ 00:01:10.809 Robert Tseng: Oh, is it cold? Okay.
15 00:01:11.280 ⇒ 00:01:13.850 Parliament Conference Room: Yeah. How are you, Robert? It’s been a while.
16 00:01:13.850 ⇒ 00:01:16.190 Robert Tseng: Yeah, one time. I’m doing well.
17 00:01:18.340 ⇒ 00:01:24.399 Robert Tseng: I guess, you’ve been pulled in… pulled into other things, so, I’ve been mostly talking to Alicia, yeah.
18 00:01:24.560 ⇒ 00:01:31.389 Parliament Conference Room: Yes, Alicia’s just better at getting things done than I am, so I figured she could get it further along than I could.
19 00:01:32.740 ⇒ 00:01:34.200 Robert Tseng: Hi, Ashley, good to meet you.
20 00:01:34.200 ⇒ 00:01:36.209 Ashley Samay: Hi, so nice to meet you!
21 00:01:37.790 ⇒ 00:01:41.209 Ashley Samay: a lot of work for README behind the scenes. Oh, yeah. I appreciate it.
22 00:01:41.210 ⇒ 00:01:51.010 Robert Tseng: We’re… we’re excited, we love, we love the Readbee team, and I don’t think… this is also Henry on my team, I don’t think Phoebe’s been him either. He can kind of… he’s been…
23 00:01:51.310 ⇒ 00:01:55.499 Robert Tseng: We put… we brought him into the… this project the past… past couple weeks, so…
24 00:01:56.050 ⇒ 00:01:57.589 Parliament Conference Room: Cool. Nice to meet you, Henry.
25 00:01:57.590 ⇒ 00:01:58.489 Henry Zhao: Nice to meet you guys.
26 00:01:59.670 ⇒ 00:02:10.850 Parliament Conference Room: Robert, so I sent the note, of Ashley and Phoebe here. I think it would be great to walk through the paid conversions dashboard, and the voiceover, I think, would be helpful.
27 00:02:11.039 ⇒ 00:02:11.619 Robert Tseng: Okay.
28 00:02:11.620 ⇒ 00:02:19.740 Parliament Conference Room: I think this group here can possibly help us start unblocking some of the challenges we might be facing and making some decisions to make this even more actionable.
29 00:02:19.740 ⇒ 00:02:20.530 Robert Tseng: Yep.
30 00:02:20.530 ⇒ 00:02:33.570 Parliament Conference Room: And then my ask for Ashley, and I guess Phoebe as well, is just, as you’re walking through, if there are any, like, metrics that are really helpful, please call it out, and then if there’s anything that you think maybe you can deprioritize for now, that’s also helpful, so we know where to focus the energy.
31 00:02:34.690 ⇒ 00:02:35.500 Ashley Samay: Awesome.
32 00:02:35.880 ⇒ 00:02:41.660 Robert Tseng: Okay, that sounds good. Yeah, so I’m going to share my screen. Yeah, sorry, I didn’t send out an agenda earlier, so kind of left to you.
33 00:02:42.180 ⇒ 00:02:56.169 Robert Tseng: hanging, but, yeah, we’ll kind of walk through the reports that we already have. We also put up a deck, so I’ll kind of have them both side by side. I’ll send it out after the call. I don’t think we’re going to get through everything in this call.
34 00:02:56.430 ⇒ 00:03:00.919 Robert Tseng: But yeah, I guess we’ll start with kind of where Alicia’s pointing us.
35 00:03:01.260 ⇒ 00:03:04.110 Robert Tseng: So, I guess…
36 00:03:04.300 ⇒ 00:03:08.500 Robert Tseng: Sorry, I have a wide screen. Let me know if I need to zoom in anywhere. I’ll just try to…
37 00:03:08.730 ⇒ 00:03:12.999 Robert Tseng: do that before… maybe that’s all I can… best I can do.
38 00:03:14.530 ⇒ 00:03:20.999 Robert Tseng: So yeah, I guess this is a pay conversions dashboard. Everything that we’ve done is in the self-service conversion space.
39 00:03:21.380 ⇒ 00:03:37.859 Robert Tseng: And so, kind of, Phoebe may recognize this, and Lisa, this is kind of… we’re continuing to build on this, but this is just, like, the paid conversion funnel, right? So, the user signs up, they create a project, we were trying to… in the building cycle, there’s, like, a little bit of…
40 00:03:37.860 ⇒ 00:03:52.600 Robert Tseng: ambiguity on, like, what selection of events to use. For now, we’re still using attempted launch as the proxy for intent to purchase, and then there’s, like, a valid subscription, right? So this is the data that’s tracked directly in Amplitude using the SDK.
41 00:03:52.940 ⇒ 00:04:10.220 Robert Tseng: Recently, we got access to Mongo and AWS, and so Henry spent some time with a couple of engineers last week, trying to, like, get to the source of truth. Like, do these numbers, like, how do we trust them? Obviously, you may not, like, the time period here, we can change the default, maybe we just do, like.
42 00:04:10.330 ⇒ 00:04:13.020 Robert Tseng: Past 8 weeks, and it’s kind of filtered by weekly.
43 00:04:13.020 ⇒ 00:04:35.280 Robert Tseng: But what I will call out here is that, when we’re validating signups, like, this is something we have a higher degree of confidence, and we know how to get this data. It doesn’t match up exactly with Amplitude yet, so I think that’s, I want to be able to assign, kind of, like, confidence levels to each of these soon, but there’s just still a little bit of, digging to do, because the data is structured differently in Mongo versus Amplitude.
44 00:04:35.930 ⇒ 00:04:41.339 Robert Tseng: for example, I mean, project creation’s pretty straightforward, I think this number will match, but, like.
45 00:04:42.110 ⇒ 00:05:00.410 Robert Tseng: plan type, for example, in Mongo can be overridden, and, it’s kind of stored as an object, whereas, like, in, you know, whereas in amplitude, there are multiple, you know, plan changes happen, and it’s, like, that’s one, you know, that’s one area that we’re still kind of digging into and have more questions.
46 00:05:00.410 ⇒ 00:05:12.510 Robert Tseng: questions around. So, long story short, yeah, this is the way that we’ve structured the kind of core conversion funnel, but the data validation piece of each of these different steps is still kind of like an ongoing
47 00:05:12.940 ⇒ 00:05:25.740 Robert Tseng: kind of discussion. So, I’ll just pause there, kind of see, you know, any questions here. Are we thinking about this the right way from your perspective? Yeah, and then if you have any questions about other constraints, happy to talk about that.
48 00:05:26.820 ⇒ 00:05:46.390 Ashley Samay: This is awesome, I always have questions. So, just to make sure I’m on the same page. So one thing is, like, I used to want to use this in Amplitude, and then it was always, like, working with marketing or other folks, the numbers were different, and so it was, like, I just stopped using it. So that makes sense that you’re trying to confirm it, but can you high level, like, help me understand
49 00:05:46.390 ⇒ 00:05:53.439 Ashley Samay: Why would it be different? These are server events, right? Which means, like, this is actually happening when it’s in the database, this is not a front-end event, what…
50 00:05:53.440 ⇒ 00:05:54.779 Robert Tseng: You’re actually not, yeah.
51 00:05:54.960 ⇒ 00:05:58.429 Robert Tseng: Oh, it’s not… Not all of them are, yeah. Okay. That’s the problem, yeah.
52 00:05:58.430 ⇒ 00:06:06.739 Ashley Samay: Got it. So on this funnel right now, which ones are server events? Like, Subscription Success, I’m assuming, is a server-side event, and Project Created, maybe?
53 00:06:06.970 ⇒ 00:06:13.960 Robert Tseng: Yeah, so… Project Created is… subscription success is actually not.
54 00:06:14.280 ⇒ 00:06:17.730 Robert Tseng: And attempt to launch, obviously it’s not, it’s just clicking that button. Yeah.
55 00:06:17.730 ⇒ 00:06:22.800 Ashley Samay: Launch makes sense. Yeah. I thought we did have a server-side event for subscription success.
56 00:06:24.280 ⇒ 00:06:26.080 Ashley Samay: like, when we get the Stripe ID back.
57 00:06:26.530 ⇒ 00:06:27.370 Ashley Samay: like…
58 00:06:29.270 ⇒ 00:06:38.240 Robert Tseng: Well, Stripe doesn’t feed… it’s not directly connected to this, so I mean, yeah, however we’re triggering that, at least one of the times that I’ve run through the flow myself, and I put in my own
59 00:06:38.860 ⇒ 00:06:43.229 Robert Tseng: Payment processor, or, like, my own credit card.
60 00:06:44.210 ⇒ 00:07:00.349 Robert Tseng: yeah, like, I am able to get this event to fire when I, like, change the different plan types. I mean, it’s… I… I guess, Henry, this is kind of just, I’m not entirely sure, like, if this matches up, like, this is one of the ones that we’re checking, right? So, like, I don’t know if you have anything else to say about this one.
61 00:07:00.540 ⇒ 00:07:08.679 Parliament Conference Room: Well, Ashley, probably there’s, like, a, subscription created, like, the ID goes for… it’s, like, binary, right? Like, there was no ID, and now there is.
62 00:07:08.840 ⇒ 00:07:14.159 Parliament Conference Room: For subscriptions, right? Like, that might be another thing that Henry and Robert can check out as…
63 00:07:14.350 ⇒ 00:07:16.610 Parliament Conference Room: Proxy subscription success.
64 00:07:16.890 ⇒ 00:07:23.299 Robert Tseng: Yeah, well, regardless of what it is, there is no Stripe ID that’s being passed as a parameter right here. So, like, there are, like.
65 00:07:23.300 ⇒ 00:07:23.700 Ashley Samay: Yeah, yeah.
66 00:07:23.700 ⇒ 00:07:29.289 Robert Tseng: like, the fact that there is no transaction data popping up in Amplitude just makes me, like, question, is this really even a…
67 00:07:29.290 ⇒ 00:07:33.339 Ashley Samay: Okay. Yeah, I’m also, just so, to be clear, I’m not, like.
68 00:07:33.730 ⇒ 00:07:41.129 Ashley Samay: pushing on you to do it a certain way. I just know that this has been fraught with, like, the data doesn’t match, so…
69 00:07:41.130 ⇒ 00:07:41.530 Robert Tseng: Yeah.
70 00:07:41.530 ⇒ 00:07:48.730 Ashley Samay: It’s helpful to create a server-side event, like, we can, but if you’re, like, you do the math, and you’re like, no, we can trust this, that’s also totally fine for me.
71 00:07:49.040 ⇒ 00:08:01.810 Robert Tseng: Yeah, so I guess to that point, we will validate that, like, what can we trust? Right now, like, I do think it’s best practice to take some of these, like, core features and send them server, which is kind of maybe something I’ve brought up, an idea brought up.
72 00:08:01.810 ⇒ 00:08:19.999 Robert Tseng: a while ago, so not everything has to be sent from the server, but obviously things like payment events, especially since they’re tied to revenue, we want those directly, kind of, like, coming from Stripe, and whatever we can get from that, which I think is best practice. So I do think that that is one engineering, you know, kind of related task that we will have to
73 00:08:20.000 ⇒ 00:08:21.870 Robert Tseng: We’ll have to put up a roadmap.
74 00:08:21.870 ⇒ 00:08:29.180 Henry Zhao: Yeah, we do have that in MongoDB. But who’s the engineer that’s responsible for setting up the events in Amplitude? Is it Mark? Is it Bill?
75 00:08:30.780 ⇒ 00:08:35.400 Ashley Samay: That is a fantastic question, and I will have to get back to you. It probably is, like, a rotating.
76 00:08:35.409 ⇒ 00:08:35.739 Parliament Conference Room: I have.
77 00:08:35.740 ⇒ 00:08:37.760 Henry Zhao: characters. I can work directly with them.
78 00:08:37.760 ⇒ 00:08:43.510 Parliament Conference Room: It was, Ashley, it was Falto on Lyra did our initial, stuff.
79 00:08:43.669 ⇒ 00:08:49.829 Ashley Samay: Okay, cool. The person who did it, like, a year ago is not at README, it was Brendan.
80 00:08:50.229 ⇒ 00:08:53.609 Ashley Samay: But… Yeah, Falco would be a good option.
81 00:08:54.910 ⇒ 00:08:55.700 Robert Tseng: Okay.
82 00:08:55.780 ⇒ 00:08:57.590 Henry Zhao: I can work directly with them, yeah.
83 00:08:58.000 ⇒ 00:08:59.130 Robert Tseng: Yeah, I can’t…
84 00:08:59.720 ⇒ 00:09:01.130 Ashley Samay: Another question, too.
85 00:09:01.130 ⇒ 00:09:01.820 Robert Tseng: Okay.
86 00:09:01.820 ⇒ 00:09:06.899 Ashley Samay: Is that… you hit the launch button, or that’s… you hit one of the plans, and the Stripe model comes up?
87 00:09:06.900 ⇒ 00:09:11.540 Robert Tseng: No, you hit the launch button. So after you hit the launch button,
88 00:09:12.000 ⇒ 00:09:16.649 Robert Tseng: Here, this is just another notebook where we go into everything a little deeply.
89 00:09:17.360 ⇒ 00:09:28.519 Robert Tseng: Yeah, after you hit the launch button, you view the managed plan, and then you can change your plan, and then there’s supposed to be this subscription created event, or whatever that is, and that this exists.
90 00:09:29.000 ⇒ 00:09:36.239 Robert Tseng: So, like, there is, like, the whole, like, flow to the whole payment, or, like, the billing… billing process. Yeah, we just…
91 00:09:36.240 ⇒ 00:09:36.590 Ashley Samay: Okay.
92 00:09:36.590 ⇒ 00:09:37.980 Robert Tseng: There’s the first one for now, yeah.
93 00:09:37.980 ⇒ 00:09:51.499 Ashley Samay: So, attempted launches, I hit launch button, manage plan views, like, I actually hit the page, so some people, like, don’t actually see the page, and then if you’re on the page, manage plan changes, I’ve hit use plan for either startup, business, free.
94 00:09:51.500 ⇒ 00:09:52.170 Robert Tseng: Yep.
95 00:09:52.170 ⇒ 00:09:52.780 Ashley Samay: Okay.
96 00:09:53.460 ⇒ 00:09:54.859 Ashley Samay: Okay, cool, this makes sense.
97 00:09:55.400 ⇒ 00:09:57.719 Robert Tseng: Exactly, yeah.
98 00:09:58.200 ⇒ 00:10:14.379 Robert Tseng: Yeah, so I mean, just kind of lightly touching on this, so we did kind of try to start to map, like, feature usage to, like, subscription conversions, and like, you know, like, what features are people using? I mean, I guess what we were say… kind of, yeah, what features…
99 00:10:14.490 ⇒ 00:10:30.589 Robert Tseng: that when used, they, you know, have… they have a higher, kind of conversion rate, although the limitation with this right now is that these are all kind of individual, features that were used. And by feature, I just mean, like, we looked at a bunch of different, kind of, README workflows and picked some of, like.
100 00:10:30.690 ⇒ 00:10:41.250 Robert Tseng: you know, API definitions, that whole, like, setting that up, when you save the definition, that’s, like, the completion of the workflow. So there was some, like, kind of deterministic kind of thinking around, like, how we determined, which
101 00:10:41.250 ⇒ 00:10:52.459 Robert Tseng: like, what considers, like, what’s a properly used feature. But still, we’re kind of just looking at them all in isolation, and I think the next iteration of this needs to involve grouping,
102 00:10:52.550 ⇒ 00:11:00.929 Robert Tseng: like, different sets of features, right? Because nobody’s really using features in isolation, and I don’t think with this view we have a good sense of, like.
103 00:11:00.960 ⇒ 00:11:17.490 Robert Tseng: what are, like, the best combinations of features that are highest… that users are using are… that, correlate with, you know, better revenue outcomes. So, that’s kind of, like, the next phase of, like, kind of where I think this should… this should be going.
104 00:11:17.860 ⇒ 00:11:27.790 Ashley Samay: How do I read this right now? So, it’s 3.57% for recipes, and then there’s that one. Does that mean, like, out of all the people who used recipes, 1 converted?
105 00:11:28.290 ⇒ 00:11:28.970 Robert Tseng: Yes.
106 00:11:28.970 ⇒ 00:11:29.550 Ashley Samay: Okay.
107 00:11:29.860 ⇒ 00:11:30.440 Robert Tseng: Yeah.
108 00:11:31.380 ⇒ 00:11:35.910 Parliament Conference Room: And Pat, Ashley, Pat had asked that we do this a couple months back.
109 00:11:36.120 ⇒ 00:11:42.319 Parliament Conference Room: Of, like, what is the cocktail features that best lends itself to conversion?
110 00:11:42.540 ⇒ 00:11:45.160 Ashley Samay: Yeah, that makes sense. I’d say that’s pretty typical.
111 00:11:45.560 ⇒ 00:11:47.699 Ashley Samay: I’m not sure how we would use it yet, but it’s basically.
112 00:11:47.700 ⇒ 00:11:48.540 Parliament Conference Room: Yeah.
113 00:11:48.540 ⇒ 00:11:52.460 Ashley Samay: you know, what features match your conversion, then you try to get those people to use it.
114 00:11:52.460 ⇒ 00:11:52.800 Parliament Conference Room: aggressive.
115 00:11:52.800 ⇒ 00:11:55.830 Ashley Samay: I’ve seen, like, if they use it within the first day, that’s, like.
116 00:11:55.830 ⇒ 00:11:56.190 Parliament Conference Room: Yeah.
117 00:11:56.190 ⇒ 00:11:58.020 Ashley Samay: the most important thing. The thing.
118 00:11:58.020 ⇒ 00:12:11.559 Parliament Conference Room: And that was, like, that was… that was our takeaway with, uploading… uploading an OAS file, which is, like, one of the early projects Robert did, of, like, people who upload an OAS file on onboarding are, like, you know, X more likely to convert. It’s, like.
119 00:12:12.010 ⇒ 00:12:15.039 Parliament Conference Room: You know, trying to replicate things like that.
120 00:12:15.780 ⇒ 00:12:25.450 Ashley Samay: It doesn’t seem like that held true, right? If you, like, go to what you were just showing, that if you’ve, like, uploaded a reference, it was basically the same as if you had, updated a guide.
121 00:12:26.340 ⇒ 00:12:29.550 Robert Tseng: This isn’t the same as the OAS file.
122 00:12:29.990 ⇒ 00:12:30.690 Robert Tseng: Whoa.
123 00:12:32.130 ⇒ 00:12:34.779 Ashley Samay: What is API definition saved pull from?
124 00:12:35.030 ⇒ 00:12:39.409 Robert Tseng: Well, I guess it is…
125 00:12:39.410 ⇒ 00:12:43.459 Ashley Samay: That would include whether you did it on the onboarding or if you did it in the app, right?
126 00:12:44.070 ⇒ 00:12:48.509 Robert Tseng: It’s… it’s not the onboarding event. I think the onboarding event is just labeled something different. I think it’s, like…
127 00:12:48.770 ⇒ 00:13:02.740 Ashley Samay: No, no, sorry, I know it’s not that event, but I just mean, like, this is now looking, like, overall, if you uploaded a reference or not, only 5% of those people, which is actually kind of shocking, only 5% of people who upload a reference.
128 00:13:02.960 ⇒ 00:13:04.380 Robert Tseng: Okay, yeah, I see what you mean.
129 00:13:04.380 ⇒ 00:13:06.099 Ashley Samay: Right? Yes. Is that what this says?
130 00:13:06.300 ⇒ 00:13:07.120 Robert Tseng: That, yeah.
131 00:13:07.680 ⇒ 00:13:11.839 Robert Tseng: Well, within the first… okay, this was, like, within the first…
132 00:13:11.970 ⇒ 00:13:17.990 Robert Tseng: day. So, like, we could extend this range. So, I don’t know if we call it, like, in the first month, I’m sure these numbers all go up.
133 00:13:20.520 ⇒ 00:13:22.030 Ashley Samay: Okay, so that makes more sense.
134 00:13:22.230 ⇒ 00:13:22.960 Robert Tseng: Yeah.
135 00:13:23.630 ⇒ 00:13:31.550 Ashley Samay: But it’s still not, it doesn’t seem as much of an indicator as I would have expected.
136 00:13:31.970 ⇒ 00:13:33.600 Ashley Samay: From what y’all saw earlier.
137 00:13:38.560 ⇒ 00:13:43.450 Parliament Conference Room: Robert, to do the combinations of features, is that where the computation feature comes in?
138 00:13:43.450 ⇒ 00:13:44.060 Robert Tseng: Yeah.
139 00:13:45.750 ⇒ 00:13:53.099 Parliament Conference Room: And do we have… do we land on an alternative solution, or… I know you were looking at, like, other tools, potentially, or AWS?
140 00:13:53.360 ⇒ 00:13:58.810 Robert Tseng: Well, yeah, I was hoping that we could just do that with… within, our… within, like, AWS,
141 00:13:58.810 ⇒ 00:14:04.070 Henry Zhao: No, no, we can’t do it within AWS. Okay. Maybe within MongoDB.
142 00:14:04.510 ⇒ 00:14:05.789 Henry Zhao: Which I’m looking into.
143 00:14:06.030 ⇒ 00:14:07.850 Robert Tseng: Yeah, the one here.
144 00:14:07.850 ⇒ 00:14:21.299 Ashley Samay: Could you also just do something hacky, if it’s like we wanted a one-time analysis, to be like, okay, here’s the cohorts of users who did guides, this, this, because you can usually, I think, upload those users to then see how they converted or tracked.
145 00:14:21.790 ⇒ 00:14:23.139 Robert Tseng: Yeah, we could do that.
146 00:14:23.440 ⇒ 00:14:25.259 Henry Zhao: Yeah, I think it should be doable in MongoDB.
147 00:14:25.920 ⇒ 00:14:27.070 Henry Zhao: Yeah, we kind of have…
148 00:14:27.070 ⇒ 00:14:49.049 Robert Tseng: Not to jump around too much, but, like, I was trying to showcase, like, kind of how we’ve been using cohorts, now that we’ve kind of had a few that we opened up. So this was on the AI features specifically, so I can show you, like, differences between users that have used AI features in the past 30 days versus not, and kind of, like, you know, this is, like, their… Oh, that’s awesome. We could do that for different,
149 00:14:49.120 ⇒ 00:14:50.650 Robert Tseng: Sets of features.
150 00:14:52.650 ⇒ 00:14:57.670 Robert Tseng: Yeah, so, like, the way that this is defined, if we go in here, and we…
151 00:14:58.090 ⇒ 00:15:01.930 Robert Tseng: pop this open… Yeah, like, I’ve kind of just, like.
152 00:15:02.230 ⇒ 00:15:10.849 Robert Tseng: you know, it’s… the logic’s pretty simple, if they’ve used any of these AI features in the past 30 days. So, I would want… I would want to define something like this for every…
153 00:15:10.910 ⇒ 00:15:12.519 Parliament Conference Room: You know, core set of features.
154 00:15:12.520 ⇒ 00:15:13.040 Ashley Samay: the fee.
155 00:15:13.040 ⇒ 00:15:14.610 Robert Tseng: Yeah, every combination, exactly.
156 00:15:16.150 ⇒ 00:15:23.930 Ashley Samay: Cool. I don’t know how, like, elucidating that’s gonna be, but it’s worth trying. Like, I’m not gonna block it, for sure.
157 00:15:24.120 ⇒ 00:15:24.800 Robert Tseng: Okay.
158 00:15:25.040 ⇒ 00:15:33.779 Robert Tseng: Yeah, I mean, obviously for this one, it didn’t actually turn out to be the most helpful, just because the volume is very low still on the… oops, I lost my…
159 00:15:36.780 ⇒ 00:16:00.489 Robert Tseng: Yeah, for AI, AI usage, obviously, like, it looks, you know, the total usage is continuing to increase, but the number of users is quite low, so I feel like anything we’re saying about retention or kind of conversion is still pretty early signal. But obviously, maybe for some of the more mature features or workflows, like, this would probably be a more valuable analysis, so if you wanted to, like, drill into the API definitions upload, we could kind of
160 00:16:00.500 ⇒ 00:16:04.469 Robert Tseng: Basically, templatize this and run it for the same… for that feature.
161 00:16:04.470 ⇒ 00:16:07.569 Ashley Samay: Yeah, we should. I actually think the AI stuff is, like.
162 00:16:08.150 ⇒ 00:16:14.799 Ashley Samay: more useful, even though it’s early indicator. Like, I’m just not sure it’s, like… If you use…
163 00:16:15.830 ⇒ 00:16:18.930 Ashley Samay: Guide and reference, like, those are the flagship features.
164 00:16:18.930 ⇒ 00:16:19.830 Robert Tseng: Yeah, okay.
165 00:16:19.830 ⇒ 00:16:27.499 Ashley Samay: I would almost assume, like, if you don’t use those features, how did you convert to README? Do you know what I mean? Like, I don’t…
166 00:16:27.800 ⇒ 00:16:30.529 Ashley Samay: Phoebe and Alicia, do you, like, track what I’m saying? It’s like…
167 00:16:30.530 ⇒ 00:16:33.530 Parliament Conference Room: Yeah, no, 100%, I don’t know what you would launch for if you’re not through this.
168 00:16:33.530 ⇒ 00:16:40.499 Ashley Samay: Yeah, like, no one pays for us just for changelog, for example, or just for recipes. It’s like, you kind of have to have used those two features.
169 00:16:40.500 ⇒ 00:16:41.190 Robert Tseng: I see.
170 00:16:41.660 ⇒ 00:16:43.010 Ashley Samay: But, like, or…
171 00:16:43.010 ⇒ 00:16:45.280 Henry Zhao: No, I can look into that. Yeah, I can look into that.
172 00:16:49.320 ⇒ 00:16:58.940 Ashley Samay: Where it’s like, okay, if people who use AI, it’s actually more sticky, then it’s like, cool, like, let’s, like, really try to get them to use AI in the first, like, couple days that they’re on the product.
173 00:16:59.870 ⇒ 00:17:00.580 Robert Tseng: Okay.
174 00:17:00.740 ⇒ 00:17:02.799 Robert Tseng: Yeah, well, so I guess, like… Go ahead.
175 00:17:02.800 ⇒ 00:17:04.310 Ashley Samay: No, no, you go ahead first.
176 00:17:04.319 ⇒ 00:17:18.009 Robert Tseng: No, I was gonna say, so it seems to me like the angle that you’re taking is you view, like, these new features, like the AI, as, like, an enhancement to kind of the core features, like, so I would like to kind of download that from your brain on, like, what are those
177 00:17:18.199 ⇒ 00:17:29.949 Robert Tseng: thing that we expect users to always be using. I mean, I can kind of, like, pick up, like, what, you know, that should look like, just based off of the volume of activity, but maybe that’s, like, kind of how we orient around, like.
178 00:17:30.139 ⇒ 00:17:31.989 Robert Tseng: You know,
179 00:17:32.219 ⇒ 00:17:43.499 Robert Tseng: you know, we assume that every user… a truly… like, kind of getting to that idea of, like, was a truly activated user? And, you know, are they… have they… have they kind of gone through
180 00:17:43.769 ⇒ 00:17:56.939 Robert Tseng: change, like, API definitions, chain logs, like, you know, some of these, some of these other core features, and then, are these additional things kind of enhancing, like, their engagement? Are they improving their conversion?
181 00:17:56.939 ⇒ 00:18:05.369 Robert Tseng: Yeah, like, I think I don’t really have a good sense of baseline cohorts for active users, other than just, like, any active event, so I’m just trying to…
182 00:18:05.369 ⇒ 00:18:08.339 Robert Tseng: Get a bit closer to what you… how you see it.
183 00:18:08.340 ⇒ 00:18:09.030 Ashley Samay: Yeah.
184 00:18:09.140 ⇒ 00:18:16.300 Ashley Samay: So, I would say, like, my high level is… you’re gonna buy README,
185 00:18:17.090 ⇒ 00:18:29.349 Ashley Samay: Most likely for our reference section, and you may or may not use guides. Some people buy us for guides only, so it’s like, you would have had to use one of those two features to convert.
186 00:18:29.360 ⇒ 00:18:40.019 Ashley Samay: What is interesting to me, though, is, like, if they use a recipe, does that mean they’re more likely to? If they use Changelog, like, changelog is definitely, like, end-user facing.
187 00:18:40.080 ⇒ 00:18:55.539 Ashley Samay: Ai is interesting to me in the sense of, like, if they’re actually using Agent in those, like, first couple days, they’ll see a differentiator on README of, like, oh, I choose README instead of Mintlify or this other platform because it has these tools.
188 00:18:56.120 ⇒ 00:19:02.999 Ashley Samay: I think some stuff we have not tracked, but I’d be interested in too, is, like, if they’re playing around with the current settings…
189 00:19:03.000 ⇒ 00:19:03.670 Parliament Conference Room: links.
190 00:19:03.670 ⇒ 00:19:16.039 Ashley Samay: Like, are they doing that because they were just a more serious buyer, or is it, like, when they actually see it look like their own docs, like, their coloring, their logo, etc, like, are they more likely to convert? Right.
191 00:19:16.870 ⇒ 00:19:19.300 Ashley Samay: So stuff like that, where I think, like…
192 00:19:21.070 ⇒ 00:19:27.769 Ashley Samay: I would still be curious to know the guides and reference question, it’s just more like, we already know that we need to get them to use those.
193 00:19:27.770 ⇒ 00:19:28.130 Robert Tseng: Okay.
194 00:19:28.130 ⇒ 00:19:29.739 Ashley Samay: Convert, if that makes sense.
195 00:19:29.920 ⇒ 00:19:30.630 Robert Tseng: Got it.
196 00:19:31.420 ⇒ 00:19:33.479 Robert Tseng: Well, I mean, that, yeah, that’s helpful. Okay.
197 00:19:33.620 ⇒ 00:19:46.139 Ashley Samay: And then I also, I, like, love looking at charts and data, and I just want to make sure that you also get out of this meeting what you need. What else do you need from me to… in order to be successful?
198 00:19:46.140 ⇒ 00:19:58.459 Robert Tseng: Sure. Let me jump into… yeah, so I mean, I’ll kind of… hopefully these links are all shared, you can kind of pour into that in your own time as well. Just kind of just summarizing kind of things that we’ve already done this week, so…
199 00:19:58.690 ⇒ 00:20:03.270 Robert Tseng: Validation, reporting, we’ve kind of walked through AI features, we talked through…
200 00:20:03.460 ⇒ 00:20:26.420 Robert Tseng: Yeah, kind of opportunities that I kind of see, as far as, like, I don’t want to run these by you, and if there are anything else, like, happy to add to this list, but this is where I was hoping to spend a few minutes kind of talking about it with you. So, we talked about grouping behaviors by key workflows. I think you kind of described to me, like, what, you know, you already know users should do, so that should easily be a cohort that we set up. But yeah, being able to replicate kind of, like.
201 00:20:26.420 ⇒ 00:20:34.279 Robert Tseng: I did for the AI features across other ones, or across other feature sets. I think that’ll help us to… to do better and
202 00:20:34.380 ⇒ 00:20:47.790 Robert Tseng: finding these other activation retention levers, so I think that’s kind of… seems like that’s aligned with kind of what you’re… how you’re already thinking about it. There’s this kind of initiative, we’re talking about proxy computed events. I talked about
203 00:20:47.790 ⇒ 00:21:05.509 Robert Tseng: plan change and, like, kind of attempted launch as being, like, kind of our starting point for some of these events, but the point is, like, we don’t… I think there are… maybe we’re not selecting the right events that you care about for different conversion points in the funnel, and so, wanting some more guidance there, like, you know.
204 00:21:05.510 ⇒ 00:21:14.530 Robert Tseng: subscription success is pretty straightforward, but, like, you know, do we care about, like, upgrade, downgrade? Just specifically on, like, kind of the pricing or revenue side, like.
205 00:21:14.530 ⇒ 00:21:29.380 Robert Tseng: you know, are those all the events that we care about? And if not, then, you know, what else do we need to be looking at there that we need to add to, the… at least the revenue… the way that we talk about revenue events?
206 00:21:29.800 ⇒ 00:21:31.660 Parliament Conference Room: And, Ashley, just, like.
207 00:21:32.100 ⇒ 00:21:36.910 Robert Tseng: the reason I wanted to join the meeting is, so we had that, conversion meeting earlier this week.
208 00:21:37.000 ⇒ 00:21:45.070 Parliament Conference Room: And I want… This team to be able to very quickly spin up the tracking to support those experiments.
209 00:21:45.100 ⇒ 00:22:02.049 Parliament Conference Room: And, like, some of the things we threw out, you know, Alicia just mentioned, like, bringing back the trial, there were other things in the meeting on Monday that, like, ideally, we could very quickly have tracking in place, but also things that I imagine you’ve been thinking about for years that you haven’t been able to…
210 00:22:02.050 ⇒ 00:22:08.620 Parliament Conference Room: to have insight on. So, ideally, like, that’s what we talk about, how we set up the team to support those things.
211 00:22:09.300 ⇒ 00:22:10.580 Ashley Samay: Okay, cool.
212 00:22:10.580 ⇒ 00:22:11.200 Robert Tseng: Yeah.
213 00:22:11.490 ⇒ 00:22:26.820 Ashley Samay: Yeah, I think the number one, it sounds like maybe you have, you just need to validate data, but it’s like, to quickly look at a conversion flow is so useful, obviously, right? It’s like, you can put an amplitude when an event happened. So as soon as you’re confident, like, I’m excited to use it.
214 00:22:27.310 ⇒ 00:22:30.519 Ashley Samay: I think for some of the other stuff.
215 00:22:30.720 ⇒ 00:22:41.949 Ashley Samay: Some of it happens outside of Amplitude, so let me tell you, like, if I had the bandwidth, or if there was, like, a PM dedicated to this, right, with our AI launch, it’s like, I want to know…
216 00:22:42.410 ⇒ 00:22:58.820 Ashley Samay: this is not all in amplitude, but it’s… I want to know not only, like, who is using it, but how often are they using it, which is in amplitude, right? Of, like, okay, do they use it? Do they use it one time? How many times do they use it? Do they hit the limit? But I also want to know, like, okay, what are people putting in their style guide? Like, what types of responses?
217 00:22:59.290 ⇒ 00:23:02.650 Ashley Samay: getting back? Did it make sense? Are they using errors? Are they using more.
218 00:23:03.170 ⇒ 00:23:17.759 Ashley Samay: Or how many times did they get, like, a failure response back from an audit log that they tried, or a linter run that they tried? Because one thing I think README does not do well on is, like, really thinking through pricing, packaging, what the upsell motion is.
219 00:23:17.760 ⇒ 00:23:18.100 Robert Tseng: Yeah.
220 00:23:18.100 ⇒ 00:23:36.870 Ashley Samay: It’s like, we just chose this random limit of 5, which is great, it’s a great start, but it’s like, it’s not based at all in user behavior. And it would be nice to have that surfaced here, even if it’s not the qualitative side, at least the quantitative side, to be like, yeah, actually in 5 minutes, people only use this one time.
221 00:23:36.870 ⇒ 00:23:52.549 Ashley Samay: Or, yeah, they use it, they hit the 5 limit. Or for the people who pay for it, they use it 6 times on average in 5 minutes. Like, that tells me that 5 is, like, way too high, or whatever it is. I don’t know if we have any of that being tracked already.
222 00:23:52.550 ⇒ 00:23:54.730 Robert Tseng: We, we can, we can, we can report on that.
223 00:23:55.480 ⇒ 00:23:56.290 Robert Tseng: Yeah.
224 00:23:56.290 ⇒ 00:24:04.630 Henry Zhao: Yeah, and I think it would be helpful, Ashley, if you just send me those questions that you just rattled off, like, I’d love to look into all of that and kind of see what insights we can get from that.
225 00:24:05.000 ⇒ 00:24:05.600 Ashley Samay: Cool.
226 00:24:05.790 ⇒ 00:24:10.050 Ashley Samay: I can definitely, like, write just… it’s not gonna be polished, but I can give you my.
227 00:24:10.050 ⇒ 00:24:10.410 Henry Zhao: Yeah, yeah.
228 00:24:10.410 ⇒ 00:24:14.110 Ashley Samay: off the cuff, here’s 10 questions that if I had time, I’d love to go answer.
229 00:24:14.110 ⇒ 00:24:15.840 Henry Zhao: Yeah, that would be great.
230 00:24:15.840 ⇒ 00:24:16.670 Ashley Samay: Okay, cool.
231 00:24:16.670 ⇒ 00:24:23.370 Henry Zhao: I think that gives me guidance while I’m digging through the data to just, like, what are some things we should be looking at, and I think we’ll be able to find insights there.
232 00:24:23.770 ⇒ 00:24:24.500 Ashley Samay: Awesome.
233 00:24:25.090 ⇒ 00:24:32.209 Parliament Conference Room: Robert, for the actual conversion funnel that we’re still QAing, do you need anything from this group to get it to, like, a good place?
234 00:24:32.830 ⇒ 00:24:38.950 Robert Tseng: I think, kind of, yeah, just to go back to that.
235 00:24:40.080 ⇒ 00:24:51.440 Robert Tseng: I mean, I don’t know, are you… I mean, attempted launch, is this okay? I mean, I just want to make sure that we’re showing it the right way, right? So, kind of the idea of, like, okay, if attempted launch doesn’t work, plan change is not the right view.
236 00:24:51.490 ⇒ 00:25:07.029 Robert Tseng: And it’s actually, we only care about upgrades or downgrades, and then subscription success, then I need to go and create that derived event. We have to have some computation there to be able to swap this out. So… but if, you know, if this is fine, then it’s just a matter of, like, QAing the
237 00:25:07.230 ⇒ 00:25:11.490 Robert Tseng: You know, QA the data at each step, and then we can just send this to you.
238 00:25:12.300 ⇒ 00:25:13.789 Ashley Samay: I think this is fine, I think.
239 00:25:13.790 ⇒ 00:25:14.180 Robert Tseng: Okay.
240 00:25:14.180 ⇒ 00:25:18.530 Ashley Samay: You could put a step between attempted launch and, like, you actually… so, like.
241 00:25:18.870 ⇒ 00:25:38.400 Ashley Samay: my expectation before we have the trial is, like, people are gonna hit this button and not know what it does, and get to the management plan page, and be like, okay, I’m ready, I’m gonna use these services again, so it’s like, that’s actually not necessarily the point in time in which I expect the conversion, and so in which case, it might be nice to have that additional step, but…
242 00:25:38.400 ⇒ 00:25:53.349 Ashley Samay: I think you had that as a separate funnel, and I think that’s fine. It’s like, we measure two things. We measure overall people that come to readme, they try us, do they convert? And then the second one is, like, are we actually optimized on this checkout flow? Which, the answer is, like, definitely not.
243 00:25:53.350 ⇒ 00:25:54.230 Parliament Conference Room: Probably not.
244 00:25:54.230 ⇒ 00:25:58.019 Ashley Samay: And I think we can, like, monitor those two questions separately.
245 00:25:58.190 ⇒ 00:26:04.729 Robert Tseng: Okay, great. Yeah, in that case, then, I don’t think the structure of this will change. Yeah, we’re just gonna make sure that the data is QA’d there.
246 00:26:04.730 ⇒ 00:26:11.620 Ashley Samay: Okay. One thing also is, I do look at it this way, but I also do instead of the bar chart, yeah.
247 00:26:12.170 ⇒ 00:26:16.890 Ashley Samay: You know how it can tell you conversion over time? Like, it actually gives you the week-to-week subscription success? Yeah.
248 00:26:19.250 ⇒ 00:26:32.989 Robert Tseng: I… yeah, we kind of have that broken out each step, and I kind of have it done, like, bi-weekly cohorts. So, those that created… well, created their project, and then got to subscription success in the first week, then the second week was the third week.
249 00:26:32.990 ⇒ 00:26:45.119 Robert Tseng: We noticed this dip in September, I think Alicia… Alicia mentioned some outage that there was, so that kind of explains that. But otherwise, you know, we… we can definitely do… do that, but cohort it by however you want.
250 00:26:45.490 ⇒ 00:26:46.640 Ashley Samay: Okay, awesome.
251 00:26:46.640 ⇒ 00:26:47.260 Robert Tseng: Yeah.
252 00:26:51.500 ⇒ 00:26:59.889 Parliament Conference Room: Okay. Ashley, Ashley, do you want to join this meeting on a weekly basis, or how do you want to be collaborating with this group here?
253 00:27:00.780 ⇒ 00:27:02.100 Parliament Conference Room: What’s most helpful?
254 00:27:06.740 ⇒ 00:27:11.479 Parliament Conference Room: A lot can be done async, but I just want to make sure you’re being included as needed.
255 00:27:11.480 ⇒ 00:27:16.550 Uttam Kumaran: Yeah, like, these decks or analysis, we can… we’ll send all this as soon as we, like, have it ready.
256 00:27:16.760 ⇒ 00:27:21.859 Uttam Kumaran: I think it’s still helpful sometimes to talk through, because sometimes we have assumption.
257 00:27:22.410 ⇒ 00:27:30.069 Uttam Kumaran: Versus the team, so we try to have, like, a weekly, at least, but, like, we fire… we’ll fire as much of this through Slack as we can.
258 00:27:30.580 ⇒ 00:27:33.749 Ashley Samay: Okay. I don’t think I’m in a Slack channel with y’all.
259 00:27:34.330 ⇒ 00:27:35.929 Parliament Conference Room: We’ll make sure you are.
260 00:27:35.930 ⇒ 00:27:50.300 Ashley Samay: Okay, I would say that’s my preferred. I just don’t want to promise something that I can’t really deliver right now, and I’m stretched really thin. That’s fair. So, like, weekly time is… is tough. Bi-weekly, like, I can commit to. Okay.
261 00:27:50.790 ⇒ 00:28:03.419 Ashley Samay: So, why don’t we try that? Is, like, let’s try to push towards Slack, and then, like, every other week, catching up on any, like, big updates or stuff that we need to talk through in person. Does that work?
262 00:28:04.550 ⇒ 00:28:26.030 Robert Tseng: Yeah, that works. Okay, in that case, I mean, we already have the Slack channel set up, we have this kind of calendar meeting where this slot held, so you can kind of join on a bi-weekly basis. I guess, kind of, another thing I just want to call out, because I know we’re not going to get through everything now, we’ll shoot all this over to you. I think once we get, like, kind of your list of questions, we can rework, kind of, the
263 00:28:26.030 ⇒ 00:28:37.339 Robert Tseng: I just like to roadmap out, kind of, like, what to expect. Like, we kind of have tried to kind of put it into a framework where we have, like, objectives and key results that we’re aiming at, like, you know, we’re just trying to, you know.
264 00:28:38.350 ⇒ 00:29:02.429 Robert Tseng: aggregate all these one-off things into something that’s, like, clearly driving towards, the right objective. That’s, you know, how you see it. So, this is our attempt at kind of, like, categorizing, like, what this bucket of work really is, but, would love your kind of, like, take on how this really rolls up into your, kind of your priorities. So would appreciate feedback anywhere that you can leave it on this deck.
265 00:29:02.900 ⇒ 00:29:07.080 Ashley Samay: Cool, for sure. I’m happy to do that. And I’m not your only stakeholder, and I’m cognizant of that, so, like, you might
266 00:29:07.480 ⇒ 00:29:13.179 Ashley Samay: get conflicting guidance. Like, marketing or ops might want to look at something slightly different than me. But.
267 00:29:13.180 ⇒ 00:29:27.380 Parliament Conference Room: I want to get away from that, Ashley. Like, I… ideally, like, we all care about the same things, and we’re all looking at the same data as well. Like, I don’t want… I don’t want marketing to use one source and you’re using another. Like, the numbers… we should… what we agree to here should be the exact same.
268 00:29:27.380 ⇒ 00:29:37.570 Ashley Samay: Sorry, that’s not what I mean, and I 100% agree with you, like, that self-serve funnel should be the same, and we all look at it, but from, like, I might be more curious about, like, the AI stuff right now…
269 00:29:37.570 ⇒ 00:29:37.960 Parliament Conference Room: Yeah.
270 00:29:37.960 ⇒ 00:29:42.309 Ashley Samay: Like, someone else might be like, okay, well, does the upload for the reference file.
271 00:29:42.310 ⇒ 00:29:42.780 Parliament Conference Room: Yeah.
272 00:29:42.780 ⇒ 00:29:59.039 Ashley Samay: makes sense on what day, so, like, to just be prepared that you might have different priorities from different folks, but I agree with Phoebe, our number one priority is, like, there should be one place where we’re measuring the funnel, we should trust the data, and someone should be, like, responsible for making sure it’s the right data.
273 00:29:59.590 ⇒ 00:30:14.100 Robert Tseng: Yeah, no, totally agree. And I think that usually starts with a product stakeholder, so I feel like, I mean, you’re our main person for now, I guess, and you know, once the other folks get more curious and want to come in, then we can… we’re used to working with multiple stakeholders.
274 00:30:14.100 ⇒ 00:30:17.440 Ashley Samay: Awesome. I’m also… I have to drop for another meeting, but,
275 00:30:17.830 ⇒ 00:30:26.129 Ashley Samay: We had a Google Doc from a long time ago. Did anyone ever give it to you of, like, what the different billing events were?
276 00:30:26.300 ⇒ 00:30:31.579 Robert Tseng: No, this is what I have in terms of Google Docs from, Phoebe a while ago.
277 00:30:31.730 ⇒ 00:30:36.999 Robert Tseng: And I think this is something that, yeah, I put together for my team. So these are the only two docs that I have.
278 00:30:37.250 ⇒ 00:30:47.250 Ashley Samay: Okay, so this is the original, let me make sure you guys have permission… Can y’all, like, put your emails in the Zoom chat really quick?
279 00:30:47.640 ⇒ 00:30:50.309 Parliament Conference Room: Ashley, you’re in the Slack channel now, so you can also drop it there.
280 00:30:50.310 ⇒ 00:30:53.810 Ashley Samay: Okay, cool. Amazing.
281 00:30:55.370 ⇒ 00:31:01.769 Ashley Samay: This should at least, like, help you see what the original,
282 00:31:01.890 ⇒ 00:31:14.629 Ashley Samay: intent was, and why it was, like, always confusing to me, because I thought a lot of these were server-side, and then it was, like, everyone was saying the data was different, so I was like, alright, well, I don’t know how to fix this, though.
283 00:31:14.630 ⇒ 00:31:15.460 Robert Tseng: Okay.
284 00:31:15.730 ⇒ 00:31:31.899 Ashley Samay: I just… I just hit send, so now all three of you should have access. But basically, like, it says which of these, subscription success… to me, it says server, so it was saying this was a server-side, where, like, viewing the managed plan was client-side.
285 00:31:32.090 ⇒ 00:31:43.190 Ashley Samay: So I don’t know, that might be helpful for you to see, like, what was in there before, and you can see my commentary from… you don’t need to, like, take any action there, but just, like, giving you resources that I had from before.
286 00:31:43.900 ⇒ 00:31:44.509 Robert Tseng: Got it.
287 00:31:45.480 ⇒ 00:31:50.529 Robert Tseng: Okay, yeah, we’ll definitely, like, yeah, we’ll look through this. Okay, this is super great.
288 00:31:51.280 ⇒ 00:31:55.550 Ashley Samay: Yup, and then also, actually, there’s another one,
289 00:31:57.120 ⇒ 00:31:59.130 Ashley Samay: Let me grab these three emails again.
290 00:31:59.690 ⇒ 00:32:03.239 Ashley Samay: Okay, I’ll… I’ll move this to Slack, because I actually, I gotta drop.
291 00:32:03.240 ⇒ 00:32:03.970 Robert Tseng: Yeah, yeah.
292 00:32:04.190 ⇒ 00:32:05.669 Ashley Samay: But I will talk to you guys soon!
293 00:32:05.990 ⇒ 00:32:07.460 Robert Tseng: Okay, sounds good. Thanks, Ashley.
294 00:32:08.310 ⇒ 00:32:11.040 Henry Zhao: Thank you Thanks. Bye.