Meeting Title: Brainforce A-B Testing Project Check-in Date: 2026-01-07 Meeting participants: Robert Tseng, Henry Zhao
WEBVTT
1 00:00:26.320 ⇒ 00:00:27.120 Henry Zhao: Alright, give me one second.
2 00:00:27.120 ⇒ 00:00:29.919 Robert Tseng: You wanna… you wanna show me what you’ve… what you’ve got?
3 00:00:30.510 ⇒ 00:00:33.070 Henry Zhao: Yeah, give me one second, let me just find the…
4 00:00:33.070 ⇒ 00:00:33.630 Robert Tseng: Yeah.
5 00:00:34.430 ⇒ 00:00:35.200 Henry Zhao: Screen.
6 00:00:48.880 ⇒ 00:00:51.269 Henry Zhao: Alright, it’s in issues…
7 00:00:55.460 ⇒ 00:01:00.019 Henry Zhao: project, I think, actually. Another repeatable marketing engine.
8 00:01:02.880 ⇒ 00:01:05.190 Henry Zhao: Oh, I didn’t see that students either.
9 00:01:05.510 ⇒ 00:01:21.739 Henry Zhao: Okay, so basically I’ve been working on just this project plan to at least figure out if this is the right structure, and these are the right questions we want to answer. I wanted to get either the team’s opinion or your opinion before I, you know, pull out the tickets, and figure out how we kind of get ELT approval on this.
10 00:01:22.100 ⇒ 00:01:25.940 Henry Zhao: But basically, we want to build a repeatable engine to run A-B tests in 2026.
11 00:01:26.250 ⇒ 00:01:45.869 Henry Zhao: And I think the important thing is we can test hypotheses and get actual results to tie to the goals, right? I think the benefit of A-B testing is you are able to say, like, because of this test we did, we directly drove this much revenue, because when you make a change, you can measure that, and then that can be tied directly to
12 00:01:46.130 ⇒ 00:01:47.769 Henry Zhao: The goals, right?
13 00:01:47.890 ⇒ 00:01:52.000 Robert Tseng: You want to just send me the link? I can just take a few minutes and just read it through, but.
14 00:01:52.000 ⇒ 00:01:52.540 Henry Zhao: That’s true.
15 00:01:52.540 ⇒ 00:01:53.120 Robert Tseng: Yeah.
16 00:01:55.220 ⇒ 00:02:03.129 Robert Tseng: Also, like, this is the quote-unquote reusable epic that we have for A-B testing.
17 00:02:05.030 ⇒ 00:02:09.610 Henry Zhao: Yeah, I had a test just for the plan, but I’ll…
18 00:02:09.970 ⇒ 00:02:11.639 Henry Zhao: Recreate it, because I can’t find it.
19 00:02:11.810 ⇒ 00:02:13.359 Robert Tseng: Already, already recreated it.
20 00:02:13.580 ⇒ 00:02:14.200 Henry Zhao: Oh, God.
21 00:02:16.390 ⇒ 00:02:18.020 Henry Zhao: I just sent you the link.
22 00:02:18.460 ⇒ 00:02:19.080 Robert Tseng: Okay.
23 00:02:25.100 ⇒ 00:02:26.689 Henry Zhao: I didn’t see there was a handbook also.
24 00:02:38.130 ⇒ 00:02:40.019 Robert Tseng: Okay,
25 00:03:19.890 ⇒ 00:03:38.939 Robert Tseng: Okay, like, I understand, high level the plan, I think you just have to… you just have to execute it, because, when you meet with Josh on Monday, he will have already gone through all of these things. So, if he comes to that call more prepared than you, he gets nothing out of you, he’s gonna go back to ELT, he’s gonna say, the Brainforce guys don’t know anything. Here’s… here’s all our findings.
26 00:03:38.940 ⇒ 00:03:55.069 Robert Tseng: Then by the end of next week, they’re gonna be like, alright, you should move off of VWO, you should… this is the first test that you should deploy, this is how we’re collecting data. He’s gonna literally just be doing the same thing, but he’s just gonna say he did it faster, and then we’re gonna lose the deal, so…
27 00:03:55.110 ⇒ 00:04:05.739 Robert Tseng: Yeah, I don’t… I don’t really think we need to go through this, like, exercise of, like, doing… doing the planning. Like, to me, like, the plan of how to stand up AB experimentation is clear, like.
28 00:04:05.820 ⇒ 00:04:13.989 Robert Tseng: we have this handbook that… I mean, I had already previously built one, somebody else worked on it, on this, a while ago.
29 00:04:14.320 ⇒ 00:04:27.910 Robert Tseng: may or may not be helpful, like, it’s not what I built, so I… I… if I were running this, I personally would just use my old playbook and not be using this one. But, you know, whatever… whatever works, I… I think just…
30 00:04:28.130 ⇒ 00:04:33.889 Robert Tseng: Yeah, like, I think that maybe I’ll help you just write the deliverables, and we just work backwards from there. So…
31 00:04:33.890 ⇒ 00:04:50.429 Henry Zhao: Or just tell me what should go into the deliverables in terms of, like, like, 20% should be this, you know what I mean? So, like, it should be, like, 5% just the setup stuff, because that’s obvious, but then 40% should be, these are the tests, and this is the goal that it’s tied to, and this is the analysis of why we think that we can reach that.
32 00:04:50.430 ⇒ 00:04:56.319 Henry Zhao: Or should it be more, like, ideas of what we want to test? So, like, if you can just guide me there, I can put that together by end of day.
33 00:04:56.740 ⇒ 00:04:57.400 Robert Tseng: Okay.
34 00:05:02.710 ⇒ 00:05:04.580 Robert Tseng: Let’s see…
35 00:05:11.830 ⇒ 00:05:15.790 Henry Zhao: In the bottom, just gonna, like, brainstorm, like, things that would go in here, right? Like, so…
36 00:05:30.420 ⇒ 00:05:36.779 Robert Tseng: No, I think, like, any of that strategy BS is, like, not necessary. This is, like, we just… the work needs to… we just need to…
37 00:05:37.050 ⇒ 00:05:39.599 Robert Tseng: We need to have, like, a clear…
38 00:05:40.160 ⇒ 00:05:45.430 Robert Tseng: this is what we’ve done, this is what we’re doing. Like, it’s this… it’s just… it’s very black and white at this point.
39 00:05:45.930 ⇒ 00:05:51.740 Robert Tseng: Yeah, you pick one experiment, like, I don’t know what you’re… pick one…
40 00:05:52.290 ⇒ 00:05:56.020 Robert Tseng: Area to, to run a mock
41 00:05:56.160 ⇒ 00:06:01.349 Robert Tseng: AB… ABX test design. So, you know, whether it’s…
42 00:06:01.600 ⇒ 00:06:08.959 Robert Tseng: Whether it’s intake, homepage, CTA, wherever, like, you have to… you have to figure out where is the most traffic.
43 00:06:09.820 ⇒ 00:06:18.920 Robert Tseng: graph we’re going to? What are the… Where is you didn’t concerned…
44 00:06:20.380 ⇒ 00:06:26.750 Robert Tseng: That, like, marketing, like, that, conversion is below benchmark.
45 00:06:28.700 ⇒ 00:06:36.860 Robert Tseng: what you, like… What is the benchmark for, like, a… for…
46 00:06:37.310 ⇒ 00:06:40.749 Robert Tseng: Yeah, what are the conversion benchmarks?
47 00:06:42.380 ⇒ 00:06:50.900 Robert Tseng: Currently… like, I don’t know, I’ll just try to… as I was looking at some stuff earlier today…
48 00:06:54.010 ⇒ 00:06:58.270 Robert Tseng: Where can I go for an additional recap?
49 00:07:15.430 ⇒ 00:07:20.940 Robert Tseng: native version of this for, CRO.
50 00:07:23.950 ⇒ 00:07:31.359 Robert Tseng: maybe Ryan has something already? Otherwise, you’ve… you’ve… They’ll find it.
51 00:07:33.650 ⇒ 00:07:35.860 Robert Tseng: Like, yeah, you just need, like, a clear…
52 00:07:36.040 ⇒ 00:07:45.969 Robert Tseng: this is what the state of affairs is. These numbers on the left are what they currently are, this is kind of where we’re headed, like, I don’t know, I think that’s… that’s one slide.
53 00:07:47.650 ⇒ 00:08:00.980 Robert Tseng: I guess these could be out of order, like, I’m not gonna necessarily lease the ordering here, like, I don’t really care. If I just… I think this probably makes more sense. You typically have to go top-down.
54 00:08:01.100 ⇒ 00:08:06.969 Robert Tseng: conversion investment most currently, pick one area to run, but it’s like, pick one underperforming…
55 00:08:10.880 ⇒ 00:08:14.349 Robert Tseng: You know, and this is, like, kind of where you… where you bring in…
56 00:08:14.650 ⇒ 00:08:20.109 Robert Tseng: you know, this is how we would design the experiments, and, like, blah blah blah, and then…
57 00:08:20.550 ⇒ 00:08:30.139 Robert Tseng: I would even do, like, a… Like a diagram of… how VWO currently works.
58 00:08:31.060 ⇒ 00:08:36.810 Robert Tseng: VS where it needs to be, like, So, you gotta go in…
59 00:08:37.000 ⇒ 00:08:43.439 Robert Tseng: learn the VWO, workflow from whoever runs it.
60 00:08:43.679 ⇒ 00:08:44.639 Henry Zhao: Ryan, yes, I’m meeting.
61 00:08:44.640 ⇒ 00:08:50.649 Robert Tseng: Diagram, like, what they’re doing versus what they should be doing.
62 00:08:51.650 ⇒ 00:08:53.959 Robert Tseng: Should they…
63 00:08:57.790 ⇒ 00:09:12.050 Robert Tseng: should they be doing… should they be staying with VWO, or what other options are they already considering? Which I’m sure there… somebody in Eden is looking for other options. So.
64 00:09:12.250 ⇒ 00:09:21.299 Robert Tseng: Yeah, like, I, you know, I think… those… Just those 3 core things.
65 00:09:21.460 ⇒ 00:09:22.630 Henry Zhao: would…
66 00:09:22.850 ⇒ 00:09:27.909 Robert Tseng: be enough. Like… To, like, kind of get…
67 00:09:29.720 ⇒ 00:09:31.620 Robert Tseng: To get you going. Like, I kind of see this…
68 00:09:31.620 ⇒ 00:09:32.109 Henry Zhao: It’s very light.
69 00:09:32.510 ⇒ 00:09:37.030 Robert Tseng: This is, like, you know, 3 to 5 slides max.
70 00:09:37.480 ⇒ 00:09:39.250 Henry Zhao: Okay, 3 to 5 sides max. Okay, got it.
71 00:09:40.590 ⇒ 00:09:41.180 Robert Tseng: Yeah.
72 00:09:41.650 ⇒ 00:09:43.319 Henry Zhao: Okay, I have a clear direction.
73 00:09:43.840 ⇒ 00:09:53.459 Robert Tseng: Okay, so… Anything else to include here?
74 00:09:58.060 ⇒ 00:10:00.839 Robert Tseng: Yeah, I mean, I would say, like, just…
75 00:10:02.410 ⇒ 00:10:17.219 Robert Tseng: I personally prefer to outline in a running, like, a long-form doc. I kind of, like, block out each, like, header as, like, a slide, just so I can kind of structure the outline. You’ve seen my outlines before, they’re kind of, like,
76 00:10:17.790 ⇒ 00:10:22.039 Robert Tseng: Is this my analysis outline? Question mark? No, it’s still my thing.
77 00:10:22.600 ⇒ 00:10:23.570 Robert Tseng: I just…
78 00:10:55.540 ⇒ 00:10:57.650 Robert Tseng: Oh my goodness, there’s too many.
79 00:11:19.130 ⇒ 00:11:21.150 Robert Tseng: Yeah, like, I mean…
80 00:11:24.520 ⇒ 00:11:28.090 Robert Tseng: I don’t like to micromanage how people do things, I just…
81 00:11:28.400 ⇒ 00:11:41.920 Robert Tseng: But, like, when I’m, like, unclear on, like, how you’re approaching it, I just… I would strongly recommend a structure like this, where you have this bit outlined, it’s clear, like, what the goal is, what…
82 00:11:42.270 ⇒ 00:11:54.090 Robert Tseng: I mean, I don’t know, this is very basic stuff, but, like, I learned this at my first job, so, like, I’m sure you probably do some version of this already, but I do think it should just be documented, because this really helps people know how you’re approaching a problem.
83 00:11:54.280 ⇒ 00:12:09.879 Robert Tseng: And I can go and fill in any gaps where I’m like, hey, you’re missing this question, or like, you know, I actually don’t know if that’s the why. You know, there’s, like, certain things that I need to… like, this helps me to review, like, what you’re doing before, you know, we get to a point.
84 00:12:09.970 ⇒ 00:12:16.459 Robert Tseng: Where, yeah, the pattern has been, you end up spitting out kind of a slide, and I’m like, what is.
85 00:12:16.460 ⇒ 00:12:17.189 Henry Zhao: Are you sharing your screen?
86 00:12:17.190 ⇒ 00:12:23.680 Robert Tseng: And… I am supposed to be sharing my screen?
87 00:12:26.110 ⇒ 00:12:26.840 Henry Zhao: Yeah, that document.
88 00:12:27.670 ⇒ 00:12:36.920 Robert Tseng: I did, I slacked it to you. But, like, you’ve seen this before, I just… regardless of analysis or any type of technical work, like.
89 00:12:37.470 ⇒ 00:12:42.000 Robert Tseng: I just like to outline how I’m approaching… how I’m approaching it.
90 00:12:43.810 ⇒ 00:12:51.789 Robert Tseng: And it just makes it very clear, like, I would send this… I would expect something like this from you today, for…
91 00:12:52.870 ⇒ 00:12:54.800 Robert Tseng: like, the…
92 00:12:55.400 ⇒ 00:13:09.890 Robert Tseng: like, I don’t need you to mock up the slides, like, I think just having this shows that you’ve already put enough thought into how to approach it. I can actually quickly review this, I can check to see if your goal is… is that actually aligned with the objective?
93 00:13:10.040 ⇒ 00:13:25.459 Robert Tseng: are these the right steps to get there? I can quickly see that. Are these the right questions that you’re asking? I can also quickly review that. And then, like, do you understand the why? So, like, that’s why I think this is helpful for a reviewer, like, to be able to go through this very quickly.
94 00:13:25.460 ⇒ 00:13:26.060 Henry Zhao: Okay.
95 00:13:26.060 ⇒ 00:13:27.000 Robert Tseng: Yeah, so…
96 00:13:27.000 ⇒ 00:13:27.690 Henry Zhao: Very helpful.
97 00:13:27.690 ⇒ 00:13:39.289 Robert Tseng: yeah, if you don’t have… yeah, I mean, if you don’t have a better way, like, I would just recommend you do something like this. Yeah, once again, like, this is… the timing is tight, like.
98 00:13:39.620 ⇒ 00:13:40.270 Henry Zhao: Yep.
99 00:13:40.270 ⇒ 00:13:59.609 Robert Tseng: your meeting… your goal, your short-term goal is, like, by the time you meet with the other agency on Monday, you should know way more than they do, and we need to be able to, like, get what we want out of them, rather than they get what they want out of us. Correct. It’s kind of like…
100 00:13:59.610 ⇒ 00:14:15.009 Robert Tseng: yeah, we’re not… we’re not, like, collaborating with them. We’re not friends. And so, they’re clearly trying to, like, take our scope. So, I mean, obviously, like, there’s a negotiation where we want to give them some information that’s helpful for them, but try to understand how they’re approaching it.
101 00:14:15.350 ⇒ 00:14:20.680 Robert Tseng: And… like… you know, like, You should be…
102 00:14:21.100 ⇒ 00:14:31.759 Robert Tseng: Yeah, once again, like, whenever I’ve gone into these head-to-heads, like, I always want to come out looking like I’m more prepared than the other person. So, yeah, like, if that’s…
103 00:14:33.250 ⇒ 00:14:45.020 Robert Tseng: Yeah, like, that’s really the goal for that one. And then, by the end of the week, next week, or in the deck, really, not even by Friday, like, I just… by the deck of the ELT deck.
104 00:14:45.020 ⇒ 00:15:02.340 Robert Tseng: I want to get ahead of them and be like, look, I know you’ve been talking to Josh from the other agency, here is the three to five slides that we put together on how we’re driving experimentation. I want them to be blown away by what we have, and then they’ll be like, alright, great, we can wind down, Josh. You guys keep the scope. That’s the outcome that I’m looking for.
105 00:15:03.030 ⇒ 00:15:03.919 Henry Zhao: Got it.
106 00:15:03.920 ⇒ 00:15:04.350 Robert Tseng: Yeah.
107 00:15:04.750 ⇒ 00:15:08.800 Henry Zhao: But can I continue to get your feedback throughout the week, just to make sure that I’m on the right track?
108 00:15:09.100 ⇒ 00:15:14.350 Robert Tseng: Yeah, like I said, I need to… I need something to review. I’m not gonna jump on multiple calls with you, you’ve seen my calendar.
109 00:15:15.280 ⇒ 00:15:26.439 Robert Tseng: No time the next two days. But yeah, so, like, I need to do stuff async. If I will need to jump on another call, that’s… that’s fine. We can… we should… we should do that by the end of the week. Nope, I’m working on this…
110 00:15:26.440 ⇒ 00:15:28.049 Henry Zhao: Yeah, I’ll work on this doc and send it to you.
111 00:15:28.210 ⇒ 00:15:28.580 Robert Tseng: Yeah.
112 00:15:28.580 ⇒ 00:15:30.519 Henry Zhao: And then we’ll talk and stand up tomorrow to see how it went.
113 00:15:30.520 ⇒ 00:15:32.159 Robert Tseng: Yeah, yeah.
114 00:15:34.100 ⇒ 00:15:36.350 Henry Zhao: Alright, thank you, Robert. This is really helpful.
115 00:15:36.350 ⇒ 00:15:37.120 Robert Tseng: Thanks.