Meeting Title: Experimentation and CRO overview Date: 2026-01-26 Meeting participants: Ryon, Greg Stoutenburg, Mitesh Patel, Danny Valdez
WEBVTT
1 00:00:33.110 ⇒ 00:00:34.620 Ryon: Hey, Greg, how’s it going?
2 00:00:34.620 ⇒ 00:00:36.190 Greg Stoutenburg: Hey, Ryan, good, how are you?
3 00:00:36.580 ⇒ 00:00:39.569 Ryon: Good, good, good. Happy Monday. Good weekend?
4 00:00:39.570 ⇒ 00:00:40.140 Greg Stoutenburg: Yeah.
5 00:00:40.420 ⇒ 00:00:44.419 Greg Stoutenburg: Great weekend, very snowy, snowpocalypse out here.
6 00:00:44.420 ⇒ 00:00:45.109 Ryon: Yeah, I was gonna say.
7 00:00:45.110 ⇒ 00:00:45.610 Greg Stoutenburg: us.
8 00:00:45.610 ⇒ 00:00:48.859 Ryon: Brad, are you getting hit by the storm? I feel like everyone I talk to is like.
9 00:00:48.970 ⇒ 00:00:51.630 Ryon: getting impacted some way or another. Why is it so tough?
10 00:00:52.910 ⇒ 00:00:56.209 Greg Stoutenburg: Yeah, we got… actually, I didn’t even check the total.
11 00:00:56.550 ⇒ 00:01:07.110 Greg Stoutenburg: I bet it’s, like, 16 inches. It’s a lot. It’s a lot. Yeah, it’s one of those days where it’s like, if you aren’t shoveling as it’s coming down, then you’re gonna have a rough morning the next day.
12 00:01:07.110 ⇒ 00:01:09.399 Ryon: Because you’re gonna get trapped inside.
13 00:01:09.400 ⇒ 00:01:27.980 Greg Stoutenburg: Yeah, exactly. Yeah, so I was away yesterday, and then when I drove in last night, like, there was enough snow that my car, like, the bottom of my car was scraping the snow and, like, pushed it back. So, quite a lot. Most snow I’ve ever seen here. I grew up in Michigan, we got more snow than central Pennsylvania here, so there’s a lot for central PA.
14 00:01:28.380 ⇒ 00:01:40.859 Ryon: I was just so… I… wow, you’re in Central PA, wow, that is a lot. So I was in Syracuse for a few years, and it was an insane experience. I… Southern California boy going to Syracuse, like, it was just like, what the hell?
15 00:01:41.470 ⇒ 00:01:42.230 Greg Stoutenburg: Yeah.
16 00:01:42.580 ⇒ 00:01:44.399 Ryon: It was an adjustment.
17 00:01:44.400 ⇒ 00:01:53.329 Greg Stoutenburg: Yeah, I have a friend, I have a friend who lives in Syracuse, and the photos are… I’m like, I didn’t… I didn’t know it snowed that much anywhere.
18 00:01:53.330 ⇒ 00:02:17.029 Ryon: They’re such good people, though. Like, everyone helps each other out. Like, I had a… back then, I had, like, a 2007 Highlander, so it was a little bit raised up, and they’re like, oh, you’re so lucky, you’ve got, like, you know, you’ve got coverage, you’ve got lift, and I’m like, oh, okay, what is that? Why does that matter? Like, well, you know, you’re gonna go down the street, and there’s gonna be these piles of snow. Like, if you’re in a small sedan, you just basically bottom out, and you get stuck. But you’re like, if you’re up a little higher, you can kind of sail on through, no problem. Like, just gas it and then roll.
19 00:02:17.270 ⇒ 00:02:18.579 Ryon: And I’m like, okay…
20 00:02:18.580 ⇒ 00:02:19.040 Greg Stoutenburg: Yes.
21 00:02:19.040 ⇒ 00:02:19.680 Ryon: Yeah.
22 00:02:19.680 ⇒ 00:02:39.380 Greg Stoutenburg: Yes, I know, yeah. I, I was telling my son yesterday that, unfortunately, if we weren’t in a neighborhood that had such narrow streets, like, with mailboxes and stuff, I’m like, man, I would have just given you the keys, even though you’re 13, because that’s, like, I learned so much more about driving when I was a teenager. My dad was like, yeah, we’re on snowy back roads, there’s nobody around, than, you know, than I ever did in driver’s ed. It’s good for you.
23 00:02:39.760 ⇒ 00:02:48.939 Ryon: There you go. Okay, let me share my screen. I’m just gonna go over our framework of everything. Some of this might be new to…
24 00:02:49.120 ⇒ 00:02:55.589 Ryon: some in the call, some of it, I hope it’s not too new, but at least we’re kind of going over, like, the high level here. Yeah.
25 00:02:56.460 ⇒ 00:03:01.030 Ryon: Is this gonna… Disappear… It’s down here anymore.
26 00:03:03.410 ⇒ 00:03:10.669 Ryon: Okay. So… zooming out, You know, 40,000 foot view.
27 00:03:12.120 ⇒ 00:03:13.810 Ryon: there are…
28 00:03:16.220 ⇒ 00:03:23.619 Ryon: From an A-B testing CRO perspective, there are really 3 key places that we are managing the roadmap as a whole.
29 00:03:23.620 ⇒ 00:03:38.039 Ryon: Okay? And kind of the framework of things, keeping in mind that I would say this is about the second, maybe the second and… we’ve got two and a half quarters of where we’ve been sort of doing this, and, like, refining our process as we go on, and it’s not perfect by any stretch, but we’re getting there.
30 00:03:38.040 ⇒ 00:04:03.010 Ryon: So, first things first. This is the shared CRO web board with Danny and I inside of Monday. I’ll give you access to it. This is kind of where we manage the backlog and everything that gets submitted to us as an A-B test request. All requests go through this form here, okay? And they generally follow this test brief template. This is something that, we had our team build out for us so that the request form would answer all of these
31 00:04:03.010 ⇒ 00:04:13.399 Ryon: test brief templates, right? Everything in detail, exactly what you’re looking for, all this kind of stuff, and then everything is carried over into the columns over here inside of our main table, right?
32 00:04:13.710 ⇒ 00:04:36.100 Ryon: But again, zooming out for a second, let me just go over our planning sheet. So, every quarter, I prepare this with Danny, our planning sheet here, which kind of talks about our mission, our program architecture, what our framework is. Our testing framework is sort of TBD on this. We sort of follow with, you know, the test brief, and then kind of some of the basic A-B testing frameworks that are out there, but we need something more formal, I think, specific to Eden.
33 00:04:36.100 ⇒ 00:04:52.770 Ryon: Our strategic themes, this is something set and agreed upon by leadership, right? So, what are we going to measure each one of our, tests by? User insights and behavioral landscape, this sort of updates over time. This is our reporting and communication structure. Some of these things have been retired.
34 00:04:52.770 ⇒ 00:05:12.710 Ryon: Right? And Danny and I, I think once we sort of onboard, support teams are going to kind of reassess this, section of our planning document, like our, you know, we still have our brainstorm and a planning session, but, you know, do we need a weekly KPI report? All this kind of stuff, quarterly executive report, all those things. Then our OKRs, right? We set our OKRs each quarter. Danny and I agree on these, we set these.
35 00:05:12.710 ⇒ 00:05:14.390 Ryon: Build everything
36 00:05:14.630 ⇒ 00:05:29.359 Ryon: towards these OKRs from an A-B testing perspective? Like, how many tests do we want live, what is our percentage winner rate, what is the expected conversion rate, and what is our incremental lift target? This is revenue, not, not LTB, I should say that. This needs to be,
37 00:05:29.430 ⇒ 00:05:48.499 Ryon: revenue here. And then other things like this. A few of these we’ve struggled with, like, example, this row here. We kind of got some pushback from different people on the survey tool we were launching in BWO, which is fine, it wasn’t designed the best way, but the concept, or the principle, was at least good, that we were trying to, you know, harvest data from our users, but at the same time, like, yeah, probably should find a better way to do that.
38 00:05:48.500 ⇒ 00:05:56.890 Ryon: KPIs, team logistics, and our sprint schedule, right? So everything we do is based on our sprint schedule of usually 2 weeks. Example, we’re just starting Sprint 2.
39 00:05:56.890 ⇒ 00:06:08.769 Ryon: I would say Sprint 1 was kind of a recalibration sprint. We definitely didn’t get a lot live, but we’re sort of kind of, you know, looking to onboard new team members, support team, and then figure out, you know, what we’re going to…
40 00:06:08.770 ⇒ 00:06:15.299 Ryon: set for the future, and so on. And then, of course, here’s the team structure, who has access to what, and who does what.
41 00:06:15.300 ⇒ 00:06:32.169 Ryon: You’ll see Zaron on here, because he obviously deals with the web analytics side, helping us with that. Casey as well, supports him. Nick helps us with intake design and structure. And then, of course, we’re hiring for these two positions… sorry, these three positions over here, right? So, web developer, serial specialist, and intake optimization specialist as well.
42 00:06:32.890 ⇒ 00:06:34.609 Ryon: So yeah,
43 00:06:34.770 ⇒ 00:06:59.260 Ryon: Briefly gonna go over the table here. All new requests go into the backlog. Danny and I groom these requests, usually every other week. We decide, based on an ICE performance estimate, what we think the request is going to be. We set an effort score. We usually tie it back to a previous other test, so it’s iterative. We’re not just, like, saying, oh, this is, you know, some random idea we came up with. Nope, we’re improving, we’re growing, we’re expanding, we’re changing.
44 00:06:59.410 ⇒ 00:07:13.699 Ryon: We will set a live by date, a start date, an end work date, and then, of course, end test date. We will set the hypothesis, variant overview, success metrics, the tools used, all this stuff here is integrated. It’s probably best for me to show this to you for any of our ended
45 00:07:13.700 ⇒ 00:07:26.000 Ryon: test, because it’s all filled out for them, you can see this much better, for them here, right? And then, of course, once it’s ended and all of our, our, you know, information is captured, you know, we’ve decided to remain a judgment call, there’s an automated
46 00:07:26.000 ⇒ 00:07:33.460 Ryon: job that runs on this board, and it will update the CRO,
47 00:07:33.920 ⇒ 00:07:46.890 Ryon: A-B test board here with the status or the information that we, added into the board for this, and of course, everyone who’s in this channel here gets an update and is notified, right?
48 00:07:47.220 ⇒ 00:07:54.519 Ryon: We always like, you know, winners, right? That’s always what we like to see, but, you know, obviously it’s A-B testing, sometimes there’s some losers.
49 00:07:54.520 ⇒ 00:08:11.439 Ryon: From a dashboard perspective, this is sort of a… it’s always been a work in progress. I think Danny and I have worked on this a couple times. I want this to be better. I think we both… everyone wants this to be better, right? So, we do have kind of a breakdown of where we stand on, like, our win-lose rate, and then you can, of course, filter this by date or time.
50 00:08:11.440 ⇒ 00:08:31.320 Ryon: a task breakdown, like, you know, what percentage of our tasks are in what bucket. In general, we like to see things assigned or running be the most, that way we can kind of see things flowing through the backlog and getting live. The type of test breakdown, this is something I think ELT has sort of vocalized to us, that we need to be more than just, you know, two different test types.
51 00:08:31.320 ⇒ 00:08:37.029 Ryon: Right? We need to be expanding and growing. And then, of course, effort breakdown here. Again, you know.
52 00:08:37.169 ⇒ 00:08:40.770 Ryon: We really focus on the easy wins, so that’s why you’re gonna see a lot of
53 00:08:40.789 ⇒ 00:09:00.090 Ryon: our tests have been kind of 1 to 3s. Fours and fives are big lift tests in general. You know, Danny and I sort of estimate that we can kind of get through only about one or two of those per sprint, because they’re just so much, they’re like a really big lift, like, we’re redesigning a whole page, there’s a whole experience, that’s a thing. So, we generally try to get through as many of those as we can, but
54 00:09:00.470 ⇒ 00:09:07.610 Ryon: bit more effort. And then, of course, our high-level roadmap, which goes by the live date, that’s over here inside of the
55 00:09:07.610 ⇒ 00:09:23.900 Ryon: backlog, you’ll see that we have everything with a live date, live by date. We try to be as good with this as we can, keeping in mind that, yes, during our first sprint, we’ve been struggling a little bit with getting some of these things live, but in general, we sort of set the roadmap and say, these are the things that we’re focusing on.
56 00:09:23.900 ⇒ 00:09:28.960 Ryon: And then we can, of course, break it down by theme, right? We try to focus on the different themes that we’re…
57 00:09:28.970 ⇒ 00:09:31.649 Ryon: Targeting per quarter.
58 00:09:32.220 ⇒ 00:09:34.629 Ryon: Down here, so strategic themes.
59 00:09:35.020 ⇒ 00:09:45.860 Ryon: That’s pretty much operationally how we operate. We used to do weeklies. We don’t anymore. We sort of have adapted to a new approach that Mitesh has proposed, where we have a
60 00:09:45.860 ⇒ 00:09:55.920 Ryon: performance report that shows us by page type how we’re doing, and gives us a kind of a metric breakdown. I would say there’s a big disconnect between
61 00:09:56.010 ⇒ 00:10:09.270 Ryon: what we’re tracking in the operations perspective, and then, like, what we’re reporting on is our performance at this point, we’re looking to kind of bridge that gap a little better, but we’ll… we’ll get there. It’s… it’s a work in progress. From a…
62 00:10:09.710 ⇒ 00:10:11.760 Ryon: Execution perspective.
63 00:10:12.130 ⇒ 00:10:22.389 Ryon: we predominantly live and use VWL. We do… we… thanks to Danny, we actually do some pretty sophisticated A-B testing now, where we can
64 00:10:23.260 ⇒ 00:10:40.320 Ryon: sort of, like, adapt pages better, and use some of the wildcard capabilities that VWO has, so we can kind of test a lot of different pages, because everything’s templatized now. So, like, let’s hypothetically say we wanted to test the CTR of wording on the main CTA for all PDPs.
65 00:10:40.390 ⇒ 00:10:54.499 Ryon: We can do that now, because all PDPs are templated, and it makes it very easy for us to change things, and then, of course, same as holds true for all other templated page types. Multivariant is something we need to get into, and then, of course, split URL testing. This is probably the bread and butter of what we do.
66 00:10:54.770 ⇒ 00:11:08.200 Ryon: we not only have our routers that we use, which is basically just a Cloudflare worker, we have all of our intake testing, like, for example, we have a test going on right now with Forbes intake.
67 00:11:09.140 ⇒ 00:11:10.399 Ryon: Where we are.
68 00:11:10.730 ⇒ 00:11:15.939 Ryon: Comparing two different, intake flows against each other.
69 00:11:15.940 ⇒ 00:11:19.239 Ryon: One uses personalization, the other does not.
70 00:11:19.240 ⇒ 00:11:39.050 Ryon: And we’re trying to benchmark these two against each other, because we’ve added unique functionality to the new one, personalized functionality. And I’d say, like, you know, some of these things, we really struggle with getting to significance, because we don’t always have a lot of, like, traffic for some of these different page types, or some of these different intakes. It’s really hard for us to reach significance
71 00:11:39.050 ⇒ 00:12:01.889 Ryon: particularly you can see here, we’re only talking about, like, maybe 100 a day for this particular intake. That’s not going to be something we reach significance quickly. In fact, if you look here, it’ll be like, oh, you’re gonna get to significance 2,000 days from now. Like, okay, yeah, 6 years later, like, yeah. So, we sort of tried to reach at least 65% confidence interval, and then sort of make a directional judgment call at that point. If we’re lucky, with certain pages, like our Google
72 00:12:01.890 ⇒ 00:12:05.429 Ryon: landing pages. We can get to significance, like, 80%
73 00:12:05.430 ⇒ 00:12:13.360 Ryon: fairly quickly, just because there’s so much traffic going to those pages. Or even, like, the homepage, or even some of the PDPs, but we sort of have to be, like, careful.
74 00:12:13.600 ⇒ 00:12:15.870 Ryon: But yeah, we do struggle with that a little bit.
75 00:12:16.000 ⇒ 00:12:19.560 Ryon: As far as other insights are concerned, earlier this…
76 00:12:19.560 ⇒ 00:12:39.379 Ryon: year, or late last year, I actually built out a lot of new events inside of GTM to be tracked and measured here, for VWO. Like, all of the metrics that are our custom events that BASC feeds us are integrated here, so you have the basics event, the safety, consent, product, ad shipping info.
77 00:12:39.420 ⇒ 00:12:52.470 Ryon: Product added, ad payment info, all of these events are there. And then, of course, I’ve also added in some of the more unique events that I created for us. The Bass Coupon error, right? Skim reading, exit intent, idle intent.
78 00:12:52.490 ⇒ 00:13:00.900 Ryon: calculator health info. This is something Danny built out on our calculators. It’s just basically a Boolean true-false. Somebody fires the calculator by hitting
79 00:13:01.890 ⇒ 00:13:09.940 Ryon: accept, right? And then, of course, you’ve got all the other ones, like purchase event, add to cart, first time purchase. I updated all of these as well, because there were some
80 00:13:10.120 ⇒ 00:13:14.249 Ryon: nuanced to them. There’s stuff here that VWO…
81 00:13:14.360 ⇒ 00:13:20.350 Ryon: sees or sends, but I don’t necessarily want to add them to the account just yet. They’re additional events that are just…
82 00:13:20.820 ⇒ 00:13:24.480 Ryon: you know, they’re just there. I don’t know if they’re necessary at this point.
83 00:13:26.520 ⇒ 00:13:34.600 Ryon: Yeah. As far as some of the other things are concerned, I’ve set up funnels here, for our main traffic.
84 00:13:34.770 ⇒ 00:13:54.180 Ryon: I’ve really struggled with this, to be honest, and the reason why is not so much the funnel itself as it is… VWO really is… got a few different things that they’ve been trying to work out with us to better understand these funnels, like, what is the blog to intake to conversion rate performance, what’s the homepage to log in, PDP to intake to purchase, first time purchase, right?
85 00:13:54.180 ⇒ 00:13:58.930 Ryon: So on. They really struggle with being able to identify these things, and…
86 00:13:58.930 ⇒ 00:14:09.409 Ryon: quantify, the actual performance. So, if you come over here to the funnel reports, you’re gonna see, like, for the past 7 days, what the actual performance is, but in some cases, you might end up with, like, a really
87 00:14:10.530 ⇒ 00:14:20.460 Ryon: low set of numbers, or something that just doesn’t make sense. So I’ve kind of worked with them to be like, hey guys, fix this bug, like, you’re not detecting this event, or something’s happening, and…
88 00:14:20.670 ⇒ 00:14:24.169 Ryon: They do their best to fix it as quick as they can.
89 00:14:24.690 ⇒ 00:14:40.419 Ryon: But yeah, outside of that, other things we use VWO for, session recordings. I use this a lot to look at intake performance, mobile recordings. We have form-related data here that we capture from the calculators, as well as from the footer.
90 00:14:40.630 ⇒ 00:14:41.990 Ryon: form submission.
91 00:14:42.090 ⇒ 00:14:47.789 Ryon: tool, so that’s the email submission tool. We just use this basically to kind of, like, assess…
92 00:14:48.270 ⇒ 00:14:52.570 Ryon: How engaging are the forms on each page, that we have?
93 00:14:52.610 ⇒ 00:15:09.289 Ryon: And is there anything we can do to inform that example, the BMI calculator form here? We could actually come in here and see, of all the people that visit it, you know, 80% of them end up engaging with it. Unfortunately, again, VWABug, they don’t detect something unless it’s an input tag.
94 00:15:09.420 ⇒ 00:15:22.789 Ryon: with the attribute form equals input in the HTML, and the way that we’ve coded it in VWO, they can’t actually see the submissions, but we can see the engagement rate, and then we can come down here and see, like, hey, for the form areas, you know.
95 00:15:22.790 ⇒ 00:15:35.409 Ryon: how much time are people spending on each one of the forms, elements here? What’s the interaction weight? What’s the hesitation time? How often do they refill it? How often does it get ignored? How often do they drop after this or that point? That kind of stuff.
96 00:15:35.610 ⇒ 00:15:38.259 Ryon: I won’t go over surveys, it’s… there’s no point.
97 00:15:38.330 ⇒ 00:15:57.840 Ryon: We can use heat maps here, this goal section here, this is being retired, but we use the heat maps a lot, and we look at not just the heat maps, we look at the click maps, we look at the scroll maps, the zonal maps, and the friction areas as well, using this, and then we sort of extrapolate from that, and, you know, of course, ideas get put into the backlog from that.
98 00:15:58.200 ⇒ 00:16:08.089 Ryon: That’s VWO. Last, but certainly not least, Danny and I have all of these different reports here. The main one I would say that we use to measure the success by our
99 00:16:08.090 ⇒ 00:16:18.839 Ryon: CRO slash web design efforts is the UX UI by page type performance inside of Mixpanel. This is kind of our bread and butter, this is what we report on on a weekly basis, because all of our…
100 00:16:20.290 ⇒ 00:16:39.320 Ryon: all of our KPIs are basically going to be affected by what we do with each of the individual page types that we are improving, or intakes, or anything. So, we look at this mainly, and I… this is what I use our… to projector, to forecast, or to measure against our forecasts. Other stuff, we have product level.
101 00:16:39.460 ⇒ 00:17:01.660 Ryon: data here. This includes… includes drop-off data by funnel or by product, for each of our funnels, and then we can, of course, compare funnel to funnel or intake to intake. Here, we also get, data around, you know, what’s the session and the intake starts, what’s the CTR and conversion rate by product here. So this is more for me, just to kind of, like, assess, like, hey, how are we doing?
102 00:17:01.660 ⇒ 00:17:04.200 Ryon: Channel level performance, this is TBD.
103 00:17:04.200 ⇒ 00:17:09.910 Ryon: We still have to build this out. Persona performance, this is something that Joseph and I have been building out. So…
104 00:17:10.280 ⇒ 00:17:22.929 Ryon: We put people into two different cohorts based on either their purchasing habit or their navigational habit. You can see under here inside of the lexicon that I had Joseph a while ago, build out
105 00:17:22.970 ⇒ 00:17:31.179 Ryon: under, I believe it’s cohorts here, weight loss cohort, by purchase or by traffic.
106 00:17:31.370 ⇒ 00:17:32.529 Ryon: Okay, so…
107 00:17:32.910 ⇒ 00:17:47.959 Ryon: If you are a weight loss purchase cohort, that means you’ve purchased something which is a weight loss product, so we’re very certain you should be in that cohort. If you are a weight loss traffic cohort, that means that you navigated to a page about weight loss, so we sort of assume that you are in that weight loss cohort.
108 00:17:47.960 ⇒ 00:17:56.099 Ryon: This is obviously way more accurate, in my opinion, than this. Navigational habits for people is kind of not the best way to measure, but we have both just in case.
109 00:17:56.100 ⇒ 00:17:59.099 Ryon: And then, of course, we sort of measure them,
110 00:17:59.580 ⇒ 00:18:18.590 Ryon: calculate them out here by segment so we can see performance that way. The affiliate board, not gonna dive into that, but there is, of course, by affiliate that I look at, very regularly. So yeah, at a high level, that’s kind of what we’re doing and how we’re structuring things. If you’re able to help with different things, 100%, we’re like.
111 00:18:18.920 ⇒ 00:18:26.380 Ryon: Would love the assistance and support, but yeah, I’m gonna stop now, tell me what your thoughts are. Is this what you were looking for? Is this kind of what you were hoping to see?
112 00:18:26.820 ⇒ 00:18:39.760 Greg Stoutenburg: Yeah, this is great, this is fantastic, thank you for that overview. Yeah, so, you know, the… the point at which I came in is Henry’s leaving, I’m asked, you know, put together this experimentation roadmap for CRO if it’s… I’m sorry.
113 00:18:48.570 ⇒ 00:18:50.860 Greg Stoutenburg: So it’s a snow day, and
114 00:18:51.270 ⇒ 00:18:56.770 Greg Stoutenburg: Now, you watch live as I learned that my son’s ringtone is the T-Rex from Jurassic Park roaring.
115 00:18:57.110 ⇒ 00:19:00.980 Greg Stoutenburg: And I was like, oh, smoke. As soon as I started to answer you, I heard this roar from the other room, like…
116 00:19:01.320 ⇒ 00:19:02.239 Ryon: All good, it’s all good.
117 00:19:02.240 ⇒ 00:19:19.450 Greg Stoutenburg: I was corrected. Spinosaurus. Okay, all right, push away the distraction. So, yeah, so it looks like you, you, you have this roadmap already. So, as I, as I look at this, and I think about this, so my, my expertise is in product analytics, especially,
118 00:19:19.460 ⇒ 00:19:34.869 Greg Stoutenburg: user onboarding and engagement, retention. So, as I’m looking at all this and I’m thinking about, you know, where can I help add value for you all, two thoughts are the ones that immediately come to mind. Well, I mean, three. One is just, if you see a particular need, you want to point me at it.
119 00:19:35.730 ⇒ 00:19:38.040 Greg Stoutenburg: Say so, feel free.
120 00:19:38.160 ⇒ 00:19:46.699 Greg Stoutenburg: my mind jumps to, I can do things like look at user behavior across the intake to purchase funnel.
121 00:19:46.700 ⇒ 00:19:58.969 Greg Stoutenburg: And see where there are optimizations to be made. Now, I do… I’ve been told that these intakes are changing shortly, so, I’d want to make sure that, you know, I’m not spending a bunch of time on something you’re about to just overhaul anyway.
122 00:19:58.970 ⇒ 00:20:08.410 Greg Stoutenburg: And then the other being, just sort of auditing the roadmap that you have in place already. Like, does it… does it look like these are the places to be putting in, you know, those 1 to 3
123 00:20:08.410 ⇒ 00:20:20.150 Greg Stoutenburg: level efforts? Are there places where it looks like maybe a 5 is justified because it’s the kind of bet that would be, you know, that would really pay off? My mind goes in those directions.
124 00:20:20.650 ⇒ 00:20:34.069 Ryon: So, here’s where I need immediate help, and I think you sort of already got there. We need to… and I think Zaron is helping with this, but we need to build out Mixpanel to be way more than it currently is.
125 00:20:34.070 ⇒ 00:20:34.530 Greg Stoutenburg: Oh, yeah.
126 00:20:34.530 ⇒ 00:20:43.460 Ryon: really… and this is not meant to be a reflection of anybody, but, like, we are really falling short on how much Mixpanel could be, but how much we are using it for. Like.
127 00:20:43.460 ⇒ 00:20:43.790 Greg Stoutenburg: Yeah.
128 00:20:43.790 ⇒ 00:20:52.990 Ryon: Yes, I’ve definitely made some leaps and strides forward, but, like, it’s definitely not where it needs to be in terms of capabilities. Example, we have significant…
129 00:20:53.480 ⇒ 00:21:05.799 Ryon: amounts of, like, user data here and user information, you know, 6.7 million people that we, like, really don’t take advantage of this in understanding what stuff is doing for each and every one of our users or audiences.
130 00:21:05.800 ⇒ 00:21:08.680 Greg Stoutenburg: Yeah. That’s one example. Another example is, like.
131 00:21:09.050 ⇒ 00:21:15.809 Ryon: We have our byproduct funnels, right? But it would be manual.
132 00:21:15.810 ⇒ 00:21:35.220 Ryon: for me to then go in there and be like, hey, let’s see intake A versus intake B. Like, I can tell you how to do it inside of VWO, but, like, building out a table or a report that kind of shows, like, here’s the drop-off performance by individual step inside of each one of these funnels, like, that’s… that’s something we’ve definitely fallen short on, so…
133 00:21:35.220 ⇒ 00:21:41.129 Ryon: If you could help with that, that’d be great. I know that when… right before Henry left, we sort of…
134 00:21:41.370 ⇒ 00:21:54.360 Ryon: engaged a Mixpanel expert who did a full audit, his name’s Josh, I don’t know if Henry told you about this. It really wasn’t meant to be, like, a reflection on anybody, it was just for him to sort of take a neutral third-party look at Mixpanel and tell us, like, hey.
135 00:21:55.150 ⇒ 00:21:57.990 Ryon: You know, where do we fall in, like.
136 00:21:59.130 ⇒ 00:22:05.540 Ryon: the land… landscape? You know, what do you see as an opportunity? Is there anything we’re doing wrong? I really want to…
137 00:22:05.840 ⇒ 00:22:06.720 Ryon: kind of…
138 00:22:08.040 ⇒ 00:22:18.699 Ryon: start fresh with Mixpanel, like, the right way. Like, we’ve got data coming in from all over the place, but, like, there’s been 3 or 4 generations of people inside of this thing doing all this stuff, and I think it’s… it’s…
139 00:22:18.830 ⇒ 00:22:32.870 Ryon: it’s really, like, kind of complicated, and I think it’s also, like, stuff is not… it’s not gonna follow through the way that it needs to. So, in that way, I really could use your help here in a big way. We don’t want to duplicate work between you and Zaron, but, like, he’s also something.
140 00:22:32.870 ⇒ 00:22:33.330 Greg Stoutenburg: Yeah.
141 00:22:33.330 ⇒ 00:22:37.970 Ryon: We were talking to him about doing that. Second is…
142 00:22:40.590 ⇒ 00:22:54.179 Ryon: VWO needs to be better than it is. We… there are some tools inside of this where we sort of… there’s, like, roadmap tools, like the planning and observations, and that’s great. I just didn’t… the reason why I don’t use these tools really comes down to one thing.
143 00:22:54.610 ⇒ 00:23:10.110 Ryon: I don’t want to put all of our ideas in a place where we may not have this platform long-term. And I want it to be in a centralized place where everyone could sit and everyone had access to it. But at the same time, there are parts of VWO that I feel like we need to do better.
144 00:23:10.140 ⇒ 00:23:18.789 Ryon: Right? Like, we need to have better segments, we need to have better events, we need to have better attributes, or we need to be integrating them into our testing a little better.
145 00:23:18.850 ⇒ 00:23:23.540 Ryon: So, on that front, I definitely could use some help from you a little bit.
146 00:23:24.540 ⇒ 00:23:24.870 Greg Stoutenburg: Great.
147 00:23:24.870 ⇒ 00:23:30.929 Ryon: I talked to them about the personalization functionality, like, about 2 months ago, and I…
148 00:23:30.980 ⇒ 00:23:45.589 Ryon: unfortunately slightly been ghosting them about this. I need to sort of follow up on this. I don’t really know how much this matters, but I do want us to look at personalization more seriously, because we are starting to test it inside of intakes. We need to start testing it on the website, too.
149 00:23:45.730 ⇒ 00:23:49.510 Ryon: And getting an idea of things, but those are kind of the two areas where I definitely need help.
150 00:23:50.480 ⇒ 00:23:50.960 Greg Stoutenburg: Okay.
151 00:23:50.960 ⇒ 00:23:55.399 Ryon: If all sounds good. And then, of course, helping me to sort of gauge the value of stuff, like.
152 00:23:55.600 ⇒ 00:23:59.520 Ryon: I don’t know if you use ice at all, but, like, I like to measure things by.
153 00:23:59.520 ⇒ 00:24:00.120 Greg Stoutenburg: Yeah.
154 00:24:00.120 ⇒ 00:24:15.179 Ryon: by ice. That’s just kind of stuff. Danny and I sort of eyeball this, I would say at the moment, and I sort of kind of work, do some back-of-the-napkin math on what I think might end up happening if we were to succeed with this kind of a thing. Having a better…
155 00:24:16.150 ⇒ 00:24:20.289 Ryon: Methodology around gauging the potential impact of our…
156 00:24:21.060 ⇒ 00:24:31.249 Ryon: our proposed, you know, ideas, that would be really powerful, because we… we… I… I kind of do, but I also kind of don’t at the same time.
157 00:24:31.410 ⇒ 00:24:33.090 Ryon: So, yeah.
158 00:24:34.970 ⇒ 00:24:38.330 Ryon: Cool. Does that answer… does that give you kind of a direction, give you an idea?
159 00:24:38.850 ⇒ 00:24:52.219 Greg Stoutenburg: Absolutely. So, that’s great. I can help with all of those things. So I think for the things that have all… were already sort of handed over to me from Henry, probably those things will just be deprioritized. I see that for the…
160 00:24:52.430 ⇒ 00:25:02.509 Greg Stoutenburg: for a large part, they are duplicating efforts that you already have well underway here with Danny. So, I think maybe a good next steps approach could be, I can…
161 00:25:02.800 ⇒ 00:25:10.549 Greg Stoutenburg: I’ll summarize the conversation that we just had, and put together maybe, you know, a plan for us over the course of the next month.
162 00:25:10.650 ⇒ 00:25:25.230 Greg Stoutenburg: to start kicking the tires on some of these things, make some progress on, those three things that you just mentioned. I… I very much do see what you were talking about with MixedPanel, like, you… and that was… that was really the occasion for me reaching out in the first place. I’m like, I see some charts, but, like, they’re just…
163 00:25:25.390 ⇒ 00:25:30.979 Greg Stoutenburg: incomplete, right? I’m looking at things like intake started, and then it either ended it in a purchase, or it didn’t, right?
164 00:25:31.200 ⇒ 00:25:34.109 Greg Stoutenburg: that doesn’t really give you that much of a sense of where to go, right?
165 00:25:34.110 ⇒ 00:25:34.640 Ryon: We need more.
166 00:25:34.640 ⇒ 00:25:35.480 Greg Stoutenburg: mix panel.
167 00:25:35.480 ⇒ 00:25:43.179 Ryon: We need way more Mixpanel, but also, we need to divert away from just a page view world. Like, we need to have.
168 00:25:43.180 ⇒ 00:25:43.590 Greg Stoutenburg: Yeah.
169 00:25:43.590 ⇒ 00:25:50.819 Ryon: user analytics, we need to have product analytics, we need to have… Yeah. I would love, and Danny and I have to report on this, and I would say that
170 00:25:51.080 ⇒ 00:26:05.779 Ryon: we’ve fallen short on this to Mitesh. We need to be reporting on, like, page speed performance, and website performance, and things like that, and we have it inside of Cloudflare. It’s not in the best form, but we do have it, and we can report on it, but kind of…
171 00:26:07.430 ⇒ 00:26:21.009 Ryon: workshopping that, or, like, kind of working it into being a report format, I think, is really what would be necessary. If it was something which could be reported on inside of Mixpanel, fabulous. Like, the server-side performance metrics are largely, like.
172 00:26:21.010 ⇒ 00:26:33.779 Ryon: gone. They’re not in Mixpanel, and that really frustrates me, because that’s actually where Mixpanel shines, is getting the combination of what’s happening on the service side with what’s happening on the client side, and, like, bringing it all together. Right. Yeah. So yeah.
173 00:26:33.780 ⇒ 00:26:34.110 Greg Stoutenburg: Yeah.
174 00:26:34.110 ⇒ 00:26:49.220 Ryon: I think the best thing for us to do would be, you know, Danny and I normally would have our planning and brainstorm meeting, every other week. I think it would be a good idea to add you to that, so we can kind of keep the cadence going here, and then work through some stuff and keep that process going, and then…
175 00:26:49.580 ⇒ 00:27:09.339 Ryon: Again, I say this, I take full responsibility. Our first sprint was kind of a… not a loss, but it was definitely kind of a recalibration sprint. We need to get back to our planning, sprint planning meetings, to our grooming meetings, and making sure that we kind of get in the backlog and keep things going so that we can report on, you know, the velocity
176 00:27:09.340 ⇒ 00:27:11.430 Ryon: of our testing a little better. This is…
177 00:27:11.970 ⇒ 00:27:12.410 Greg Stoutenburg: Yeah.
178 00:27:12.410 ⇒ 00:27:35.999 Ryon: something where I think, leadership kind of understands at, like, a high level, like, sure, yeah, 66% of your tests ended up in a win, whereas 16% ended up ended up in a loser, like, and then you can filter by date, but, like, that’s not really telling them that much. We need to see, like, for all of the wins that we’ve been having, you know, what is the impression or conversion rate been, or what’s the intake CTR been? Like, can we correlate between these two?
179 00:27:36.000 ⇒ 00:27:44.049 Ryon: Right. Sort of see, like, yes, we are causing incremental lift, not, hey guys, we did X and it did Y. Like, that’s one dimension.
180 00:27:44.050 ⇒ 00:27:45.939 Greg Stoutenburg: It didn’t make anything worse this time.
181 00:27:45.940 ⇒ 00:27:54.990 Ryon: Yeah, and that’s… it’s very, like, I think that everyone understands there’s going to be a degree of experimentation with it, that’s why it’s called A-B testing, but also we need to sort of
182 00:27:55.270 ⇒ 00:27:58.489 Ryon: dot our I’s and cross our T’s a little bit more. So, yeah.
183 00:27:58.490 ⇒ 00:28:02.710 Greg Stoutenburg: Cool. Great. Yeah, that sounds great.
184 00:28:02.710 ⇒ 00:28:08.040 Ryon: Okay. Mitesh, any thoughts on this? Any feedback? Any… Yeah.
185 00:28:08.210 ⇒ 00:28:27.659 Mitesh Patel: No, this is good. Again, the goal being, you know, make sure Greg knows what we’re doing and how he can help. I think you accomplished that. I think Greg, sort of… this is, I think, you know, phase one, if you will. Phase two, then, and I’ll just kind of see the thought as you’re kind of thinking about it, is…
186 00:28:27.860 ⇒ 00:28:29.780 Mitesh Patel: How can we…
187 00:28:29.970 ⇒ 00:28:41.520 Mitesh Patel: you know, do… kind of get more, you know, smarter about, maybe more predictive, if you will, about our tests. Like, which ones can we predict could have a bigger impact?
188 00:28:41.610 ⇒ 00:28:52.860 Mitesh Patel: You know, how do we take learnings from our existing or historic tests and apply them for…
189 00:28:53.010 ⇒ 00:29:03.630 Mitesh Patel: you know, kind of going forward, right? Yeah. So that we take this entire… you know, just so we lift the entire framework, testing framework, and testing velocity.
190 00:29:04.400 ⇒ 00:29:04.930 Greg Stoutenburg: Yep.
191 00:29:05.170 ⇒ 00:29:08.380 Greg Stoutenburg: Yep, great. Yes.
192 00:29:08.560 ⇒ 00:29:20.969 Ryon: One more note as well. Danny and I are really working to sort of onboard support, resources that are going to allow us to, I think, take a step back and level up in being able to look at this stuff with you from that
193 00:29:21.180 ⇒ 00:29:29.700 Ryon: really curated, critical perspectives and understand exactly what’s going to make us move the needle and what’s not.
194 00:29:29.700 ⇒ 00:29:42.089 Ryon: you know, like, in the background, I think he and I both work on different campaigns, like, we know this needs to be worked on, we kind of, like, spell it out at least a little bit, but, like, we need to get into the, like, the nitty-gritties more, and let the execution be handled by people who can just, you know…
195 00:29:42.710 ⇒ 00:29:47.690 Ryon: deal with that while we kind of deal with the planning and the, and the data a little bit. Yeah.
196 00:29:48.350 ⇒ 00:29:48.880 Greg Stoutenburg: Yep.
197 00:29:49.090 ⇒ 00:29:51.340 Greg Stoutenburg: Yeah, cool. All sounds good.
198 00:29:51.940 ⇒ 00:29:52.790 Greg Stoutenburg: This is great.
199 00:29:53.530 ⇒ 00:29:55.209 Greg Stoutenburg: Yeah, cool.
200 00:29:55.420 ⇒ 00:30:01.910 Ryon: I may add you to our script planning meetings every other week, if you’re okay with that, just so we can kind of… Yeah, that’s a great idea. Does that work?
201 00:30:02.130 ⇒ 00:30:04.699 Greg Stoutenburg: Yeah, that’s a great idea, and if I could get access to Monday.
202 00:30:04.850 ⇒ 00:30:05.460 Ryon: Oh, yeah, yeah, yeah.
203 00:30:05.460 ⇒ 00:30:11.600 Greg Stoutenburg: even if it’s as a reader, you know, I won’t… I won’t go in and start changing everything, but to have that visibility would be great.
204 00:30:11.790 ⇒ 00:30:16.009 Ryon: Please do. Is that Adam P. Mitesh? No?
205 00:30:16.220 ⇒ 00:30:16.870 Ryon: Monday?
206 00:30:16.870 ⇒ 00:30:17.850 Mitesh Patel: Yeah. Yeah.
207 00:30:17.850 ⇒ 00:30:23.089 Ryon: Okay, I’ll connect you with Adam Palma, and he can get you access to that. Cool?
208 00:30:23.220 ⇒ 00:30:26.169 Greg Stoutenburg: Great. Awesome. Yeah, thanks for this overview, guys. Excited about it.
209 00:30:26.610 ⇒ 00:30:31.859 Danny Valdez: I have nothing to add, but I just want to say hi, nice to meet you, and… Hey, Danny, and Greg.
210 00:30:32.250 ⇒ 00:30:38.120 Danny Valdez: Appreciate you being able to follow along with that, like, onslaught of information that Ryan just gave you, so…
211 00:30:38.120 ⇒ 00:30:47.270 Ryon: Yeah, I talk a lot, and I sometimes just basically data dump, so please, if it’s, like, way too much, ask me questions, and, like, be like, hey, that was, like, like, I need to go back.
212 00:30:47.480 ⇒ 00:31:00.390 Greg Stoutenburg: No, that was, that was a very impressive, sort of, concentrated, like, ideas per word count, and word count per… per minute. Those… that was impressive.
213 00:31:00.390 ⇒ 00:31:02.750 Ryon: Never had anybody tell me that, but I’m taking a major positive.
214 00:31:02.750 ⇒ 00:31:05.499 Mitesh Patel: Ryan… Ryan didn’t even do intros, though.
215 00:31:05.740 ⇒ 00:31:06.420 Greg Stoutenburg: Yeah.
216 00:31:06.730 ⇒ 00:31:10.119 Ryon: I skipped over that, no one cares. Everyone knows everybody, like…
217 00:31:10.120 ⇒ 00:31:12.149 Mitesh Patel: Alright guys, I gotta jump, thank you.
218 00:31:12.590 ⇒ 00:31:14.209 Greg Stoutenburg: Yeah, Vitesh, yeah.
219 00:31:14.220 ⇒ 00:31:24.910 Greg Stoutenburg: Cool, great. Alright, so, so let’s do that, so you’ll add me to the meeting, add me to Monday. I’ll, I’ll kick around with these ideas. You identified 3 priorities.
220 00:31:24.910 ⇒ 00:31:37.020 Greg Stoutenburg: I’ll come up with maybe, like, a summary of ideal next steps, and then let’s see the right way to fit this into your, your Sprint 2 for the quarter. Please. And, map this out and get some work done.
221 00:31:37.200 ⇒ 00:31:48.539 Ryon: On that note, I didn’t… if I didn’t say it, the audit, I will share it with you, but execution will be the Brain Forge team’s responsibility, i.e. you. So…
222 00:31:49.720 ⇒ 00:31:55.469 Ryon: I sort of asked Josh, the auditor guy, like, what he thought. It’s like, can we, like.
223 00:31:55.770 ⇒ 00:32:03.029 Ryon: rescue this instance of Mixpanel, his perspective was, probably you want to start fresh. So…
224 00:32:03.030 ⇒ 00:32:04.190 Greg Stoutenburg: Interesting. Okay.
225 00:32:04.380 ⇒ 00:32:13.979 Ryon: I’m not, like, against that idea, we just kind of lose historical reference, so it’s a slow process of us, like, you know, incrementally moving over to a new world, but, like.
226 00:32:14.230 ⇒ 00:32:14.550 Greg Stoutenburg: Okay.
227 00:32:14.620 ⇒ 00:32:16.219 Ryon: We kind of do need to make it better.
228 00:32:16.880 ⇒ 00:32:17.460 Greg Stoutenburg: Yeah.
229 00:32:17.620 ⇒ 00:32:25.250 Greg Stoutenburg: Cool. Yeah, okay. That sounds good. Can I see what Josh’s findings were? I mean, a video.
230 00:32:25.250 ⇒ 00:32:28.679 Ryon: I don’t have him yet. He’s only shown us, like, he’s only shown.
231 00:32:28.680 ⇒ 00:32:29.040 Greg Stoutenburg: Okay.
232 00:32:29.040 ⇒ 00:32:30.560 Ryon: He hasn’t sent the audit.
233 00:32:30.560 ⇒ 00:32:31.160 Greg Stoutenburg: Okay.
234 00:32:31.160 ⇒ 00:32:37.580 Ryon: I just pinged and said, you’re not part of that channel? Let’s do this. Let’s make sure you’re a part of all necessary channels at this point.
235 00:32:37.580 ⇒ 00:32:38.159 Greg Stoutenburg: Sounds good.
236 00:32:38.160 ⇒ 00:32:42.160 Ryon: To be a part of… Yeah, you need to be a part of this channel.
237 00:32:50.980 ⇒ 00:32:54.910 Ryon: Okay, you also need to be a part of this channel here.
238 00:32:58.970 ⇒ 00:33:01.400 Greg Stoutenburg: Yup, cool. I’m seeing notifications come through.
239 00:33:01.740 ⇒ 00:33:04.140 Ryon: Yay, that’s make sure that was chill.
240 00:33:05.270 ⇒ 00:33:11.399 Ryon: Some of these are… Danny, is there a dedicated web channel for, like, web requests?
241 00:33:11.770 ⇒ 00:33:14.749 Ryon: Like, there is for my… like, that attaches to your board?
242 00:33:16.040 ⇒ 00:33:19.440 Danny Valdez: A… Oh.
243 00:33:19.730 ⇒ 00:33:26.529 Danny Valdez: on, on Slack, shoot, is there…
244 00:33:28.090 ⇒ 00:33:32.580 Danny Valdez: There may be, but it isn’t used super often. Let me double check here.
245 00:33:43.570 ⇒ 00:33:49.879 Danny Valdez: There’s a web requests board, yes, that just does automated Monday messages. I don’t know if that really…
246 00:33:50.070 ⇒ 00:33:53.789 Ryon: That’s basically what I was… that’s what CRO request does, too.
247 00:33:53.790 ⇒ 00:33:54.110 Danny Valdez: Yeah.
248 00:33:54.110 ⇒ 00:34:00.069 Ryon: So at least he sees it. But yeah, okay, so I’ve added to you to the CRO stuff, and to,
249 00:34:02.190 ⇒ 00:34:05.110 Ryon: Excuse me, Mixed panel VIPs.
250 00:34:05.110 ⇒ 00:34:05.600 Greg Stoutenburg: Cool.
251 00:34:05.600 ⇒ 00:34:07.520 Ryon: That’s where we were managing.
252 00:34:07.950 ⇒ 00:34:12.010 Ryon: All of the, information around…
253 00:34:12.670 ⇒ 00:34:18.599 Ryon: what Josh was uncovering. Let me add you to this channel, too. I don’t know how much you really are gonna need to be in this channel, but, like.
254 00:34:18.929 ⇒ 00:34:20.940 Ryon: Why not?
255 00:34:21.280 ⇒ 00:34:30.620 Ryon: This channel… Yeah, for now, the more information, the better. This channel is our Everything Intakes channel, it’s where I make announcements on the, the intakes that are going live, and, stuff that’s…
256 00:34:31.530 ⇒ 00:34:32.719 Ryon: being announced.
257 00:34:32.960 ⇒ 00:34:39.139 Ryon: I’ll get you accessed. I don’t know why it’s not letting me add you to this channel. I don’t think it’s… I think it’s coming down to owner. But I’ll add you to that channel.
258 00:34:39.550 ⇒ 00:34:43.870 Ryon: Thanks. I think that’s it. Anything else? Missing anything else? Nope.
259 00:34:45.330 ⇒ 00:34:46.100 Ryon: Cool.
260 00:34:46.310 ⇒ 00:34:47.190 Ryon: Alright.
261 00:34:47.420 ⇒ 00:34:51.349 Ryon: Really excited about this. Thanks, Tom. It was great. Alright, see ya.
262 00:34:51.350 ⇒ 00:34:53.560 Danny Valdez: Alright, see y’all. Bye. Bye.