Meeting Title: Experimentation Process Follow-up with Greg Date: 2026-02-10 Meeting participants: Greg Stoutenburg, Ryon
WEBVTT
1 00:08:57.830 ⇒ 00:09:08.239 Ryon: Hey, Greg. Sorry, I, yeah, the day has been full of meetings I did not anticipate, and then some that got rescheduled, and… Excuse me.
2 00:09:08.410 ⇒ 00:09:09.580 Greg Stoutenburg: Yep, yep. No, that’s right.
3 00:09:09.580 ⇒ 00:09:13.890 Ryon: I’m just now looking at your page here and requesting access to it.
4 00:09:13.890 ⇒ 00:09:14.490 Greg Stoutenburg: Nope.
5 00:09:15.130 ⇒ 00:09:16.760 Ryon: Sorry, I had not done that yet.
6 00:09:16.760 ⇒ 00:09:20.130 Greg Stoutenburg: Sorry about that. What did I do?
7 00:09:25.430 ⇒ 00:09:26.670 Greg Stoutenburg: Okay.
8 00:09:33.090 ⇒ 00:09:39.560 Greg Stoutenburg: Okay, alright, you should have access to it now. Yeah, so this is the follow-up for when we have a conversation,
9 00:09:39.880 ⇒ 00:09:48.979 Greg Stoutenburg: last week or the week before about, ways to just sort of tighten up the improvement… the experimentation process. So, I sort of just put together this brief
10 00:09:49.260 ⇒ 00:09:55.160 Greg Stoutenburg: with some recommendations based on what I saw. And so, like.
11 00:09:55.360 ⇒ 00:10:09.009 Greg Stoutenburg: Bottom line, main recommendations are, one, to shift to Rice over ICE or Pi. The reason being just to make sure that we’re including the number of users or sessions that would be touched by a change.
12 00:10:09.010 ⇒ 00:10:09.410 Ryon: Yeah.
13 00:10:09.410 ⇒ 00:10:14.370 Greg Stoutenburg: way to quantify, you know, what the overall result is going to be.
14 00:10:14.370 ⇒ 00:10:27.509 Ryon: For background there, I’ve used rice in every other job. Certain people here before the other two. I don’t. It’s just a preference, I guess. But I like rice, so I’m just gonna adopt it across the board anyway. There’s a few things where I just kinda need to make, like, an executive decision and say, yep, this is happening.
15 00:10:27.780 ⇒ 00:10:46.029 Greg Stoutenburg: Yep, yeah, yeah, that sounds good, and yeah, right, and, you know, for all this, like, let’s just talk through it, figure out which direction we want to go in, but, like, if there’s internal resistance to including the R, let’s just include a column that says how many users would be touched by this, right? And then just put the number. Same thing.
16 00:10:46.060 ⇒ 00:10:53.810 Greg Stoutenburg: tying every proposed experiment to an OKR. There was a note in there about wanting to do something like this. I think it’s just time to standardize it.
17 00:10:54.150 ⇒ 00:11:10.539 Ryon: Yeah, I need to go through, again, not an excuse, a lot of this is stuff I know I need to do. I need to go in there and revamp the boards entirely. We have sort of allowed them to adapt or evolve into a state which is both informative and useful, but not quite what they need to be.
18 00:11:10.540 ⇒ 00:11:20.890 Ryon: This is one of those examples where it just needs to be… every OKR per quarter has a identifier, and we are targeting that OKR this quarter, basically. Yep.
19 00:11:20.890 ⇒ 00:11:37.190 Greg Stoutenburg: Yep, yeah, exactly. And then, down at the bottom, like, for… for next steps and for, like, what to do with the Monday board, I’m gonna suggest some changes to the Monday board structure to just go, like, just have a drop-down for that, you know? Have a column for reach, and so on, and the overall score.
20 00:11:37.190 ⇒ 00:11:41.080 Greg Stoutenburg: And then takeaways at the conclusion of an experiment.
21 00:11:41.080 ⇒ 00:11:54.570 Greg Stoutenburg: small changes, mostly. We mentioned this the other day, right? Dedicating… I said 20%, this is 1 out of 5, right? In any individual sprint. Pick something that, if it worked out, would be a bigger win. And,
22 00:11:54.640 ⇒ 00:11:55.360 Greg Stoutenburg: you know.
23 00:11:55.780 ⇒ 00:12:02.040 Greg Stoutenburg: the most important OKR at the time. That’s just to say, pick something with an impact score that’s, like, a 2 or a 3.
24 00:12:02.450 ⇒ 00:12:03.100 Ryon: So…
25 00:12:03.100 ⇒ 00:12:03.779 Greg Stoutenburg: Yeah, for a one.
26 00:12:04.030 ⇒ 00:12:08.569 Ryon: Let me share my screen real quick. I want to walk through this because… with you, if you don’t mind.
27 00:12:08.570 ⇒ 00:12:08.960 Greg Stoutenburg: Yep.
28 00:12:08.960 ⇒ 00:12:17.100 Ryon: Historically, what Danny and I have tried to do is, as we have… he should actually probably be in on this meeting.
29 00:12:17.650 ⇒ 00:12:19.920 Ryon: I’ll update him later.
30 00:12:20.580 ⇒ 00:12:30.269 Ryon: So, as we’ve gone through the process of kind of scheduling a sprint and figuring things out, if you look at effort here, this column, we always have tried.
31 00:12:30.270 ⇒ 00:12:31.679 Greg Stoutenburg: You’re not sharing.
32 00:12:31.940 ⇒ 00:12:33.650 Ryon: I’m not sharing? How am I not?
33 00:12:37.060 ⇒ 00:12:39.570 Ryon: Share entire screen, yes.
34 00:12:39.570 ⇒ 00:12:41.800 Greg Stoutenburg: Okay, now I see it. I don’t know what may have happened.
35 00:12:42.830 ⇒ 00:13:02.369 Ryon: This column here, effort, 1 being the least, 5 being the most. We’ve always tried to include a 4, 1, 4, or 5 in each sprint that we, target. And of course, the process of doing that is kind of discussing what it’s going to take. Essentially, if you imagine for a second a 1 being just equivalent to a day’s worth of work.
36 00:13:02.370 ⇒ 00:13:06.729 Ryon: Right, so 1 would be one day, two would be two days, that’s kind of how we figured it out.
37 00:13:06.730 ⇒ 00:13:12.070 Ryon: The problem I’ve had with this is, right now, Danny and I really only have the bandwidth to really handle about one
38 00:13:12.310 ⇒ 00:13:26.969 Ryon: 4 or 5 score effort. But, that leads me to why I presented this. What resources can you guys help with this? Because I’m not above asking for help if I need it, and to be honest with you right now, we’re in the process of onboarding a new web dev.
39 00:13:26.970 ⇒ 00:13:43.170 Ryon: We are in the process of onboarding, or gonna be onboarding a new CRO, and we’re gonna be looking for an intake specialist. All these people are bringing great for execution, helping to monitor or manage things. Yeah. But if we can work together, you can take on some testing, some deployments, some ideas. I’m not against that idea.
40 00:13:43.170 ⇒ 00:13:43.960 Greg Stoutenburg: Yeah, yeah.
41 00:13:44.280 ⇒ 00:13:54.670 Greg Stoutenburg: Yeah, no, that sounds good, and let me… let me check in internally with what we have for resources. Like, we’ve got some front-end folks and stuff, so, you know, we can do at least some amount of design and implementation. So.
42 00:13:54.670 ⇒ 00:14:06.020 Ryon: You can also leverage Nick, he’s front-end web dev, UI, UX, intake web dev as well. We’re gonna be, you know, utilizing designers, like, whatever’s necessary, yeah.
43 00:14:06.290 ⇒ 00:14:24.139 Greg Stoutenburg: Yeah, that sounds good. And I do want to clarify, like, when I’m talking about impact, I mean, like, something that would be expected to make a big improvement, so even independent of effort, something that would be expected to make a big improvement, if it went the way we wanted to. So, when we were talking the other day, so a lot of what gets prioritized is those.
44 00:14:24.140 ⇒ 00:14:31.799 Greg Stoutenburg: Experiments that are low effort, because they won’t be hard to implement, but also tend to have a low impact, but maybe a wider reach.
45 00:14:31.800 ⇒ 00:14:42.129 Greg Stoutenburg: The suggestion I’m trying to make here is, you know, if you don’t make room for those ones that seem like they’re bigger bets, they just won’t make their way onto the roadmap, because it’s pretty safe to just make small iterations.
46 00:14:42.130 ⇒ 00:14:52.189 Greg Stoutenburg: And then, you know, a half a percent or a 1% increase, like, looks great. So I think… I think we want to make some room in the roadmap for those, you know, higher impact experiments as well.
47 00:14:52.580 ⇒ 00:15:08.939 Ryon: Yeah, we need to, and then I think right now, one of the biggest things that we sort of are falling short on is… well, we have two areas. First, resources. Second, we need more ideas, right? So, like, as an example, I want to be… I want to actually have the resources available to, pursue
48 00:15:09.220 ⇒ 00:15:11.100 Ryon: Things that are, like.
49 00:15:12.410 ⇒ 00:15:36.670 Ryon: big bets. Yeah. Like, we’re doing this whole thing right now with this, like, this pre-intake page, which is fine, and I’m kind of curious to see how it turns out, but I want to look across the competitor landscape, I want to look across the, web landscape, and I want to try bigger bets. Stuff I’ve known from the past, stuff I’ve done in the past that I know I’ve done really well, and track some of those big bets here. If we can get the resources in place to do that, then, yeah, no objection.
50 00:15:36.670 ⇒ 00:15:53.230 Greg Stoutenburg: to get that done. Yeah, and that sounds great. So maybe, maybe one of the follow-ups from this will be, to jot some of those down on the Monday board, and then we can go, alright, what resources are actually needed to do this? All right, given the constraints we know about, let’s just…
51 00:15:53.230 ⇒ 00:15:56.850 Greg Stoutenburg: Let’s find a way to tackle 3 of these in the next 6 months.
52 00:15:56.890 ⇒ 00:16:02.960 Greg Stoutenburg: Even, you know, even something like that, even if it’s not a lot, just to make sure that we do actually budget for those things you’re talking about.
53 00:16:03.350 ⇒ 00:16:05.340 Ryon: Maybe. Okay. Yeah, I agree. I agree.
54 00:16:05.340 ⇒ 00:16:09.090 Greg Stoutenburg: Alright, sounds like you don’t need to be convinced of this one, basically.
55 00:16:09.090 ⇒ 00:16:11.079 Ryon: No, no, no, I’m not going to Disney.
56 00:16:11.080 ⇒ 00:16:21.649 Greg Stoutenburg: We’ll just, we’ll just skip the rice stuff, and then, you know, but maybe others internally would like to reference this as, like, a justification for why we would include REACH.
57 00:16:21.800 ⇒ 00:16:26.010 Greg Stoutenburg: something that I try to demonstrate here, as far as, like, how to score these things, is
58 00:16:26.090 ⇒ 00:16:44.940 Greg Stoutenburg: you can be principled without being so overly picky that you start just saying, oh, this is too much work, I don’t want to do it this way, right? You know, if you look in, you know, well, in a couple weeks or so. If you look at Mixpanel, it’s like, alright, well, it looks like this many users get to this page, you propose an experiment on that page.
59 00:16:44.940 ⇒ 00:16:51.800 Greg Stoutenburg: That number of users is the number that constitutes the reach score for any experiment that you do there. You know, things… things as simple as that.
60 00:16:53.100 ⇒ 00:17:00.839 Greg Stoutenburg: Something I want to run by you, related to effort, since you just mentioned this one, is, intercom, and so I took…
61 00:17:01.220 ⇒ 00:17:16.090 Greg Stoutenburg: again, another resource you can share internally is… I use… I shared here Intercom’s guidance for applying RICE. They use the concept of a person week, or a person month, as the amount of work that one person could do full-time inside of a month.
62 00:17:16.560 ⇒ 00:17:20.909 Greg Stoutenburg: Which, you know, that’s approximately…
63 00:17:21.109 ⇒ 00:17:24.489 Greg Stoutenburg: 80 hours, so a 1 equals, like, 80 hours of work.
64 00:17:24.640 ⇒ 00:17:38.679 Greg Stoutenburg: that’s kind of a lot, you know, for some of these changes. So, something that, something that we could talk about or decide on is, like, if you want to go with, like, story points, or maybe person weeks instead, since you run two-week cycles,
65 00:17:38.920 ⇒ 00:17:41.930 Greg Stoutenburg: That’s something that we could… decide on.
66 00:17:43.070 ⇒ 00:17:46.940 Ryon: Definitely, do you know story points?
67 00:17:48.410 ⇒ 00:17:58.820 Ryon: used that in the past, usually try to calculate the total number of hours or effort that’s put into a single thing as a way of sort of measuring or trying to estimate what the amount of time it is to get something done.
68 00:17:58.980 ⇒ 00:18:02.510 Ryon: But I like rice better. Yeah. Cool.
69 00:18:02.900 ⇒ 00:18:06.419 Greg Stoutenburg: Yeah, yeah, and then, like, for effort, I’ve seen…
70 00:18:06.600 ⇒ 00:18:08.170 Greg Stoutenburg: I mean, you can really just pick.
71 00:18:08.290 ⇒ 00:18:15.450 Greg Stoutenburg: you just get to decide what 1 equals, as long as it’ll scale, right? So, like… I’ve seen…
72 00:18:16.280 ⇒ 00:18:21.470 Greg Stoutenburg: I’ve seen the effort score be that 1 equals, like, an engineer’s day.
73 00:18:21.670 ⇒ 00:18:32.349 Greg Stoutenburg: So then something that, you know, gets up to 8 is a pretty big lift for someone. That’s gonna be, you know, if you’re running a one-month sprint, anything that’s an 8 is gonna be
74 00:18:32.520 ⇒ 00:18:39.329 Greg Stoutenburg: you know, a third of their time for an entire month. So, let’s just pick a number, and decide what it means, and make it official.
75 00:18:39.440 ⇒ 00:18:49.650 Greg Stoutenburg: The OKR stuff, I already talked about this, right? Recommend just adding a column with dropdowns for the OKRs that you want to be executing against.
76 00:18:50.000 ⇒ 00:18:54.680 Greg Stoutenburg: And then, finally, some new columns to add to the Monday board.
77 00:18:55.760 ⇒ 00:18:58.840 Greg Stoutenburg: pertaining to OKRs, reach, impact, confidence.
78 00:18:59.080 ⇒ 00:19:07.229 Greg Stoutenburg: overall rice score, and then insights. I think it’ll be worth the… doing the exercise to actually come up with the number.
79 00:19:07.580 ⇒ 00:19:09.419 Greg Stoutenburg: And put it on the board.
80 00:19:09.730 ⇒ 00:19:23.980 Ryon: We need to be doing this more in the planning meetings. What I want to do is, rather than… so, again, a lot of the stuff that Danny and I’ve been working on has been onboarding new people and trying to, like, hold the line where we can. We need to get back away from that towards the administrative side of things, and, like, these meetings need to be…
81 00:19:24.420 ⇒ 00:19:27.370 Ryon: Essentially, hey,
82 00:19:28.270 ⇒ 00:19:35.940 Ryon: we rice score this, everyone agrees on the rice score, okay, what are the things that we think are going to be the most impactful, move them into the sprint kind of thing, you know what I mean? Yeah.
83 00:19:36.280 ⇒ 00:19:52.299 Greg Stoutenburg: Yeah, I think operationally, you know, I know items go on the board when someone submits something through the intake, or someone has an idea, you just put it on the board. I think it could be a valuable exercise to do something like, an hour a week, go through what’s been added to the board in the previous week.
84 00:19:52.300 ⇒ 00:20:08.749 Greg Stoutenburg: put that right score on there, any other documentation pertaining to the experiment, and then during the planning meeting, you know, anything that hasn’t been filled in, we can fill it in there. But the goal would be that, for the most part, they are filled in, and then we go, alright, well, now let’s just rank these things, and…
85 00:20:08.750 ⇒ 00:20:09.990 Greg Stoutenburg: Put them in a sprint.
86 00:20:11.920 ⇒ 00:20:12.570 Ryon: Okay.
87 00:20:12.860 ⇒ 00:20:13.670 Ryon: Sounds good.
88 00:20:13.850 ⇒ 00:20:14.690 Greg Stoutenburg: Cool.
89 00:20:14.950 ⇒ 00:20:18.560 Greg Stoutenburg: For the…
90 00:20:18.670 ⇒ 00:20:38.580 Greg Stoutenburg: Yeah, so for anything that we want to put forward as an experiment, we just want to make sure to include all these. You do, in different places, include all these. So this is another one of those ones that’s just like, hey, for documentation purposes, let’s just try to be consistent on these things. And then, yeah, I mean, finally, next steps. So, as far as effort measurement.
91 00:20:38.860 ⇒ 00:20:45.000 Greg Stoutenburg: Let’s… we can… we can do it now, or we can do it later. Let’s just agree on the meaning of one effort, or two effort.
92 00:20:45.620 ⇒ 00:20:50.380 Greg Stoutenburg: Do we want to say it’s someone’s day? Do we want to say it’s, you know, approximately 8 hours of work?
93 00:20:51.030 ⇒ 00:20:55.000 Ryon: I would just say 8 hours of work is one. That’s how I’m gonna calculate it myself.
94 00:20:57.420 ⇒ 00:20:58.360 Ryon: Yep, yep, yep.
95 00:20:58.360 ⇒ 00:21:04.290 Greg Stoutenburg: So, if 4 people put in 2 hours each, that’s still 1 point. If 1 person puts in 8 hours, that’s 1 point.
96 00:21:05.350 ⇒ 00:21:06.230 Greg Stoutenburg: Yup.
97 00:21:06.800 ⇒ 00:21:10.769 Greg Stoutenburg: Revised Monday board with new fields, I can do it or you can do it, or we can ask someone else to do it.
98 00:21:10.770 ⇒ 00:21:15.919 Ryon: I’m gonna… I’m gonna do it. I gotta do that for various reasons, various other boards. This is necessary.
99 00:21:16.220 ⇒ 00:21:16.920 Greg Stoutenburg: Okay.
100 00:21:17.130 ⇒ 00:21:20.549 Greg Stoutenburg: Review existing roadmap to ensure tickets are filled in completely.
101 00:21:20.820 ⇒ 00:21:23.329 Greg Stoutenburg: I can do this, you can do this, we can split it up.
102 00:21:24.450 ⇒ 00:21:25.610 Greg Stoutenburg: We could schedule a call.
103 00:21:25.610 ⇒ 00:21:29.240 Ryon: Let’s, let’s, let’s, let’s go through it together. Okay.
104 00:21:29.430 ⇒ 00:21:30.710 Ryon: Let’s go through it with Danny.
105 00:21:31.360 ⇒ 00:21:34.440 Ryon: But yeah, let’s do this together. Schedule a call, like, 30 minutes.
106 00:21:34.570 ⇒ 00:21:35.170 Greg Stoutenburg: Cool.
107 00:21:35.500 ⇒ 00:21:39.599 Greg Stoutenburg: Perfect. Yeah, that’s it. So,
108 00:21:39.860 ⇒ 00:21:46.549 Greg Stoutenburg: Any other follow-ups on this, or you can share it internally, and we’ll take this as the direction that we want to go in for…
109 00:21:46.550 ⇒ 00:21:54.989 Ryon: share this with Danny Valdez and myself, we’ll take a look at this, and he and I can start working on some of the stuff, and then let’s schedule a meeting for, like, Friday morning, the three of us.
110 00:21:54.990 ⇒ 00:21:55.660 Greg Stoutenburg: Sounds good.
111 00:21:55.660 ⇒ 00:21:56.689 Ryon: Adapting to it.
112 00:21:56.690 ⇒ 00:22:00.550 Greg Stoutenburg: Yep. Cool. Alright, let’s do it. Cool, that’s it. Alright, thanks, man.