Meeting Title: Product Analytics Mapping Review Date: 2025-09-04 Meeting participants: Uttam Kumaran, Henry Zhao, Amber Lin, Demilade Agboola, Shreya Chowdhury
WEBVTT
1 00:00:44.030 ⇒ 00:00:45.100 Henry Zhao: Hey, with them.
2 00:00:47.800 ⇒ 00:00:48.440 Uttam Kumaran: Hello.
3 00:00:49.430 ⇒ 00:00:50.530 Henry Zhao: My number?
4 00:00:52.290 ⇒ 00:00:56.500 Amber Lin: Hi, I just talked to Shreya, so she should be joining soon.
5 00:01:03.520 ⇒ 00:01:17.430 Amber Lin: Mostly wanted to use this meeting to talk about the different product analytics possibilities, and then to sort of divide up tasks of who’s gonna own the playbooks for which area.
6 00:01:18.250 ⇒ 00:01:23.410 Uttam Kumaran: Another thing, Amber, we could also go through is I’ve made some progress on our services.
7 00:01:23.620 ⇒ 00:01:27.320 Uttam Kumaran: We can also talk about the product and what the service is.
8 00:01:27.320 ⇒ 00:01:28.070 Amber Lin: Oh, yeah.
9 00:01:28.070 ⇒ 00:01:30.870 Uttam Kumaran: One or many playbooks per service, right?
10 00:01:30.870 ⇒ 00:01:31.570 Amber Lin: Yes.
11 00:01:31.570 ⇒ 00:01:33.739 Uttam Kumaran: But yeah, I’ll talk about those.
12 00:01:34.000 ⇒ 00:01:34.720 Amber Lin: Okay.
13 00:01:35.410 ⇒ 00:01:36.570 Amber Lin: Sounds good.
14 00:01:36.960 ⇒ 00:01:38.000 Amber Lin: Bye!
15 00:01:39.410 ⇒ 00:01:40.960 Shreya Chowdhury: Hey, how’s it going?
16 00:01:42.390 ⇒ 00:01:44.160 Amber Lin: Good.
17 00:01:44.290 ⇒ 00:02:01.079 Amber Lin: I think we can get started. We want to talk about the different playbooks we can make, the different services that we have, and then divide up the tasks. Shreya, do you have a document you can share with us, and then we can all work on that together in this meeting?
18 00:02:01.390 ⇒ 00:02:07.299 Shreya Chowdhury: Yeah, so, I’m gonna drop two links in the chat.
19 00:02:07.430 ⇒ 00:02:24.690 Shreya Chowdhury: One of them, is the template for the A-B testing that I made, and then I made some edits to that one, and then I also made a playbook, or, like, a table of contents, sort of, with,
20 00:02:25.830 ⇒ 00:02:27.510 Shreya Chowdhury: the other…
21 00:02:27.700 ⇒ 00:02:41.459 Shreya Chowdhury: the other product analytics templates we can have, and then… I haven’t had a chance to map it to services, so I would really appreciate some, comments here, and, like, know if this is the right route to go.
22 00:02:42.210 ⇒ 00:02:47.300 Shreya Chowdhury: Let me just find… Boom, wink.
23 00:03:27.420 ⇒ 00:03:40.000 Shreya Chowdhury: Okay, let me know if you guys are able to view this. But basically, I just intend to kind of make this more like a table of contents, and then it’ll link to each of the individual pages.
24 00:03:44.980 ⇒ 00:03:52.439 Shreya Chowdhury: And it just has, like, the different, the different sort of analyses that we can do.
25 00:04:06.910 ⇒ 00:04:10.509 Uttam Kumaran: So, today we’re just going to be reviewing…
26 00:04:10.940 ⇒ 00:04:15.770 Uttam Kumaran: The first one is, should we take some time for everyone to add comments, or, like, what’s the best…
27 00:04:16.380 ⇒ 00:04:28.670 Shreya Chowdhury: Yeah, I was gonna say, maybe we can all take, like, five, if you guys want to read through, like, one, the playbook. So I added, like, a few different types of analyses that I think are most important. Obviously, there are other ones,
28 00:04:28.990 ⇒ 00:04:44.549 Shreya Chowdhury: if we feel like there are other ones that are more important, we can swap some of these out, or if this isn’t the best way to go about it, that’s totally fine too. And then we can also go through the AB1. I made some edits and added, like, a little bit more… just tried to make it more robust, and added, like.
29 00:04:44.850 ⇒ 00:04:50.700 Amber Lin: Visuals and whatever, so maybe we can take 5, and then once everyone’s done, we can walk through it together.
30 00:04:54.220 ⇒ 00:04:56.029 Uttam Kumaran: I’ll sleep bye.
31 00:04:56.820 ⇒ 00:04:57.440 Shreya Chowdhury: Okay.
32 00:09:46.530 ⇒ 00:09:51.910 Uttam Kumaran: Sorry, this is really good, I’m just taking my time.
33 00:09:51.910 ⇒ 00:09:53.210 Shreya Chowdhury: Yeah, no worries.
34 00:12:11.900 ⇒ 00:12:14.529 Henry Zhao: Yeah, honestly, I think it’s very thorough.
35 00:12:14.870 ⇒ 00:12:16.519 Henry Zhao: Don’t know if there’s anything I would add.
36 00:12:19.000 ⇒ 00:12:28.009 Uttam Kumaran: Yeah, probably the only overarching thing, and I think this is something that I’m just gonna be a little pain about, is, on how we communicate.
37 00:12:28.330 ⇒ 00:12:41.519 Uttam Kumaran: I think as part of the playbook, a key part, for us to indicate in each area is how it gets communicated to various stakeholders, whether it’s a technical stakeholder, like another data team person.
38 00:12:41.520 ⇒ 00:12:53.870 Uttam Kumaran: Whether it is the executive team, I think from a, like, from the engineering standpoint, this is great. Like, I think, like, I would love if we can get into this much detail for our clients. This is tremendous.
39 00:12:53.870 ⇒ 00:13:07.469 Uttam Kumaran: I think probably the only pushback, and again, this is something you’re gonna hear from me, is, like, does this end up in a slide? What is the format of the slide? What, like, what decisions do we need from the executive team? Because…
40 00:13:07.760 ⇒ 00:13:24.619 Uttam Kumaran: we don’t… we’re not gonna… we’re coming in as outsiders, so we’re not gonna have, like, they’re just… they may not have ever done this before. They may have done this and maybe got burned by past AVTS, like, screwed something up, so I just want to make sure we’re mindful of that.
41 00:13:25.030 ⇒ 00:13:32.299 Uttam Kumaran: But yeah, I mean, what does that mean? That means, like, okay, maybe we just need to send, like, a template slide deck, or, like, a,
42 00:13:32.610 ⇒ 00:13:38.380 Uttam Kumaran: quick, like, blurb on, like, what… how this is gonna impact, but this is lovely otherwise. Okay.
43 00:13:38.380 ⇒ 00:13:52.740 Shreya Chowdhury: Yeah, sounds good. I feel like one thing that would be nice to add here is maybe, like, we can link to a sample slide deck and, like, present, like, the results of this if we were gonna present them. And I can… I can make time to add that later.
44 00:13:57.340 ⇒ 00:13:58.030 Shreya Chowdhury: Yeah.
45 00:13:59.090 ⇒ 00:14:04.979 Shreya Chowdhury: And then, I think there’s a couple of comments, that I didn’t get a chance to go through, but I’ll…
46 00:14:05.080 ⇒ 00:14:08.629 Shreya Chowdhury: I’ll go through them and then clean up the…
47 00:14:09.130 ⇒ 00:14:12.100 Shreya Chowdhury: Clean up the template based on any edits.
48 00:14:13.240 ⇒ 00:14:25.959 Uttam Kumaran: Yeah, so I guess, like, my question also is, like, in the past, like, how long do these typically take and demand? Like, what is the sort of time… what is the time duration on some of these, and…
49 00:14:25.960 ⇒ 00:14:27.460 Shreya Chowdhury: Yeah, so it depends.
50 00:14:27.460 ⇒ 00:14:31.890 Uttam Kumaran: on, I guess, what the experiment is, and.
51 00:14:31.890 ⇒ 00:14:42.960 Shreya Chowdhury: like, the urgency of it. So, like, I’ve… I’ve worked on experiments that ran for 2 weeks, and then there’s ones that ran for a month, and then there’s ones where it was, like.
52 00:14:43.500 ⇒ 00:15:01.889 Shreya Chowdhury: it, like, not that it wasn’t a priority, but it was just something that, like, we would like to know for the future, but it was such a lower priority thing, and we wanted, like, really thorough results that we let it run for 3 months. I figure for most of our clients, like, we’re not going to be running experiments for 3 months, because they’ll want results sooner than later.
53 00:15:01.890 ⇒ 00:15:05.760 Shreya Chowdhury: But I figure, like, depending on
54 00:15:05.790 ⇒ 00:15:10.349 Shreya Chowdhury: that’s, like, a very case-by-case basis. I guess it would have to be, like,
55 00:15:10.610 ⇒ 00:15:20.700 Shreya Chowdhury: figure out, depending on, like, what the sample size is, like, when they want to roll out or need a decision by, so I would say anywhere between 2 weeks to a month is what I’d expect.
56 00:15:20.940 ⇒ 00:15:22.830 Shreya Chowdhury: For most of our cases.
57 00:15:22.830 ⇒ 00:15:38.269 Uttam Kumaran: Okay, makes sense. I guess my other question is about, like, sort of, like, what the rituals are going on for executing a task, and again, like, put yourself in the position of a medium-sized business that is, like.
58 00:15:38.550 ⇒ 00:15:52.730 Uttam Kumaran: not doing a great job on data, like, is there… okay, we propose a… a test, there’s, like, maybe a meeting with, like, the front-end folks, like, the… whoever our direct stakeholders, and basically get approval to run this test.
59 00:15:52.930 ⇒ 00:16:09.969 Uttam Kumaran: we run it, we then have, like, some time period for a measurement, something that probably can happen in parallel, and then there’s, like, a closeout. I almost want to get… like, what we’re gonna have trouble with is there’s gonna be a lot of, like, nuances and, like, edge cases here, but I almost want to get
60 00:16:10.280 ⇒ 00:16:11.949 Uttam Kumaran: If we were to have, like.
61 00:16:12.790 ⇒ 00:16:26.430 Uttam Kumaran: if we were to just literally hand this document to somebody and be like, run this test, I would almost want to have, like, every… like, what are the key steps? Do you think that that roughly outlines how it would go with a client? Like, those rituals?
62 00:16:28.750 ⇒ 00:16:40.190 Shreya Chowdhury: So, like, if we were to hand this template to someone and then give them a case, like, would they be able to summarize the results of the experiment from this doc? Is that what you’re asking?
63 00:16:40.190 ⇒ 00:16:58.070 Uttam Kumaran: I guess what we’re asking is, like, what are the key, like, communication or touchpoints that we need to have? Like, these can be internal meetings, these can be meetings with the client, but, like, in order to execute an end-to-end A-B test, like, what are the different meetings or touchpoints that we need to have, basically?
64 00:16:59.070 ⇒ 00:17:17.609 Shreya Chowdhury: Yeah, so I… off the top of my head, I would think, like, one, like, introductory meeting, just to get an idea of, like, what their needs are, like, what… like, just a… in very plain English, like, we’d get an idea of what their main questions are, what they’re trying to optimize for, etc, and then probably, like.
65 00:17:17.609 ⇒ 00:17:23.040 Shreya Chowdhury: A second, like, internal meeting just to,
66 00:17:23.250 ⇒ 00:17:36.000 Shreya Chowdhury: narrow down the scope and, like, translate the plain English to data, like, what inputs we would need, what metrics we’re gonna look at, and then design the experiment.
67 00:17:36.880 ⇒ 00:17:45.959 Shreya Chowdhury: And then, probably, like, more stand-ups throughout, just to, like, show that, hey, like, this has been implemented. We can have, like, a mid-experiment check-in just to see, like.
68 00:17:45.960 ⇒ 00:17:47.040 Uttam Kumaran: Oh, like…
69 00:17:47.940 ⇒ 00:18:07.029 Shreya Chowdhury: this is what we’re seeing, if it’s one of those experiments where you could look at the results early, otherwise, like, we would just present them at the very end. But I can… I can make a section adding that. I think that also differs a little bit, case by case, but… Yeah. I can add a section with that.
70 00:18:07.320 ⇒ 00:18:17.869 Uttam Kumaran: Yeah, two weeks to a month seems fair, and then, yeah, I think exactly that, because what this will allow is it’ll allow our project management team to basically pre-book those, and, like, they’re gonna be running
71 00:18:18.000 ⇒ 00:18:32.870 Uttam Kumaran: getting those people in the room. The one thing that I just want to avoid is that, like, hey, we’re running this experiment, we run it, maybe, like, we forget about it, or, like, it runs, but, like, nothing happens, but then they’re like, did we run this? Like, we don’t… we just…
72 00:18:33.030 ⇒ 00:18:38.790 Uttam Kumaran: We’re coming in externally, we just have to be really crystal clear, especially for when we propose running this.
73 00:18:38.890 ⇒ 00:18:48.609 Uttam Kumaran: common objections could be, okay, like, when are we gonna meet to discuss results? How long will results take? Is this gonna impact more than just this feature?
74 00:18:49.210 ⇒ 00:18:55.360 Uttam Kumaran: Right? Stuff like that. So, yeah, that’s… that’s probably, again, more biased on the communications.
75 00:18:55.640 ⇒ 00:18:56.690 Demilade Agboola: That’s awesome.
76 00:18:57.560 ⇒ 00:19:00.390 Demilade Agboola: Something to also consider is, like.
77 00:19:00.560 ⇒ 00:19:12.999 Demilade Agboola: having… I won’t just say, like, in-depth, like, technical stuff, but just an idea of how, like, some of the technical things behind it might work. Because, like, just, you know, how I would have, like, two different, you know.
78 00:19:13.140 ⇒ 00:19:22.980 Demilade Agboola: web pages loading for two different, like, cohorts, for instance. And how that data should be stored for optimal use could also be very helpful.
79 00:19:24.820 ⇒ 00:19:29.939 Demilade Agboola: Just so that the technical team isn’t left high and dry. So, like, the business stakeholders would obviously know
80 00:19:29.960 ⇒ 00:19:45.070 Demilade Agboola: what they’re, like, we’ll figure out what their… what the goal is, but just being able to, like, have an idea of, like, hey, potentially we would want the cohort data together with this, but this is how we would want it marked out, or maybe we would want it as a separate
81 00:19:45.070 ⇒ 00:19:50.040 Demilade Agboola: Instances where we might then join it after to make sense of it.
82 00:19:50.210 ⇒ 00:19:57.510 Demilade Agboola: Or, like, just some high-level idea. Again, obviously not in-depth, because again, these things vary based off
83 00:19:57.920 ⇒ 00:20:01.370 Demilade Agboola: The use cases and the different environments and stuff.
84 00:20:01.500 ⇒ 00:20:03.790 Demilade Agboola: But just, like, a high-level understanding of…
85 00:20:04.290 ⇒ 00:20:12.000 Shreya Chowdhury: Yeah, so that… that’s the part that I… I totally agree, I think we should have that. I just don’t have,
86 00:20:12.370 ⇒ 00:20:13.979 Shreya Chowdhury: I think, like.
87 00:20:14.240 ⇒ 00:20:20.210 Shreya Chowdhury: and I can ask Robert if he has details on, like, a previous case that he’s done, but that’s,
88 00:20:21.110 ⇒ 00:20:40.439 Shreya Chowdhury: I just don’t have details on how we’ve implemented it before, so that’s, like, a to-do section that I don’t… I haven’t added yet, but once, like, we run another one, I can document that a lot more thoroughly, or ask Robert if we have done it before, like, what platforms we used, how we implemented it, and, like.
89 00:20:40.870 ⇒ 00:20:45.259 Shreya Chowdhury: yeah, I can go through and, like, summarize the details there.
90 00:20:45.590 ⇒ 00:20:49.169 Shreya Chowdhury: But yeah, I totally agree. If you want to leave a comment like that, like.
91 00:20:49.830 ⇒ 00:21:01.070 Shreya Chowdhury: somewhere, or, like, a note saying, like, we need a section with this. It’ll be, like, a to-do that’ll probably be left blank for, like, a little bit, and then we can add that later.
92 00:21:09.630 ⇒ 00:21:23.630 Uttam Kumaran: Cool, yeah, that would be great. I actually think the way, Shaya, that you’re architecting this with it, which is, like, the theoretical and then, like, a real example, is, like, perfect, because some of these playbooks we will have not run before.
93 00:21:23.800 ⇒ 00:21:37.730 Uttam Kumaran: But it doesn’t mean we can’t run them, and it doesn’t mean we have to wait until we run them to fill that out. So having those examples is really, really helpful. To give you a sense of how this is gonna get adapted into the sales process.
94 00:21:37.730 ⇒ 00:21:45.170 Uttam Kumaran: when a client asks us, hey, what is part of your data analytics service? I’m gonna, probably at maximum, actually just send them this notion.
95 00:21:45.170 ⇒ 00:21:53.679 Uttam Kumaran: at minimum, we’re gonna… we’ll have a marketing… marketing will design, like, a one-pager that just basically takes this and makes it really beautiful in, like, a PDF.
96 00:21:53.750 ⇒ 00:22:07.499 Uttam Kumaran: So, in the sales side, this is just, like, attacking objections and continuing to build confidence that we’re the people to do it. So, some of these, I think, yeah, it’s actually a great point to ask Robert, if we’ve done this before at Brainforge.
97 00:22:07.500 ⇒ 00:22:17.219 Uttam Kumaran: Either way, hypothetical examples, I think in this room, probably plus a wish, we’ve done everything. You know, one of us has, so we can fill that out for sure.
98 00:22:17.400 ⇒ 00:22:35.990 Shreya Chowdhury: Okay, cool. Sounds good. And then on my end, like, one of the to-dos I can do, even just with this case example, is, like, make, like, a mock-up slide deck as if, like, we were presenting the strategy, and then we’ll have that as a template. I can send it over, we can go through another round of, like, edits, whatever, yeah.
99 00:22:38.030 ⇒ 00:22:45.300 Uttam Kumaran: Okay, perfect. And then I… I started to just look through the overall list. Did everyone have a chance to look at, like, the list of
100 00:22:45.520 ⇒ 00:22:50.380 Uttam Kumaran: Product analytics playbooks that are, like, to-do.
101 00:22:59.460 ⇒ 00:23:02.180 Shreya Chowdhury: If not, we can all take a few now, if people.
102 00:23:02.180 ⇒ 00:23:05.139 Uttam Kumaran: Yeah, maybe let’s just take, like, 1 or 2 minutes. Yeah.
103 00:23:05.140 ⇒ 00:23:05.750 Shreya Chowdhury: Cool.
104 00:23:06.150 ⇒ 00:23:15.660 Uttam Kumaran: I think the biggest thing for everyone, just look if there’s anything that can be consolidated, if you want to add any boards to anything, but yeah.
105 00:25:21.050 ⇒ 00:25:36.380 Uttam Kumaran: I guess my only question, maybe for Shreya, for everybody here, like, one of the things that I try to do when I… especially when I work with AI and, like, kind of things, like, these, I think, all make a lot of sense. Is there anything that you saw at, like, a past company that was really, like.
106 00:25:36.380 ⇒ 00:25:42.369 Uttam Kumaran: blow your socks off type of analysis that you think we could add to this list.
107 00:25:42.450 ⇒ 00:25:55.849 Uttam Kumaran: I feel like everything on here, though, I’m really comfortable with feels pretty standard. Probably just have a couple of, like, section changes, and, like, but, yeah, I don’t know, kind of like, maybe we think even a little bit bigger. What else do you think we could do?
108 00:25:56.890 ⇒ 00:26:08.749 Shreya Chowdhury: Yeah, this is… this is mainly just what I had to start with. I wanted to make sure we had the most important and fundamental ones, because these are the ones that I feel like will have the most, like, applications.
109 00:26:08.750 ⇒ 00:26:17.060 Uttam Kumaran: I’m trying to think of anything that I’ve done that blew people’s socks off. I mostly…
110 00:26:17.300 ⇒ 00:26:26.010 Uttam Kumaran: Like, yeah, I don’t know, like, we’ve done some stuff on, like, inventory and, like, logistics. Like, I guess my question would be more about, like,
111 00:26:26.160 ⇒ 00:26:29.350 Uttam Kumaran: When you talk about, for example, like.
112 00:26:29.590 ⇒ 00:26:47.550 Uttam Kumaran: in shipping data, looking at, like, shipping costs, shipping speed, inventory, turnover, that is, like, a huge thing in that logistics domain. So I guess my question would be, like, would that fall under any of these categories, or should we start building, like, domain-specific playbooks soon?
113 00:26:47.690 ⇒ 00:27:02.130 Shreya Chowdhury: Yeah, so I think we could definitely have domain-specific playbooks, but I think that would be more of, like, a useful, like, not an appendix, but, like, a separate section with, like, oh, like, interesting use cases that we’ve done, but I think the shipping one could probably be mapped to, like.
114 00:27:03.190 ⇒ 00:27:03.930 Uttam Kumaran: sad.
115 00:27:03.930 ⇒ 00:27:11.979 Shreya Chowdhury: like, I think North Star metrics here, like, if we’re talking about shippings and logistics, then, like, we could do that one, or, like…
116 00:27:15.880 ⇒ 00:27:24.249 Shreya Chowdhury: Yeah, I think that would be North Star Metrics or, like, the KPI Deep Dive. Like, we could look at it over time and, like, how it’s changed,
117 00:27:26.510 ⇒ 00:27:27.420 Shreya Chowdhury: Yeah.
118 00:27:27.420 ⇒ 00:27:28.120 Uttam Kumaran: No.
119 00:27:28.120 ⇒ 00:27:43.580 Shreya Chowdhury: Let me think about that one a little bit. I assume it can be mapped to one of the ones here, or, like, another one that I probably didn’t include, but I think it would be really cool to have each specific use case, like, in a separate section, and like you said, yeah, we can map it out to the domains.
120 00:27:49.560 ⇒ 00:27:55.229 Uttam Kumaran: Yeah, I feel like this is great, I mean, Henry or Demolati, any, like, thoughts on this list?
121 00:27:55.970 ⇒ 00:27:57.389 Henry Zhao: Oh, that’s good to me as well.
122 00:27:58.470 ⇒ 00:27:59.609 Henry Zhao: It’s pretty standard.
123 00:28:01.360 ⇒ 00:28:03.980 Demilade Agboola: Yeah, I think it’s a fair enough list,
124 00:28:07.630 ⇒ 00:28:10.359 Demilade Agboola: I think the… the only other thing…
125 00:28:10.700 ⇒ 00:28:14.699 Henry Zhao: I have seen that comes to mind is just more of, like.
126 00:28:15.200 ⇒ 00:28:21.690 Demilade Agboola: more… geometric… like, but that was when I was working in logistics, but it was more, like.
127 00:28:21.860 ⇒ 00:28:27.750 Demilade Agboola: keep my… sensitive, like, heatmap analysis, so, like, geodata.
128 00:28:27.750 ⇒ 00:28:29.480 Shreya Chowdhury: Oh, that would be cool.
129 00:28:29.640 ⇒ 00:28:30.120 Demilade Agboola: Yeah.
130 00:28:30.120 ⇒ 00:28:32.100 Uttam Kumaran: Yeah, just geospatial.
131 00:28:32.730 ⇒ 00:28:43.270 Demilade Agboola: And so being able to, like, associate orders with time and all of that, so we can start to have, like, people who do, like, logistics and have an idea of, you know, how heavy they need to, like, load.
132 00:28:43.410 ⇒ 00:28:54.059 Demilade Agboola: like, we had… because it was logistic to have riders in different locations, so it allowed us to be able to forecast, like, hey, like, this area should have more riders at this particular point in time.
133 00:28:54.060 ⇒ 00:28:56.539 Shreya Chowdhury: That would be super cool, actually.
134 00:28:56.710 ⇒ 00:28:57.890 Demilade Agboola: Yeah.
135 00:28:57.890 ⇒ 00:28:59.749 Shreya Chowdhury: Yeah, that’s a good one to add.
136 00:29:00.100 ⇒ 00:29:16.190 Demilade Agboola: Yeah, because it was, like, it was, like, Uber, but in, like, Saudi Arabia. It wasn’t, like, it was, like, yeah, Uber Foods or Uber Eats. So you need to know where the riders need to be, because obviously, if people were ordering a lot, and it was taking too long, they would cancel. So it was a huge use case for us.
137 00:29:16.820 ⇒ 00:29:20.010 Shreya Chowdhury: Okay, yeah, cool, if you want to leave a comment for, like, a…
138 00:29:20.280 ⇒ 00:29:25.530 Shreya Chowdhury: for, like, a to-do or to add for that one, I think that would be a really interesting one.
139 00:29:25.850 ⇒ 00:29:26.740 Shreya Chowdhury: Yeah.
140 00:29:26.740 ⇒ 00:29:44.689 Uttam Kumaran: Yeah, and another thing is, like, I would like us to start to get, you know, probably next is, like, more opinionated about, like, the tooling that we’re gonna use to enable this. Like, do we always start stuff in Jupyter, or, like, let’s say we get hacks or something, and then it moves to, like, slides?
141 00:29:44.950 ⇒ 00:29:46.780 Shreya Chowdhury: Yeah, so that one…
142 00:29:46.780 ⇒ 00:29:47.570 Uttam Kumaran: Yeah, go ahead.
143 00:29:47.570 ⇒ 00:30:06.969 Shreya Chowdhury: Yeah, so that one I left, like, a little bit open-ended for now, intentionally, just because I don’t know how we’ll be implementing a lot of these experiments. So, let’s say, like, if the client already has a platform to implement that, then, like, we do the experiment on their side, and then once we pull in the data, like.
144 00:30:06.970 ⇒ 00:30:17.149 Shreya Chowdhury: we analyze it on our end. For the A-B testing playbook, like, I like pulling the data into, like, eventually into Jupyter to analyze it,
145 00:30:18.440 ⇒ 00:30:30.620 Shreya Chowdhury: Yeah, but I can also, like, as we’re going through this, I don’t know, we can have, like, notebooks open with, like, code that can be recycled. That one just really gets a little bit, like…
146 00:30:30.620 ⇒ 00:30:31.070 Uttam Kumaran: I…
147 00:30:31.070 ⇒ 00:30:37.170 Shreya Chowdhury: I worry about, like, over-indexing on those templates, just because I feel like most of that stuff is, like.
148 00:30:37.170 ⇒ 00:30:39.699 Uttam Kumaran: Oh, totally, totally.
149 00:30:39.700 ⇒ 00:30:47.999 Shreya Chowdhury: easy enough to search up, and then if we make it too specific, I almost feel like it becomes less applicable to, like, different cases.
150 00:30:48.420 ⇒ 00:31:02.340 Uttam Kumaran: Yeah, I guess more it’s like, what I would be looking for is, like, you can probably assume for every client, we have some data stored, right? Like, it’s either in CSVs, or in the best case, it’s already modeled in a warehouse, right?
151 00:31:02.340 ⇒ 00:31:17.779 Uttam Kumaran: that’s one assumption, so you know that, like, okay, we can… we can always plug this into Jupyter, and so, like, basically what I’m hoping for is the product analytics sort of crew, the data crew, is able to say, like, let’s say someone new starts on the product analytics team tomorrow, and for a client.
152 00:31:17.780 ⇒ 00:31:31.360 Uttam Kumaran: part of our deliverable that we promised them is one of these analyses, a feature analysis. Okay, like, what are the tools that we commonly use? And we don’t even have to be specific. If, for example, if we want to, say, use a notebook tool, we can start there.
153 00:31:31.550 ⇒ 00:31:47.850 Uttam Kumaran: And for the BI tool, it’s gonna be really specific. Like, if they already have Looker or Tableau, then that’s where things have to end up. But I think we should have some opinionated ways, because also, our ability to get things reviewed is… is gonna be tough. Like, for example, if someone just writes SQL, like, on their machine.
154 00:31:47.850 ⇒ 00:31:54.549 Uttam Kumaran: produces screenshots, puts them to Google Doc, that’s, like, horrible, right? It’s a horrible review. So that’s where I want to, like.
155 00:31:54.600 ⇒ 00:31:59.710 Uttam Kumaran: Have some floor for, like, what is the minimal technical acceptance for, like.
156 00:31:59.850 ⇒ 00:32:08.960 Uttam Kumaran: one or many of these, which is, like, great. If you consider one of these done, we should see a Jupyter notebook with queries and with, like.
157 00:32:08.960 ⇒ 00:32:24.310 Uttam Kumaran: inline comments about analysis, and then finally, like, something. That is, like, a great artifact to review. And then it could move… then it can be more, like, case by case, which is, like, this needs to move into a fixed dashboard, or it needs to move into a deck or something, but that’s sort of what I’m trying to get at, which is, like.
158 00:32:24.430 ⇒ 00:32:29.070 Uttam Kumaran: What is the minimum engineering criteria for, like, executing on somebody is?
159 00:32:29.070 ⇒ 00:32:29.750 Shreya Chowdhury: Yeah.
160 00:32:30.110 ⇒ 00:32:30.850 Shreya Chowdhury: Okay.
161 00:32:32.050 ⇒ 00:32:46.210 Uttam Kumaran: Okay, cool. Alright, I feel pretty good about where we landed today. I think, I mean, I think it’s probably, one, we can get Robert’s eyes on some of this, but I feel pretty good. I think…
162 00:32:46.640 ⇒ 00:33:00.280 Uttam Kumaran: like, what’s the… should we plan on ticketing out, like, the rest of the playbooks, and then, sort of Shreya, based on, like, your timing with the other projects, we could, like, maybe take one of these, like, a week or something? Like, I don’t know.
163 00:33:00.280 ⇒ 00:33:00.890 Shreya Chowdhury: Yeah, so…
164 00:33:00.890 ⇒ 00:33:02.310 Uttam Kumaran: really great, yeah, so that’s probably…
165 00:33:02.310 ⇒ 00:33:20.480 Shreya Chowdhury: I think it would be great to ticket out the rest of the playbooks, and then, I guess, like, prioritize them based on, like, which ones would be most useful, or, like, based on the, like, our current client base. And then also, I know we have the existing linear.
166 00:33:20.480 ⇒ 00:33:24.579 Uttam Kumaran: ticket for the A-B testing one, I was going.
167 00:33:24.580 ⇒ 00:33:32.160 Shreya Chowdhury: to close that out, or I think that one maybe just said playbooks, so I’ll have to double check, but if there is one for just the A-B testing.
168 00:33:32.550 ⇒ 00:33:35.849 Uttam Kumaran: I don’t know if we want to update that ticket to add, like…
169 00:33:36.330 ⇒ 00:33:45.720 Shreya Chowdhury: a couple of the things, like, it would probably take me, like, a little bit of… like, maybe half a day or something to build out a slide deck.
170 00:33:45.720 ⇒ 00:33:50.949 Uttam Kumaran: do is, if you give it to the marketing team and give them, like, a slide outline, they’ll build it for you.
171 00:33:50.950 ⇒ 00:33:53.660 Shreya Chowdhury: Okay, yeah. I think at the very least, like.
172 00:33:53.780 ⇒ 00:34:01.799 Shreya Chowdhury: if I’m allowed to, like, work on the data slide, like, I’ll just, like, at least add the templates, like, for how to present the data, and then they can, like.
173 00:34:01.930 ⇒ 00:34:06.119 Shreya Chowdhury: Take that and, like, make it prettier, or, like, make it all different, but as long as, like.
174 00:34:07.240 ⇒ 00:34:12.370 Shreya Chowdhury: I feel like whoever’s working on the data should at least, like, have a hand in presenting the data of the slide.
175 00:34:12.550 ⇒ 00:34:32.299 Uttam Kumaran: Totally, yeah, and I think, like, as, like, basically what I’m hoping for is, like, one, we create some templates. So, this is a drop-in template where you can add an image, and you can have, like, some bullets or text, and then we have slides for, like, presenting a hypothesis and things like that. But yeah, I think that, the person running the experiment, hopefully, isn’t know this.
176 00:34:32.300 ⇒ 00:34:34.059 Uttam Kumaran: Okay, so that makes sense, and I think…
177 00:34:34.060 ⇒ 00:34:52.310 Uttam Kumaran: sort of for Demolade, and probably a ways to catch up on this meeting, is we can start to plan these for the AE and DE teams as well. So maybe, Amber, one thing that we can… we could probably do a similar meeting just to this, like, taking one playbook for the analytics engineering team, and then also working on sort of this, like.
178 00:34:52.480 ⇒ 00:34:57.139 Uttam Kumaran: playbooks, like, playbook repository for…
179 00:34:57.170 ⇒ 00:35:13.270 Uttam Kumaran: data engine, analytics engine, I’m just gonna combine them, but there will… there is gonna be a lot, of variation, but I think that’s a great place to start. And I could also… like, that’s where I can contribute as well, and need them a lot of information, start to piece out items there.
180 00:35:13.320 ⇒ 00:35:15.270 Uttam Kumaran: Okay, I…
181 00:35:15.270 ⇒ 00:35:20.599 Amber Lin: I have a few questions. So, first is, do we have a template on the engineering side?
182 00:35:21.900 ⇒ 00:35:25.769 Uttam Kumaran: I guess, like, I’ve…
183 00:35:25.770 ⇒ 00:35:26.330 Amber Lin: Great one.
184 00:35:26.330 ⇒ 00:35:29.590 Uttam Kumaran: Yeah, like, I guess I don’t know…
185 00:35:30.290 ⇒ 00:35:47.010 Uttam Kumaran: Yeah, I mean, the A-B testing handbook is… is a good place to start, but I… the parts I like is that we have these different sections, and then we have, like, examples, right? So at a minimum, in our template, I would like to have that type of format, which is, like, the…
186 00:35:47.700 ⇒ 00:35:55.200 Uttam Kumaran: like, what is the goal of this entire playbook? Like, when to use this playbook? And then steps, and then each step, you have, like.
187 00:35:55.500 ⇒ 00:36:02.400 Uttam Kumaran: the description and an example. Apart from that, I don’t know how much concurrency there is between, you know.
188 00:36:02.400 ⇒ 00:36:05.409 Amber Lin: Yeah, I agree. We also have the technical design document
189 00:36:05.980 ⇒ 00:36:07.919 Amber Lin: Is that something that we can model after?
190 00:36:10.970 ⇒ 00:36:15.610 Uttam Kumaran: The TDD is really, really broad.
191 00:36:15.610 ⇒ 00:36:16.449 Amber Lin: I see, gotcha.
192 00:36:16.450 ⇒ 00:36:35.649 Uttam Kumaran: Yeah, like, this is more meta in that this isn’t about a specific solution, this is about, like… let’s take, like, any… for example, we walk into a client, like, for Urban Sems, we’re running, like, we’re running, like, probably 10 different playbooks right now. We’re running, like, new data model creation playbooks, data model restructuring, new ETL playbook, right?
193 00:36:36.270 ⇒ 00:36:47.630 Uttam Kumaran: it’s sort of like we’re running these plays, so it’s more about, like, what is the step-by-step to… what requirements are needed, how do you execute, and then what does that look like?
194 00:36:47.930 ⇒ 00:36:48.620 Amber Lin: Yeah.
195 00:36:48.730 ⇒ 00:36:50.290 Amber Lin: Okay.
196 00:36:51.160 ⇒ 00:36:53.270 Amber Lin: Do we need that for the AI team?
197 00:36:54.540 ⇒ 00:37:09.150 Uttam Kumaran: Yes, they will need it as well. Theirs also will be a little bit different. But regardless, kind of, like, the meta thing is, like, when we go into a client, we are running one or many playbooks when we sell a series, right?
198 00:37:09.610 ⇒ 00:37:14.220 Uttam Kumaran: And so we want to almost be like, cool, we’re running these, like, 6 plays.
199 00:37:16.000 ⇒ 00:37:25.000 Amber Lin: Yes. My next question is, can I have at least one of these as priorities, so Shaya knows what to start working on?
200 00:37:25.000 ⇒ 00:37:26.530 Uttam Kumaran: Oh, yeah, yeah, yeah, so…
201 00:37:26.530 ⇒ 00:37:27.460 Amber Lin: When would it be?
202 00:37:27.460 ⇒ 00:37:45.299 Uttam Kumaran: Yeah, so let me, So just given that, like, I’m thinking about default and insomnia, like, I would probably pick feature adoption and, I’d probably pick 2, 3, and 4.
203 00:37:46.330 ⇒ 00:37:53.139 Amber Lin: 3 and 4. Are we dividing that between Henry and… Shreya, or is this all…
204 00:37:54.260 ⇒ 00:38:00.700 Uttam Kumaran: I… I would… I would say if Henry has experience in running one of these, I would totally split it up.
205 00:38:02.280 ⇒ 00:38:07.729 Uttam Kumaran: Yeah. So, I don’t mind, I think we should… I think we should split this up. Ideally, these should be split up between…
206 00:38:07.950 ⇒ 00:38:13.690 Amber Lin: Robert, Treya, and Henry. Yeah. Treya, and then I’ll ask in the channel.
207 00:38:13.870 ⇒ 00:38:25.050 Uttam Kumaran: Cool. 2, 3, and 4 are most important because many of our clients don’t even have, like, clear KPI understanding, and what we do in 2, 3, and 4 will get them to start a big KPI soon.
208 00:38:35.820 ⇒ 00:38:49.909 Amber Lin: Okay, sounds good. That’s the first stand. On the engineering side, I’m checking next week. You guys have an engineering meeting on Wednesday, and a meeting on Thursday. I think that’s a good place to check the playbooks, or do you want a separate meeting?
209 00:38:50.980 ⇒ 00:39:07.119 Uttam Kumaran: I mean, it’s… yeah, it’s sort of like, right now, we’re doing this, like, AI meeting on Thursday. I don’t care, I… I would sort of let this team decide, like, as long as I think everybody on data can meet once a week, and we sort of do these reviews, I think it’s fine.
210 00:39:07.860 ⇒ 00:39:08.580 Amber Lin: Huh?
211 00:39:09.620 ⇒ 00:39:17.660 Uttam Kumaran: I don’t… like, we have all engineering meetings on Thursdays, we didn’t take part of that, but totally up to the team.
212 00:39:17.660 ⇒ 00:39:25.340 Amber Lin: Okay, so I have… Let’s see…
213 00:39:26.280 ⇒ 00:39:27.360 Amber Lin: Okay.
214 00:39:27.550 ⇒ 00:39:29.029 Amber Lin: Right now…
215 00:39:29.620 ⇒ 00:39:39.880 Amber Lin: for this… for this one, we have a template task, and then a list out tasks, a list-out offerings task. How are we assigning these?
216 00:39:41.150 ⇒ 00:39:43.600 Uttam Kumaran: Wait, sorry, which…
217 00:39:43.600 ⇒ 00:39:45.890 Amber Lin: Or, for engineering playbooks.
218 00:39:48.390 ⇒ 00:39:51.119 Uttam Kumaran: I don’t know what these are haunt.
219 00:39:51.120 ⇒ 00:39:59.820 Amber Lin: So, just like how we… what we did for product analytics, we need a template first, and then we need to map out what playbooks we need to create.
220 00:40:00.510 ⇒ 00:40:13.110 Uttam Kumaran: Yeah, I mean, for AI, I would assign it to… for AI, it should be SAM, and for DataEng, it should be, Demolade, or… For data engineering, for analytics engineering, it should be Demolade.
221 00:40:13.110 ⇒ 00:40:13.820 Amber Lin: Go ahead.
222 00:40:14.060 ⇒ 00:40:20.900 Uttam Kumaran: Yeah. So they can take the first crack, ideally one playbook and a kind of template of options, and then…
223 00:40:21.350 ⇒ 00:40:24.389 Uttam Kumaran: Yeah, we should just be able to run through that in a tree. Okay.
224 00:40:24.390 ⇒ 00:40:26.990 Amber Lin: And are you going to do the…
225 00:40:27.430 ⇒ 00:40:33.790 Uttam Kumaran: I can build… I can take on one of the playbooks, but I don’t know… oh, the template for the playbook? Yeah.
226 00:40:34.320 ⇒ 00:40:36.350 Uttam Kumaran: Sure.
227 00:40:36.350 ⇒ 00:40:41.259 Amber Lin: I think once you do that, people can do the playbook, and you can just do review.
228 00:40:41.610 ⇒ 00:40:48.249 Uttam Kumaran: Can you, okay, yeah, if you can make sure there’s sections somewhere for me to put that, and yeah, I’ll work on a template for each area.
229 00:40:50.430 ⇒ 00:40:53.909 Amber Lin: Sorry, what do you mean, section?
230 00:40:53.910 ⇒ 00:40:56.499 Uttam Kumaran: Where should I… where should I create this stuff?
231 00:40:56.500 ⇒ 00:40:57.180 Amber Lin: Oh, I’ll…
232 00:40:57.180 ⇒ 00:40:58.000 Uttam Kumaran: process.
233 00:40:58.000 ⇒ 00:41:03.080 Amber Lin: Yeah, I will find… Yeah, I think it’s just in services.
234 00:41:05.580 ⇒ 00:41:11.800 Amber Lin: I will make a… actually, here.
235 00:41:12.820 ⇒ 00:41:20.560 Amber Lin: Say… A engineering player.
236 00:41:21.150 ⇒ 00:41:22.070 Amber Lin: template.
237 00:41:22.260 ⇒ 00:41:23.590 Amber Lin: I’ll copy…
238 00:41:23.590 ⇒ 00:41:24.610 Uttam Kumaran: Yeah, I don’t know.
239 00:41:24.610 ⇒ 00:41:25.630 Amber Lin: I’ll copy this into
240 00:41:31.150 ⇒ 00:41:39.049 Amber Lin: Alright, list of all… Is the AI playbook templates the same as data?
241 00:41:39.550 ⇒ 00:41:42.169 Uttam Kumaran: No. All of these are different, yeah.
242 00:41:42.460 ⇒ 00:41:45.550 Amber Lin: Okay, so we need a template for AE…
243 00:41:45.550 ⇒ 00:41:46.400 Uttam Kumaran: Yeah.
244 00:41:46.690 ⇒ 00:41:50.660 Uttam Kumaran: Well, you could just… you could just put one template ticket, like, I don’t need 40.
245 00:41:50.660 ⇒ 00:41:51.050 Amber Lin: Okay.
246 00:41:52.920 ⇒ 00:41:55.229 Uttam Kumaran: Let’s put my coffee in my house.
247 00:41:55.480 ⇒ 00:41:58.100 Amber Lin: D-E-A-I.
248 00:41:58.100 ⇒ 00:42:06.449 Uttam Kumaran: But honestly, I may not… I’m not gonna… I don’t wanna work on the templates first, because I kind of want each of the function folks to try it out themselves.
249 00:42:06.630 ⇒ 00:42:10.840 Uttam Kumaran: Like, if I give the boundaries, like, we’re not gonna get much creativity.
250 00:42:10.990 ⇒ 00:42:11.370 Amber Lin: Okay.
251 00:42:11.370 ⇒ 00:42:17.370 Uttam Kumaran: So, I may… I kind of want to wait to see everything first before I would say everything.
252 00:42:17.370 ⇒ 00:42:22.470 Amber Lin: Okay, so I’ll ask people to just list out… list it out first, and then give it a stack.
253 00:42:22.470 ⇒ 00:42:29.110 Uttam Kumaran: Basically, everything, basically, like, whatever’s stranded today, we can get that from those three areas.
254 00:42:29.110 ⇒ 00:42:37.240 Amber Lin: Awesome. I will send that in the data platform channel. Is there a better channel for that?
255 00:42:38.700 ⇒ 00:42:44.370 Uttam Kumaran: You can just send it to engineering or whatever. I have to drop 4 now, though.
256 00:42:44.640 ⇒ 00:42:47.360 Amber Lin: All good, that’s all I needed.
257 00:42:48.660 ⇒ 00:42:50.100 Uttam Kumaran: Thank you, guys.
258 00:42:50.100 ⇒ 00:42:51.260 Shreya Chowdhury: Thank you.
259 00:42:52.410 ⇒ 00:42:54.240 Amber Lin: Alright, thanks everyone!