Meeting Title: Analytics Engineering Daily Sync Date: 2025-03-05 Meeting participants: Luke Daque, Uttam Kumaran, Demilade Agboola, Caio Velasco
WEBVTT
1 00:01:14.080 ⇒ 00:01:15.050 Demilade Agboola: Hello!
2 00:01:18.220 ⇒ 00:01:20.119 Luke Daque: Hello! Hi, there, Melanie!
3 00:01:20.510 ⇒ 00:01:21.609 Luke Daque: How’s it going.
4 00:01:22.550 ⇒ 00:01:23.749 Demilade Agboola: Pretty good. How are you.
5 00:01:24.880 ⇒ 00:01:26.109 Luke Daque: Yeah doing? Wellice.
6 00:01:29.370 ⇒ 00:01:29.920 Caio Velasco: Hello!
7 00:01:31.380 ⇒ 00:01:32.490 Luke Daque: Hey! Tayo!
8 00:01:33.567 ⇒ 00:01:35.179 Caio Velasco: How’s it going.
9 00:01:37.390 ⇒ 00:01:38.629 Luke Daque: So we think.
10 00:01:41.760 ⇒ 00:01:47.680 Luke Daque: have you started them ladi on like stuff already working on stuff.
11 00:01:49.079 ⇒ 00:01:56.100 Demilade Agboola: No, I start fully next week. So for now I’m still kind of like just trying to
12 00:01:57.150 ⇒ 00:02:00.730 Demilade Agboola: off board and on board. It’s it’s like
13 00:02:00.880 ⇒ 00:02:06.770 Demilade Agboola: flex. It’s not like flux period. But like, yeah. But next week I’m fully fully hunt.
14 00:02:20.320 ⇒ 00:02:24.229 Luke Daque: Yeah, it looks like Utah might be a little bit late.
15 00:02:25.780 ⇒ 00:02:29.309 Luke Daque: They just mentioned it in the data team channel.
16 00:02:34.430 ⇒ 00:02:37.729 Demilade Agboola: Okay, how do things normally? Go ahead.
17 00:02:39.690 ⇒ 00:02:48.570 Luke Daque: Yeah, really, usually, just discuss about like what we’re like working on. And like, if there are any blockers or anything.
18 00:02:50.360 ⇒ 00:02:56.880 Luke Daque: Yeah, how’s the stuff for? Javi? I
19 00:02:57.330 ⇒ 00:03:00.641 Luke Daque: have you sorted that out yet, like the one
20 00:03:01.010 ⇒ 00:03:04.050 Luke Daque: So yeah. So Boston.
21 00:03:04.050 ⇒ 00:03:08.766 Caio Velasco: Yeah. Yeah. So the Cox thing. No cause I was well,
22 00:03:09.310 ⇒ 00:03:15.899 Caio Velasco: trying to like go over slack yesterday and see across all messages. What people have been.
23 00:03:17.720 ⇒ 00:03:18.070 Caio Velasco: Yeah.
24 00:03:18.070 ⇒ 00:03:25.680 Caio Velasco: in regards to you know the financial stuff which I which I believe is connected to the Amazon dashboard and the
25 00:03:26.230 ⇒ 00:03:28.470 Caio Velasco: the net margin. Dashboard
26 00:03:29.158 ⇒ 00:03:38.920 Caio Velasco: but then I was doing that. But then you done told me to focus on gorgeous dashboard. Then I switched it over to to the gorgeous stuff.
27 00:03:39.310 ⇒ 00:03:42.889 Caio Velasco: But then it was already too late, and and I well.
28 00:03:43.120 ⇒ 00:03:47.559 Caio Velasco: initially, I’ve never really worked with Meta Base or anything.
29 00:03:48.140 ⇒ 00:03:54.460 Caio Velasco: So I was doing that as well. But then I think Pi has already sold the, the.
30 00:03:54.650 ⇒ 00:03:56.310 Caio Velasco: the, the issues.
31 00:03:57.390 ⇒ 00:04:02.210 Caio Velasco: So then, now I think you can go back to the documentation idea.
32 00:04:02.972 ⇒ 00:04:07.179 Caio Velasco: So I will just confirm with him. But I believe that
33 00:04:07.881 ⇒ 00:04:17.029 Caio Velasco: I can go back to that to what I was telling you yesterday’s they’re trying like to really understand all aspects of every variable involved
34 00:04:17.230 ⇒ 00:04:24.790 Caio Velasco: in all the dashboards, like something that goes almost towards inside the client outdated.
35 00:04:25.150 ⇒ 00:04:29.570 Caio Velasco: and whatever like calls or margin or shipping.
36 00:04:31.470 ⇒ 00:04:36.960 Caio Velasco: So yeah, I think I will be working on that not sure how far I have to go
37 00:04:37.981 ⇒ 00:04:42.289 Caio Velasco: but well, what they told me was a was was a bit broad
38 00:04:42.520 ⇒ 00:04:45.483 Caio Velasco: by model. I might might as well just ask him.
39 00:04:46.670 ⇒ 00:04:51.710 Caio Velasco: But yeah, it’s been. It’s been okay. I think it’s a bit more
40 00:04:52.120 ⇒ 00:04:54.200 Caio Velasco: on the documentation side. I guess.
41 00:04:55.980 ⇒ 00:04:56.950 Luke Daque: Yeah, nice.
42 00:04:58.340 ⇒ 00:04:58.880 Caio Velasco: So
43 00:04:58.880 ⇒ 00:05:05.430 Caio Velasco: also a a question that I would have for you is like, if I start to have questions like those I mentioned to you yesterday.
44 00:05:05.720 ⇒ 00:05:13.990 Caio Velasco: Who should I? Who should I be pinging? Or should I just go on on slack and put it out? Put it out there so that folks can.
45 00:05:14.550 ⇒ 00:05:15.880 Luke Daque: Yeah, I think that’s correct.
46 00:05:15.880 ⇒ 00:05:23.730 Luke Daque: Yes, approach, I would say, like, just just ask out in slack. And anybody can answer, really.
47 00:05:24.460 ⇒ 00:05:26.079 Luke Daque: especially like this?
48 00:05:26.330 ⇒ 00:05:34.909 Luke Daque: Yeah, especially like the one you asked yesterday, where? Nobody else really had any context except for maybe me and pay us.
49 00:05:35.320 ⇒ 00:05:42.920 Luke Daque: So yeah, if you would have asked that to Robert, or like Tom, then maybe they wouldn’t be able to provide
50 00:05:43.790 ⇒ 00:05:49.280 Luke Daque: well, like a definite answer on it. So yeah, maybe asking in slack would be the best approach there.
51 00:05:50.460 ⇒ 00:05:52.139 Caio Velasco: No, perfect. Okay. I will do.
52 00:05:52.690 ⇒ 00:05:58.349 Luke Daque: It can be in the data team or the the client. Channels. I guess.
53 00:06:00.740 ⇒ 00:06:06.480 Caio Velasco: Yeah, we’ll try to gather some questions and export
54 00:06:06.610 ⇒ 00:06:12.480 Caio Velasco: the database and understand what what I mean. Basically, understanding the data model would be would be better.
55 00:06:12.480 ⇒ 00:06:12.990 Luke Daque: Yeah.
56 00:06:12.990 ⇒ 00:06:15.860 Caio Velasco: Next step, and then question and recover.
57 00:06:17.070 ⇒ 00:06:18.200 Luke Daque: Yeah, make 10,
58 00:06:21.510 ⇒ 00:06:22.170 Luke Daque: cool.
59 00:06:22.170 ⇒ 00:06:24.980 Caio Velasco: What about what about you? How? How has it been working with them?
60 00:06:26.090 ⇒ 00:06:31.599 Luke Daque: Yeah. Working mostly on stack blit stuff so.
61 00:06:31.600 ⇒ 00:06:32.240 Caio Velasco: Please.
62 00:06:32.450 ⇒ 00:06:38.450 Luke Daque: Yeah, they they are like, so I’m I’m focusing at the model beta model for
63 00:06:39.209 ⇒ 00:06:46.110 Luke Daque: their recurring revenue. So both monthly recurring revenue and annual recurring revenue.
64 00:06:46.902 ⇒ 00:06:56.320 Luke Daque: They initially already have a dashboard for that in bare metrics. But it’s very general. It’s like the total monthly or recurring revenue, and
65 00:06:56.470 ⇒ 00:07:03.870 Luke Daque: like, they want to be able to drill down or like categorize it by. For for each plan, for example.
66 00:07:04.520 ⇒ 00:07:14.069 Luke Daque: because just just for context, stack. Blitz is a AI company, and like they have.
67 00:07:14.830 ⇒ 00:07:21.750 Luke Daque: they have they have plans for the user like that are either monthly.
68 00:07:22.508 ⇒ 00:07:26.909 Luke Daque: They’re they’re actually monthly, but they can be paid either monthly or annually.
69 00:07:27.370 ⇒ 00:07:34.640 Luke Daque: And what that does is basically they have a set of tokens for each month that they are able to use
70 00:07:36.450 ⇒ 00:07:40.010 Luke Daque: for their Api for their calls through the chat.
71 00:07:40.849 ⇒ 00:07:43.980 Luke Daque: what do you call it? AI agents and stuff like that.
72 00:07:44.760 ⇒ 00:07:52.379 Luke Daque: And yeah, so there are different plans. There’s like, Pro 100 pro 50, whatever. And then.
73 00:07:52.500 ⇒ 00:08:04.710 Luke Daque: yeah, so basically, I’m trying to create an Mmr Mrr model monthly recurring revenue model that has all those dimensions, whether it’s plan or the type of payment, whether it’s monthly or annually.
74 00:08:04.940 ⇒ 00:08:12.820 Luke Daque: and then also all the other dimensions that I’m trying to join, like from a customer perspective, for example.
75 00:08:14.400 ⇒ 00:08:20.360 Luke Daque: whether it’s an organization account or if it’s an individual user depending on
76 00:08:22.540 ⇒ 00:08:25.060 Luke Daque: like their email address and stuff like that.
77 00:08:26.190 ⇒ 00:08:35.139 Luke Daque: And like one thing I’m trying to figure out at the moment is to see if we we can move that monthly recurring revenue to a
78 00:08:35.600 ⇒ 00:08:39.059 Luke Daque: token based consumption.
79 00:08:40.179 ⇒ 00:08:42.660 Luke Daque: Because, yeah, like, I mentioned, it’s like
80 00:08:42.830 ⇒ 00:08:46.249 Luke Daque: they give you a set of number of tokens per month.
81 00:08:46.550 ⇒ 00:08:51.040 Luke Daque: but then there’s a limit per month. If you reach that limit, then
82 00:08:51.780 ⇒ 00:08:57.940 Luke Daque: that would mean you can’t use the service until the next month. Something like that unless you top up
83 00:08:58.050 ⇒ 00:09:02.440 Luke Daque: the tokens and like topping up would be another
84 00:09:04.740 ⇒ 00:09:09.959 Luke Daque: I guess revenue, you would say it’s not cause it’s like outside the monthly payment.
85 00:09:10.430 ⇒ 00:09:17.554 Luke Daque: So yeah, that’s that’s where it’s getting complicated in in that part, like when they’re topping up tokens
86 00:09:18.910 ⇒ 00:09:20.739 Luke Daque: outside the monthly payment.
87 00:09:21.040 ⇒ 00:09:21.810 Luke Daque: No.
88 00:09:22.830 ⇒ 00:09:23.180 Caio Velasco: Okay.
89 00:09:23.230 ⇒ 00:09:23.820 Luke Daque: Yeah.
90 00:09:23.820 ⇒ 00:09:31.750 Caio Velasco: It sounds it sounds interesting. Should. Do you think I could somehow help you with this documentation idea?
91 00:09:32.060 ⇒ 00:09:33.720 Caio Velasco: Maybe.
92 00:09:35.170 ⇒ 00:09:35.590 Luke Daque: Yeah.
93 00:09:35.590 ⇒ 00:09:48.519 Caio Velasco: Because when when you can mention, he said, like, I would like to continue this documentation for our core dashboards and models as part of the data platform spreadsheet. That’s what he said. Is this.
94 00:09:48.780 ⇒ 00:09:49.180 Luke Daque: One of the.
95 00:09:49.180 ⇒ 00:09:50.520 Caio Velasco: 4 Dashboards.
96 00:09:51.740 ⇒ 00:09:58.469 Luke Daque: I would say, so, yeah, this would be like one of their. This is the highest priority dashboard. They’re asking as well. So
97 00:09:59.090 ⇒ 00:09:59.580 Luke Daque: yeah.
98 00:09:59.580 ⇒ 00:10:07.990 Caio Velasco: Okay, do you think you need like help with something like in terms of understanding all this definition.
99 00:10:08.660 ⇒ 00:10:10.200 Luke Daque: Yeah, maybe.
100 00:10:10.560 ⇒ 00:10:14.319 Caio Velasco: Or yeah, maybe live. Maybe something like that.
101 00:10:14.320 ⇒ 00:10:22.079 Luke Daque: Yeah, yeah, yeah. I guess I’ll I can maybe schedule a call with you or something. So we can
102 00:10:22.660 ⇒ 00:10:28.350 Luke Daque: try to figure out how to document this. If that’s like, you’re okay with that.
103 00:10:29.200 ⇒ 00:10:31.410 Caio Velasco: Perfect. No, for me. All good. Yeah.
104 00:10:33.160 ⇒ 00:10:33.720 Luke Daque: Cool.
105 00:10:37.160 ⇒ 00:10:39.769 Luke Daque: Yeah, that’s basically it on my end. Like.
106 00:10:39.950 ⇒ 00:10:43.360 Luke Daque: after that, I’ll be also working on like other
107 00:10:43.910 ⇒ 00:10:49.940 Luke Daque: the other asks, or like other other dashboards they want to see from
108 00:10:51.440 ⇒ 00:10:54.810 Luke Daque: from their end. There’s like customer churned.
109 00:10:55.420 ⇒ 00:11:00.029 Luke Daque: and that would have to be broken down as well like per plan or like.
110 00:11:01.320 ⇒ 00:11:05.460 Luke Daque: and in other other than other dimensions as well.
111 00:11:06.360 ⇒ 00:11:08.500 Luke Daque: And then, aside from churn
112 00:11:10.630 ⇒ 00:11:12.740 Luke Daque: let me see what you have here.
113 00:11:17.520 ⇒ 00:11:20.090 Luke Daque: Oh, yeah, like lifetime value.
114 00:11:21.690 ⇒ 00:11:28.079 Luke Daque: Active customers distribute easier, I would say, like active customers and new customers.
115 00:11:29.810 ⇒ 00:11:39.039 Luke Daque: Yeah, upgrades and boundaries. This would be another, probably different way.
116 00:11:39.740 ⇒ 00:11:43.289 Luke Daque: Well, I guess I guess this this
117 00:11:44.909 ⇒ 00:11:48.999 Luke Daque: client isn’t really similar to what we’re doing with others
118 00:11:50.040 ⇒ 00:11:52.870 Luke Daque: like jabby coffee. So it’s pretty much
119 00:11:53.580 ⇒ 00:12:03.750 Luke Daque: new in terms of like how we would structure with a data model for this, because it’s it’s not existing. It’s not like Javi coffee. What’s a sales dashboard?
120 00:12:06.040 ⇒ 00:12:13.640 Luke Daque: Yeah, this one is more on like subscription basis, subscriptions and.
121 00:12:13.640 ⇒ 00:12:14.530 Caio Velasco: And
122 00:12:15.840 ⇒ 00:12:23.289 Caio Velasco: a question that I have is when you’re, for example, if you in this case they already have a dashboard.
123 00:12:23.450 ⇒ 00:12:27.880 Caio Velasco: so you kind of can already look at the metric and then see how you get
124 00:12:28.260 ⇒ 00:12:36.386 Caio Velasco: you the data model for that metric, or at least a data model to support that metric when you start doing this work.
125 00:12:36.870 ⇒ 00:12:51.610 Caio Velasco: What do you think is the most important part? For example, when I was given the gorgeous dashboard one of the question was, let’s say, which macros have been used the most, or how macros interact with agents.
126 00:12:51.930 ⇒ 00:12:55.209 Luke Daque: And then, after looking at what bias did.
127 00:12:55.450 ⇒ 00:13:09.599 Caio Velasco: I noticed, like many other things that would probably say, Take me like couple of days to really see that those things come from whatever other sources in gorgeous, and then join them, and then do a lot of magic.
128 00:13:09.910 ⇒ 00:13:16.019 Caio Velasco: So but when I went to do myself the dashboard, like one of the metrics or something.
129 00:13:16.520 ⇒ 00:13:19.270 Caio Velasco: Then I started to really feel the pain.
130 00:13:19.880 ⇒ 00:13:27.960 Caio Velasco: and then it was easier to go back to the, to the sources and look for what I’m what I would like to have on the dashboard.
131 00:13:28.463 ⇒ 00:13:46.309 Caio Velasco: So I don’t know. Like in this, back and forth like, where do you usually start. Do you start building a like a chart and say, like, Hey, I want to do a chart like this, and somehow have to make this happen. Given the sources that I have, and then you go to the sources and then you build them, in fact.
132 00:13:46.540 ⇒ 00:13:48.200 Caio Velasco: or is it the other way around.
133 00:13:49.360 ⇒ 00:13:51.279 Luke Daque: Yeah, that’s a good question, miss.
134 00:13:52.240 ⇒ 00:13:58.950 Luke Daque: I don’t know if I like. I don’t think we have a standard approach to that yet, but for in my experience, at least for this
135 00:13:59.320 ⇒ 00:14:00.920 Luke Daque: specific client.
136 00:14:04.696 ⇒ 00:14:14.179 Luke Daque: like. So yeah, I can let me probably share my screen. Maybe that easier to the explain.
137 00:14:19.370 ⇒ 00:14:21.159 Luke Daque: Oh, yeah. Can you see my screen.
138 00:14:22.630 ⇒ 00:14:23.180 Caio Velasco: Yes.
139 00:14:24.360 ⇒ 00:14:28.839 Luke Daque: Yeah. So basically, this was coming, this, the
140 00:14:29.090 ⇒ 00:14:37.649 Luke Daque: this, both metrics. Google Sheet. This was coming from the client like Mitch Mitch was
141 00:14:38.020 ⇒ 00:14:41.190 Luke Daque: yeah from stack, which basically, and they have
142 00:14:41.500 ⇒ 00:14:48.740 Luke Daque: like fence. For example, this is coming from segment
143 00:14:49.772 ⇒ 00:14:54.609 Luke Daque: which is the source. And these are all the types of events that they want
144 00:14:55.020 ⇒ 00:14:56.510 Luke Daque: to be able to track
145 00:14:57.010 ⇒ 00:15:04.730 Luke Daque: so. And yes, you can see, these are just very general. These are just events. I don’t really, we don’t really know
146 00:15:05.448 ⇒ 00:15:09.350 Luke Daque: where this is coming from. So what I did is basically, I added another
147 00:15:09.710 ⇒ 00:15:13.890 Luke Daque: column here called Snowflake Mapping Cause. After we got all the
148 00:15:14.900 ⇒ 00:15:18.539 Luke Daque: we got the source integrated to. Oh, I got logged out.
149 00:15:18.650 ⇒ 00:15:22.940 Luke Daque: But yeah, after the source integrated to Snowflake, I tried to look for
150 00:15:23.190 ⇒ 00:15:28.130 Luke Daque: the tables that would have these events, and added it here
151 00:15:28.480 ⇒ 00:15:35.310 Luke Daque: before I created a model for this. So what then? What happened is, once I got this
152 00:15:35.530 ⇒ 00:15:41.550 Luke Daque: figured out, I did create the model that would be essentially just
153 00:15:42.830 ⇒ 00:15:50.200 Luke Daque: getting all the Events Union together, to have like one mark table for
154 00:15:50.410 ⇒ 00:15:56.700 Luke Daque: all the different events, whether it’s coming from chats or message events or
155 00:15:57.020 ⇒ 00:16:00.089 Luke Daque: template event, token events, and then
156 00:16:00.250 ⇒ 00:16:08.180 Luke Daque: union everything together to have like one like normalized event table, basically
157 00:16:08.320 ⇒ 00:16:10.630 Luke Daque: which we can use as a dimension
158 00:16:11.050 ⇒ 00:16:14.990 Luke Daque: to join to any other tables in the future.
159 00:16:16.970 ⇒ 00:16:21.410 Luke Daque: Yeah, that would that would like used any of these today.
160 00:16:22.790 ⇒ 00:16:25.373 Luke Daque: and for the other stuff.
161 00:16:26.410 ⇒ 00:16:36.630 Luke Daque: they do. They also did already send this like generic table.
162 00:16:37.170 ⇒ 00:16:42.479 Luke Daque: like what they want to see for a subscriptions model, and what kind of
163 00:16:42.660 ⇒ 00:16:47.950 Luke Daque: what fields they want. And they say, like the group group, subscription details
164 00:16:48.120 ⇒ 00:16:53.380 Luke Daque: would be all of these, the plan, the current status. So I have to. I had to like look
165 00:16:54.100 ⇒ 00:17:00.720 Luke Daque: into the snowflake tables one by one, to see if this exists there, and if it doesn’t exist, then
166 00:17:01.360 ⇒ 00:17:02.360 Luke Daque: I did
167 00:17:02.930 ⇒ 00:17:07.930 Luke Daque: flag this. That’s why, like, I just colored this red for now, because I didn’t really know how to
168 00:17:08.319 ⇒ 00:17:13.260 Luke Daque: tackle this. So I did have a call with Mitch
169 00:17:13.420 ⇒ 00:17:23.869 Luke Daque: to understand like where this link from. And then out of that call he did mention. We’ll have to ignore this for now, and like these 2, for example, these were
170 00:17:24.849 ⇒ 00:17:33.530 Luke Daque: initially not in the in the stable. So these we will have to create like logic or calculated
171 00:17:35.040 ⇒ 00:17:38.080 Luke Daque: okay, so assessments, or whatever that we need.
172 00:17:38.730 ⇒ 00:17:41.550 Luke Daque: Yeah. And like, these are like.
173 00:17:42.530 ⇒ 00:17:47.349 Luke Daque: how we should be getting those. This, these are also coming from Mitch, basically.
174 00:17:48.300 ⇒ 00:17:50.809 Luke Daque: And so yeah, that’s like.
175 00:17:51.510 ⇒ 00:17:57.509 Luke Daque: that’s what I did. I didn’t really create any kind of mapping or something. Aside from just
176 00:17:58.030 ⇒ 00:18:02.849 Luke Daque: trying to look at the snowflake tables and see if these exist, and if not, then
177 00:18:04.650 ⇒ 00:18:10.380 Luke Daque: ask Mitch about it. How how do we get go about getting these?
178 00:18:11.510 ⇒ 00:18:12.350 Luke Daque: Do that? So.
179 00:18:12.350 ⇒ 00:18:19.620 Caio Velasco: So let let me see if I understand again. So this this spreadsheet came from the client, and this
180 00:18:19.790 ⇒ 00:18:24.979 Caio Velasco: initially, has nothing to do with what we have in the snow place, or does it.
181 00:18:27.720 ⇒ 00:18:30.260 Luke Daque: Do we probably bring into raw.
182 00:18:32.800 ⇒ 00:18:36.099 Caio Velasco: So why would would it? Wouldn’t it be there.
183 00:18:37.690 ⇒ 00:18:42.859 Luke Daque: Like not all of the fields. If I show you, let me log into my
184 00:18:44.950 ⇒ 00:18:49.929 Luke Daque: No, thank you. Not all the fields that they were asking for were in the raw
185 00:18:50.160 ⇒ 00:18:54.969 Luke Daque: data sets, and sometimes we need to like join them from other tables.
186 00:18:55.670 ⇒ 00:19:00.150 Luke Daque: And that’s where, like I had to like manually. Look, we wouldn’t.
187 00:19:07.046 ⇒ 00:19:08.500 Luke Daque: Let me.
188 00:19:23.080 ⇒ 00:19:26.130 Luke Daque: for example, if you go to subscription.
189 00:19:56.560 ⇒ 00:19:58.349 Luke Daque: So if you go here
190 00:20:03.291 ⇒ 00:20:05.399 Luke Daque: the only way to know.
191 00:20:06.130 ⇒ 00:20:14.570 Caio Velasco: Actually let let me let me ask you something that goes. I I think, one layer back. So before this, things have come into raw?
192 00:20:17.856 ⇒ 00:20:22.979 Caio Velasco: I think the question is, why exactly these things coming too wrong?
193 00:20:24.840 ⇒ 00:20:31.560 Caio Velasco: Do they know already with what they need to to give us? Or is it part also the discovery process.
194 00:20:32.600 ⇒ 00:20:34.839 Luke Daque: Yeah, we we did.
195 00:20:35.850 ⇒ 00:20:46.969 Luke Daque: So we use poly. They’re using polytonic for the data ingestion, like for stripe, for example, they’re using polytonic to ingest their stripe data into into the snowflake
196 00:20:47.728 ⇒ 00:20:51.790 Luke Daque: data warehouse. And then also, like they, they also have, like postgress
197 00:20:53.410 ⇒ 00:21:00.049 Luke Daque: tables that they wanted to put in here as well. So that’s also being added here.
198 00:21:00.260 ⇒ 00:21:07.360 Luke Daque: And then I’ll stripe, oh, yeah, stripe here, like some of the Google sheets for the mapping tables.
199 00:21:07.550 ⇒ 00:21:10.309 Luke Daque: because they also have some similar mapping.
200 00:21:11.040 ⇒ 00:21:18.650 Luke Daque: Just so we’ll know which of the code names are legacy, what tier they are. What’s linking the code name something like that.
201 00:21:18.810 ⇒ 00:21:24.530 Luke Daque: and then also and this one would be the events these are coming from segment.
202 00:21:24.630 ⇒ 00:21:28.050 Luke Daque: and I think I believe these were like added in.
203 00:21:28.948 ⇒ 00:21:30.640 Luke Daque: What do you call this?
204 00:21:31.250 ⇒ 00:21:38.389 Luke Daque: Through through segment? There’s like a direct connection from segment to snowflake. So they added, all these event tables to it.
205 00:21:39.080 ⇒ 00:21:40.390 Luke Daque: So yeah, we did.
206 00:21:40.960 ⇒ 00:21:49.390 Luke Daque: We did have. We did help them out without connections, like setting up a service account and
207 00:21:49.680 ⇒ 00:21:52.430 Luke Daque: and stuff like that. But like what the
208 00:21:52.560 ⇒ 00:21:55.880 Luke Daque: where to like the database for the raw database.
209 00:21:56.480 ⇒ 00:21:57.310 Luke Daque: So yeah.
210 00:21:58.550 ⇒ 00:21:59.010 Caio Velasco: Okay.
211 00:21:59.010 ⇒ 00:22:01.049 Luke Daque: And then, once these were in.
212 00:22:01.500 ⇒ 00:22:08.750 Luke Daque: then we I had to like, look for like for their one of their questions here regarding this subscription, or they asked.
213 00:22:09.220 ⇒ 00:22:10.940 Luke Daque: I had to look for
214 00:22:11.170 ⇒ 00:22:19.010 Luke Daque: like what potential tables we might be able to get those from. So in this case I found a subscriptions and subscription items.
215 00:22:19.250 ⇒ 00:22:21.659 Luke Daque: Table here that’s coming from stripe.
216 00:22:22.280 ⇒ 00:22:26.050 Luke Daque: And these have, you know.
217 00:22:26.590 ⇒ 00:22:27.050 Luke Daque: But yeah.
218 00:22:27.050 ⇒ 00:22:32.040 Caio Velasco: That’s a question. Well, in the spreadsheet, when you have the attribute subscription
219 00:22:34.330 ⇒ 00:22:39.079 Caio Velasco: or at least attribute group subscription detail that was made. Everything was made.
220 00:22:39.590 ⇒ 00:22:41.050 Caio Velasco: I decline.
221 00:22:41.400 ⇒ 00:22:41.830 Luke Daque: Yeah.
222 00:22:41.830 ⇒ 00:22:43.429 Caio Velasco: Subscriptions with 2.
223 00:22:43.550 ⇒ 00:22:47.920 Caio Velasco: And do you know where this come where this come from? Initially on their side?
224 00:22:48.670 ⇒ 00:22:49.250 Luke Daque: No.
225 00:22:49.250 ⇒ 00:22:55.400 Caio Velasco: For example. Why, if they have this, why they why wouldn’t they know that is in the subscription table, for example.
226 00:22:56.560 ⇒ 00:22:59.789 Luke Daque: Well at at this point, like Mitch already
227 00:23:00.140 ⇒ 00:23:13.540 Luke Daque: had a vague idea like where to get subscriptions from, like they they know that we have, like, they have it in stripe. So that’s why, yeah, that’s but yeah, like, this doesn’t have the complete.
228 00:23:13.990 ⇒ 00:23:18.130 Luke Daque: So yeah, this would have been better if, like, we can transfer this to the documentation. But
229 00:23:18.760 ⇒ 00:23:24.179 Luke Daque: we have standard dice that way. We can have the stores and then stuff like that.
230 00:23:25.670 ⇒ 00:23:28.510 Luke Daque: Yeah, I think I did start working on that
231 00:23:28.890 ⇒ 00:23:32.059 Luke Daque: for the subscriptions user and organizations.
232 00:23:32.940 ⇒ 00:23:38.680 Luke Daque: Yeah, let me see, let me find that, actually.
233 00:23:40.020 ⇒ 00:23:47.330 Luke Daque: But I did work on that. I did like manually add them there.
234 00:23:52.760 ⇒ 00:23:54.300 Luke Daque: Yeah, I think it’s Eastern.
235 00:23:56.700 ⇒ 00:24:01.840 Luke Daque: Yeah. So I added this, the source, for example, coming from post guest segment. Hubspot.
236 00:24:02.520 ⇒ 00:24:06.940 Luke Daque: So segment and Hubspot are both native. The others are like Polyton.
237 00:24:07.862 ⇒ 00:24:13.169 Luke Daque: Yeah, this is already the documentation that we wanted to be standardized.
238 00:24:13.860 ⇒ 00:24:19.189 Luke Daque: And then for the this one is, Yeah, I. I just basically added this. But
239 00:24:19.420 ⇒ 00:24:26.719 Luke Daque: this was already after the time. So I already worked on the March models before we discussed about this standard
240 00:24:27.170 ⇒ 00:24:28.060 Luke Daque: template.
241 00:24:28.790 ⇒ 00:24:35.270 Luke Daque: So basically, what I did is I just copied all the fields coming from the snowflake table, from the March models.
242 00:24:35.760 ⇒ 00:24:39.120 Luke Daque: And like what the source tables are.
243 00:24:41.230 ⇒ 00:24:44.159 Luke Daque: Yeah, like Skype and stuff like that.
244 00:24:44.660 ⇒ 00:24:46.810 Luke Daque: But I guess ideally, this would
245 00:24:47.010 ⇒ 00:24:51.300 Luke Daque: happen before creating the actual markets. Models that way we can.
246 00:24:52.600 ⇒ 00:24:56.660 Luke Daque: You can go back to the client, for example, if you don’t know where this
247 00:24:56.780 ⇒ 00:25:00.040 Luke Daque: specific field is coming from, what table? It’s coming from
248 00:25:00.180 ⇒ 00:25:06.809 Luke Daque: it. Then, yeah, we can. We can discuss that. Or if there’s any kind of logic like this one.
249 00:25:07.501 ⇒ 00:25:09.160 Luke Daque: At this point I just.
250 00:25:09.450 ⇒ 00:25:12.579 Luke Daque: I already have the logic. So I added it here. But then.
251 00:25:12.860 ⇒ 00:25:18.910 Luke Daque: yeah, we can ask this directly. Basically, with the client
252 00:25:20.110 ⇒ 00:25:24.259 Luke Daque: like, it doesn’t have to be a case. Wednesday something it can be like just
253 00:25:24.440 ⇒ 00:25:34.130 Luke Daque: just how we did it here, like it says, indicates, if this is the 1st time user, so it can be just this, and then have to figure out what the logic is for for this specific.
254 00:25:34.490 ⇒ 00:25:35.850 Luke Daque: Ask Greg.
255 00:25:38.060 ⇒ 00:25:40.310 Caio Velasco: Okay, I’ll see.
256 00:25:42.790 ⇒ 00:25:44.150 Caio Velasco: Yeah.
257 00:25:46.540 ⇒ 00:25:49.530 Luke Daque: And then, what else did I have in here?
258 00:25:51.320 ⇒ 00:25:54.319 Luke Daque: Yeah, I think that’s that’s about it for for this.
259 00:25:56.320 ⇒ 00:26:00.479 Luke Daque: hey? Guys? Yeah. Oh, yeah, hey, welcome.
260 00:26:00.480 ⇒ 00:26:00.880 Uttam Kumaran: Hey!
261 00:26:00.880 ⇒ 00:26:01.600 Luke Daque: Good morning.
262 00:26:02.200 ⇒ 00:26:03.790 Luke Daque: We’re just thinking.
263 00:26:04.550 ⇒ 00:26:11.379 Luke Daque: But like, how like staff, how how like the
264 00:26:12.440 ⇒ 00:26:22.479 Luke Daque: I I like, how we are like in stack with like how we are getting the request from the client like Kyle App is asking, How do we know where will these come from?
265 00:26:23.450 ⇒ 00:26:25.730 Luke Daque: And like the sources and stuff like that?
266 00:26:26.650 ⇒ 00:26:34.339 Uttam Kumaran: Yeah, I I’m you know. I’m thinking about like how we can do this in a more automated way.
267 00:26:36.490 ⇒ 00:26:44.089 Uttam Kumaran: I’m you know. I’m talking to a few vendors that do this like that aren’t like. There’s a couple of companies like Colibra Atlin that do this, but
268 00:26:44.190 ⇒ 00:26:49.899 Uttam Kumaran: they’re so expensive, and I don’t. I’m not really interested in adding like a whole nother
269 00:26:52.400 ⇒ 00:26:59.640 Uttam Kumaran: I’m not interested in adding like a whole nother tool, but I am looking for a way to do this, I mean for smaller clients.
270 00:26:59.950 ⇒ 00:27:05.249 Uttam Kumaran: I think if there’s limited tables, this is fine, but for bigger clients, where, like.
271 00:27:05.440 ⇒ 00:27:10.390 Uttam Kumaran: think about clients where we have, like a hundred models like even for Eden, we already are at like
272 00:27:10.510 ⇒ 00:27:12.530 Uttam Kumaran: probably 40 Martz models.
273 00:27:13.100 ⇒ 00:27:13.415 Luke Daque: Yeah.
274 00:27:13.730 ⇒ 00:27:16.540 Uttam Kumaran: There’s no way like it’s so hard to document.
275 00:27:16.930 ⇒ 00:27:17.410 Uttam Kumaran: You’ve been.
276 00:27:17.410 ⇒ 00:27:18.160 Uttam Kumaran: I mean, like, yeah, yeah.
277 00:27:18.160 ⇒ 00:27:22.700 Luke Daque: It’s just like 3 model 4 models here. It took quite a while to actually like.
278 00:27:23.470 ⇒ 00:27:24.690 Luke Daque: put this in here.
279 00:27:24.860 ⇒ 00:27:25.480 Luke Daque: So.
280 00:27:25.480 ⇒ 00:27:35.210 Uttam Kumaran: Yeah, I think let’s bring it up also. Let’s bring it up into the in the Friday Retro. I’ll see if I can get a demo ready by then to share with you guys
281 00:27:35.590 ⇒ 00:27:38.889 Uttam Kumaran: of like a pro of like a proof of concept of something. But
282 00:27:39.280 ⇒ 00:27:42.569 Uttam Kumaran: we need to try to make this a little bit easier.
283 00:27:42.760 ⇒ 00:27:43.260 Uttam Kumaran: I mean.
284 00:27:43.260 ⇒ 00:27:43.770 Luke Daque: Yeah.
285 00:27:43.770 ⇒ 00:27:51.489 Uttam Kumaran: One thing is actually getting the the things in here. But also it’s like, sometimes like.
286 00:27:51.760 ⇒ 00:27:53.201 Uttam Kumaran: I don’t know. I think.
287 00:27:55.290 ⇒ 00:27:59.060 Uttam Kumaran: I wonder if we can use AI to do the 1st 1st version of this.
288 00:28:02.370 ⇒ 00:28:03.149 Uttam Kumaran: I don’t know.
289 00:28:03.710 ⇒ 00:28:04.520 Luke Daque: Yeah.
290 00:28:05.930 ⇒ 00:28:10.409 Uttam Kumaran: So I don’t want you guys spending this much time. Well, I don’t know. It’s worth doing once
291 00:28:10.560 ⇒ 00:28:15.150 Uttam Kumaran: and then figuring out automate, you know, because otherwise we’re not going to know what the real pain points are. But.
292 00:28:15.540 ⇒ 00:28:16.230 Luke Daque: Yeah.
293 00:28:16.770 ⇒ 00:28:17.790 Uttam Kumaran: I agree.
294 00:28:19.355 ⇒ 00:28:47.520 Caio Velasco: My question to Luke, so that you can understand why I was asking. This is because, for example, I, I watched the the videos from from us. And then I went to Meta Base to to check what he was doing. And then I went to a specific metric in the macro part, for example. And then I went into the the database in Meta base, you pointing to that metric.
295 00:28:47.590 ⇒ 00:28:56.309 Caio Velasco: And then it’s like, okay. So there’s a a bigger thing they did here. And then I went back into the Dbt models. And I was trying to understand, like, for example, just the fact tickets.
296 00:28:56.400 ⇒ 00:28:59.579 Caio Velasco: It’s like, of course, a joint of many other things. So
297 00:28:59.730 ⇒ 00:29:20.519 Caio Velasco: when but this is post work like, after everything is done, then I can kind of reverse engineering, understand? Where does it come from? But when I’m starting the the example, a gorgeous dashboard, or any any metric or something, I always wonder like, what is the process to get to that point
298 00:29:20.720 ⇒ 00:29:39.470 Caio Velasco: to avoid the logic? The, of course, the logic is there at the end of the day, but it’s on, always on someone’s mind, and I think for some reason I feel that when I’m the the documentation is not something that should be done after the for me. The documentation is part of the process of getting there.
299 00:29:39.920 ⇒ 00:29:46.439 Caio Velasco: Maybe it’s a it’s a skill that I have to learn, but I just don’t know how to do it without documenting.
300 00:29:46.440 ⇒ 00:29:47.110 Uttam Kumaran: Yeah, yeah, yeah.
301 00:29:47.110 ⇒ 00:29:48.610 Caio Velasco: First.st You know what I mean. I don’t know.
302 00:29:48.610 ⇒ 00:29:51.690 Uttam Kumaran: Yeah, yeah, no, no. I think that’s I think that’s just different
303 00:29:51.690 ⇒ 00:29:54.349 Uttam Kumaran: people. I think it’s just different one. It’s like.
304 00:29:54.460 ⇒ 00:30:00.830 Uttam Kumaran: I’ve been staring at the Javi data for 6 months, right? And so has, like a lot of people on that client.
305 00:30:01.110 ⇒ 00:30:08.819 Uttam Kumaran: So like, there is some knowledge that’s like, if you ask me that there’s an issue with orders. I know the 3 tables that feed that
306 00:30:08.980 ⇒ 00:30:14.919 Uttam Kumaran: that’s just in my brain, and so that just probably comes with time. But you’re right in that.
307 00:30:16.200 ⇒ 00:30:21.139 Uttam Kumaran: It’s something that anybody on the team should be able to pick up right like, for example.
308 00:30:21.270 ⇒ 00:30:31.320 Uttam Kumaran: people are always going to be out right. But let’s say something happens. And we want to try to do some like, okay, can you cover me on this client issues? Come, how do we all resolve those? Right? And so
309 00:30:31.430 ⇒ 00:30:47.199 Uttam Kumaran: I think that’s that’s an example. And yeah, I think we will arrive at this. I’m I’m talking to them a lot about this. We have a couple of versions, so I think in the next 2 weeks we’ll we’ll have a better version of this. We’ll have both like some people. Are some people more like, Hey, I just need to go. Query the tables.
310 00:30:47.340 ⇒ 00:30:49.599 Uttam Kumaran: Some people are like, I need to go look at the code.
311 00:30:49.790 ⇒ 00:30:56.189 Uttam Kumaran: Some folks were like, okay, let me look at the lineage. I think we I want to have all 3 available because all 3 serve different purposes.
312 00:30:59.090 ⇒ 00:31:02.400 Uttam Kumaran: You know. But it it makes sense, I mean, I I just think it’s just like
313 00:31:03.670 ⇒ 00:31:23.960 Uttam Kumaran: there, there will be a time when when you get sophisticated with the client where a lot of this questions you won’t need to go. Reference, because you’ll know. Okay, this these it’s like, I’m sure, your last one. You’re like, okay, it’s always this table has problems, or, like these 5 columns always have issues it all. It usually resolves to that. It’s like 80 20, right? Like, for example.
314 00:31:24.170 ⇒ 00:31:39.839 Uttam Kumaran: a question, a model that’s just doing a straight select from portable, not gonna have much problems, but like something like fact, orders where there’s like 10 layers. And there’s all this stuff it’s challenging. However, I I also I don’t know. This is maybe a thing we should talk about
315 00:31:40.670 ⇒ 00:31:48.160 Uttam Kumaran: on Friday is like how we should take time out of our schedule to go do this to do 2 things, one
316 00:31:48.440 ⇒ 00:31:52.740 Uttam Kumaran: to work on this sort of documentation. The second piece is also for like
317 00:31:53.130 ⇒ 00:31:56.210 Uttam Kumaran: for stuff that’s like entrenched knowledge, like.
318 00:31:56.640 ⇒ 00:32:06.580 Uttam Kumaran: for example, I know, like, I know, if there’s a job issue what the top 5 problems could be. But like, can we put that somewhere? Can I get that out of my brain? Is that like.
319 00:32:07.080 ⇒ 00:32:12.360 Uttam Kumaran: I don’t know whether that lives in this document, whether we need like an Faqs. Whether we need like.
320 00:32:12.680 ⇒ 00:32:17.010 Uttam Kumaran: Do you know what I mean? Like? It’s a cheat sheet, right? Basically.
321 00:32:17.250 ⇒ 00:32:17.790 Caio Velasco: Yep.
322 00:32:18.980 ⇒ 00:32:45.750 Caio Velasco: And and another question that I would have. It comes even before that. For example, when I I saw some questions, I think, from Robert regarding does this, whatever variable includes shipping cost or not? But is this a question that we have? Because we are trying to discover something new, or the client would already have this answer, because they have to find it, because if they have.
323 00:32:45.750 ⇒ 00:32:49.530 Uttam Kumaran: Yeah. Yeah. So let me. So that’s a good. So that’s a good thing. And there’s probably.
324 00:32:49.830 ⇒ 00:32:55.299 Uttam Kumaran: you know, whereas we’re thinking about playbooks, you know, I’m sending messages about right working on these playbooks.
325 00:33:00.171 ⇒ 00:33:02.030 Uttam Kumaran: one of the things that
326 00:33:02.630 ⇒ 00:33:10.920 Uttam Kumaran: we’re noticing is that when analysts are sending dashboards they’re not explaining the logic. And so when Robert goes through, he’s like.
327 00:33:11.490 ⇒ 00:33:13.489 Uttam Kumaran: does this include shipping or not?
328 00:33:14.120 ⇒ 00:33:26.020 Uttam Kumaran: Right? That question is actually not. It’s just for, like the whole team. But of course, like our our job kind of like, we, we haven’t built a dashboard. So really, that’s a question for the analyst.
329 00:33:26.280 ⇒ 00:33:32.500 Uttam Kumaran: But also, it’s like, where can he go to at least try to validate that he goes into Meta Base? He sees the metric.
330 00:33:32.820 ⇒ 00:33:35.630 Uttam Kumaran: And then he where does he go after that?
331 00:33:36.130 ⇒ 00:33:40.989 Uttam Kumaran: Right? So that was like he saw gross margin. He’s like, does this include shipping or not?
332 00:33:41.570 ⇒ 00:33:44.220 Uttam Kumaran: Okay, what are the steps to diagnose that, you know.
333 00:33:44.370 ⇒ 00:33:55.030 Uttam Kumaran: like, okay, he has to go into the into the Meta base tile. Look at the select statement. Okay, it’s not doing any more logic here, or selecting from a Dbt table.
334 00:33:55.180 ⇒ 00:34:04.470 Uttam Kumaran: Okay, let me go to the data platform documentation. Let me look for the table. Let me go find the column. Let me go. See the logic. Is that it? Right? I don’t know.
335 00:34:05.361 ⇒ 00:34:10.139 Uttam Kumaran: My like. My process is like I go straight to the repo, and I look at the
336 00:34:10.929 ⇒ 00:34:14.849 Uttam Kumaran: so I do 2 things. I go straight to the repo, or I go into cursor, and I ask.
337 00:34:15.110 ⇒ 00:34:17.440 Uttam Kumaran: I asked the AI the question.
338 00:34:17.800 ⇒ 00:34:22.120 Uttam Kumaran: I both of those things usually gets me to the answer pretty quickly. That’s my process.
339 00:34:25.510 ⇒ 00:34:26.620 Caio Velasco: Okay.
340 00:34:26.850 ⇒ 00:34:28.270 Demilade Agboola: It sounds.
341 00:34:29.610 ⇒ 00:34:31.959 Demilade Agboola: It sounds like there’s a bit of a
342 00:34:32.139 ⇒ 00:34:49.759 Demilade Agboola: huge divide, but there’s a bit of a divide between. How like the analysts work and how like the A’s work. And it’s kind of tied to the question which I asked in the data teams, where, if the if there are a lot of facts and the dimension tables in the math.
343 00:34:50.210 ⇒ 00:35:00.770 Demilade Agboola: It means the analysts are doing a lot of the logic as well in the bi tools. Which means that there is that device. So you don’t necessarily know how the analysts are using it to
344 00:35:01.207 ⇒ 00:35:10.739 Demilade Agboola: create the final dashboard. So you just know that they’re facts and dimension tables that you’ve presented to them, and you’ve created for them. But you don’t necessarily always know
345 00:35:10.950 ⇒ 00:35:16.280 Demilade Agboola: how it was used to calculate. Say, yeah, 6, and all of that. So
346 00:35:16.450 ⇒ 00:35:32.120 Demilade Agboola: in that case, is there a way we can like? Which is the question I asked. We can move that, like some of that logic and aggregation back into like Dbt. In such a way that at least now the analysts have an idea the analyst can also set up things like Dbt. Tests, and at least can set up.
347 00:35:32.120 ⇒ 00:35:32.570 Uttam Kumaran: Yeah.
348 00:35:32.570 ⇒ 00:35:43.359 Demilade Agboola: Incrementality. So the the tables don’t have to like. Because, you know, we like a lot of these bi tools. They’re running the joins and everything from scratch every single day, but we could set up.
349 00:35:43.360 ⇒ 00:35:44.160 Uttam Kumaran: Yeah, yeah.
350 00:35:44.160 ⇒ 00:35:45.870 Demilade Agboola: Models and things like that.
351 00:35:46.360 ⇒ 00:35:50.360 Demilade Agboola: So yeah, it was just something I was thinking about when I was looking at Javi.
352 00:35:51.170 ⇒ 00:35:55.999 Uttam Kumaran: Yeah, so let’s okay, let’s actually, I mean, we have 10 min. I would love to
353 00:35:56.260 ⇒ 00:35:59.390 Uttam Kumaran: like, I actually wanna do this conversation today.
354 00:35:59.913 ⇒ 00:36:14.116 Uttam Kumaran: I know, I I really, I mean, we could. We could do stand up. I mean, we’re gonna have. Stand up. Let’s see if we can get through that quickly. We can continue on this. But let me, let’s actually like, let’s just spend 10 min talking about this. I’m gonna let me just
355 00:36:14.630 ⇒ 00:36:18.930 Uttam Kumaran: Luke, if you don’t mind, I’m gonna share. And I’m just gonna move this to our.
356 00:36:19.160 ⇒ 00:36:22.581 Uttam Kumaran: I’m just gonna open our like retro
357 00:36:24.050 ⇒ 00:36:29.079 Uttam Kumaran: basically our like retro, fig jam. And like, let’s just talk about like what this process
358 00:36:31.590 ⇒ 00:36:32.919 Uttam Kumaran: could look like.
359 00:36:33.380 ⇒ 00:36:35.010 Uttam Kumaran: Let’s see.
360 00:36:39.410 ⇒ 00:36:46.279 Uttam Kumaran: like, okay, we have this like playbooks thing. Okay, let’s let’s have this as like the problem statement.
361 00:36:51.380 ⇒ 00:36:54.035 Uttam Kumaran: sorry, I know I’m not sharing anything. Okay here.
362 00:36:54.650 ⇒ 00:37:00.555 Uttam Kumaran: I’m sending the link here. But I’m also gonna share. Then follow me, or whatever
363 00:37:01.700 ⇒ 00:37:10.470 Uttam Kumaran: in figma. If you actually click on the on my my profile, icon, you can follow me. But okay, so let’s just talk about this. So
364 00:37:11.780 ⇒ 00:37:20.970 Uttam Kumaran: notice a lot of models in the Javi. More fact and did would suggest some of the modeling occurs. Not sure if there’s a standard across clients, what’s the process? Increase these aggregation? Okay, so let’s talk about like.
365 00:37:21.310 ⇒ 00:37:25.532 Uttam Kumaran: let’s talk about what the problem is here.
366 00:37:28.120 ⇒ 00:37:33.020 Uttam Kumaran: right? And then we’ll talk about the problem. I want to talk about the problem. And then I want to talk about the symptoms.
367 00:37:33.360 ⇒ 00:37:36.064 Uttam Kumaran: That’s 1 thing that I don’t know. I’ve been watching this.
368 00:37:37.400 ⇒ 00:37:42.090 Uttam Kumaran: I’ve been watching this show on, on on Hbo called the pit recently.
369 00:37:42.820 ⇒ 00:37:45.090 Uttam Kumaran: It’s a really great show. It’s like a
370 00:37:45.480 ⇒ 00:37:49.619 Uttam Kumaran: about an emergency room, and every episode is A is an hour in the emergency room.
371 00:37:49.820 ⇒ 00:37:56.199 Uttam Kumaran: But I have all this medical lingo so sorry if I start using like a lot of medical terms. But the symptoms
372 00:37:56.490 ⇒ 00:37:57.980 Caio Velasco: No worries, no worries. That’s good.
373 00:37:57.980 ⇒ 00:38:04.940 Uttam Kumaran: Le, let’s talk about the problem like I, I honestly think the symptoms are easier to talk about than the problem. Right?
374 00:38:06.290 ⇒ 00:38:11.620 Uttam Kumaran: So let’s actually talk about. Let’s talk about yeah. I mean, let’s just keep it here.
375 00:38:12.347 ⇒ 00:38:14.489 Uttam Kumaran: And let’s talk about what the symptoms are.
376 00:38:15.192 ⇒ 00:38:21.060 Uttam Kumaran: Actually, maybe let’s just use post-its. And each of us can we can put so
377 00:38:21.060 ⇒ 00:38:29.079 Uttam Kumaran: that that’s a good point, because you you, the problem is diagnosed after symptom right? That makes total sense. Yeah.
378 00:38:29.080 ⇒ 00:38:34.230 Uttam Kumaran: yeah, like, that’s the thing is, sometimes people cause people will say, my leg hurts.
379 00:38:34.950 ⇒ 00:38:38.139 Uttam Kumaran: That’s not the that’s not the problem, right?
380 00:38:38.280 ⇒ 00:38:42.260 Uttam Kumaran: So I want to talk about like, how does this actually manifest into issues?
381 00:38:43.990 ⇒ 00:38:46.019 Uttam Kumaran: So the 1st thing is that
382 00:38:47.640 ⇒ 00:38:58.530 Uttam Kumaran: there, there, there is logic in the bi tool that no one is accountable for right.
383 00:38:59.160 ⇒ 00:39:03.920 Uttam Kumaran: For example, if Pious goes ahead and adds, shipping costs there.
384 00:39:04.720 ⇒ 00:39:08.899 Uttam Kumaran: whose fault is that? And I I wanna say fault, I mean, of course, like
385 00:39:09.240 ⇒ 00:39:17.600 Uttam Kumaran: it’s like it’s not like that serious but like who like who is on the hook for that right? Is it our team. Is it their team? So that’s totally one thing.
386 00:39:18.256 ⇒ 00:39:19.290 Uttam Kumaran: What else?
387 00:39:24.540 ⇒ 00:39:27.030 Uttam Kumaran: Like, I think enough. Yeah, yeah. Go ahead.
388 00:39:27.560 ⇒ 00:39:32.720 Demilade Agboola: I’ll probably say that changes to the
389 00:39:32.820 ⇒ 00:39:39.219 Demilade Agboola: logic in terms of like data quality and like change rapidly, and no one might necessarily be aware.
390 00:39:39.660 ⇒ 00:39:44.870 Demilade Agboola: because, like, we could like, if you have everything in one table, you could set up data tests, for instance.
391 00:39:45.800 ⇒ 00:39:55.420 Demilade Agboola: so that like, if you have a joint that like is one too many windows expected to be one to one. You can catch it in Dvt. But you might not catch it if it’s in the
392 00:39:56.260 ⇒ 00:39:58.050 Demilade Agboola: in the tableau layer.
393 00:40:01.870 ⇒ 00:40:06.950 Uttam Kumaran: So analysts are not aware of changes.
394 00:40:07.840 ⇒ 00:40:16.330 Uttam Kumaran: Okay, I would say, there’s another problem. Here is how time to answer.
395 00:40:16.750 ⇒ 00:40:21.520 Uttam Kumaran: How is this metric defined right.
396 00:40:22.150 ⇒ 00:40:24.489 Uttam Kumaran: This takes hours.
397 00:40:25.470 ⇒ 00:40:26.270 Caio Velasco: Exactly.
398 00:40:26.990 ⇒ 00:40:32.080 Uttam Kumaran: I don’t wanna put days, but it also may take days right now.
399 00:40:34.300 ⇒ 00:41:00.179 Caio Velasco: I go even like before, like, for example, I don’t know if it’s exactly what you want here, but when when I 1st had the the ticket about like, okay, we need to build that gorgeous dashboard. And then there was a 1st question, like, what what Macros are being used the most. And then I was like, Okay, what is a macro? Let’s say the macro is fine. Just go in the website. It’s a text. You understand. This is a macro fine, but if it’s
400 00:41:00.660 ⇒ 00:41:07.580 Caio Velasco: gross margin, and there is 100 variables inside it, it should be somewhere. We don’t have to like all the time. Would
401 00:41:07.750 ⇒ 00:41:21.840 Caio Velasco: kind of like, go to the client, and and that’s like, Hey, is this part of it or not? Or like, where can I find it? I think those things, even though we know that most of the times I I won’t believe that the back end engineers would just document everything this way.
402 00:41:22.020 ⇒ 00:41:24.499 Caio Velasco: But I think this is part of the job
403 00:41:24.930 ⇒ 00:41:28.720 Caio Velasco: we have to like, at least in the beginning. It should be somewhere.
404 00:41:29.760 ⇒ 00:41:35.569 Uttam Kumaran: So this is almost like a I mean, it’s like the
405 00:41:36.430 ⇒ 00:41:39.550 Uttam Kumaran: maybe this I don’t know. I think the symptom is that
406 00:41:39.870 ⇒ 00:41:42.760 Uttam Kumaran: we have a click, a cold start problem, right?
407 00:41:43.100 ⇒ 00:41:45.839 Uttam Kumaran: Yep. That’s how it’s sometimes described in like
408 00:41:46.880 ⇒ 00:41:53.630 Uttam Kumaran: in physics. And then other stuff which is like analysts.
409 00:41:54.230 ⇒ 00:42:01.590 Uttam Kumaran: Ask questions, takes hours to make.
410 00:42:02.720 ⇒ 00:42:08.590 Uttam Kumaran: I mean I I will say analysts ask a question where and then like
411 00:42:08.850 ⇒ 00:42:17.709 Uttam Kumaran: provides little help for ae on where to begin. So this is a symptom like we don’t have to talk. I think the one thing to
412 00:42:17.820 ⇒ 00:42:21.220 Uttam Kumaran: the one thing that’s tough in these moments. And again, this is
413 00:42:21.370 ⇒ 00:42:25.823 Uttam Kumaran: this is something that I learned a lot when I was a Pm, is that
414 00:42:27.250 ⇒ 00:42:54.059 Uttam Kumaran: When you’re talking like, think it’s it’s it’s funny, because at the moment, like we are, we are our own doctors. Right? Like, we are gonna diagnose our own problems here. But it’s helpful to talk about the problems. And before we solution, right? Cause? The answer to this could be several things right? And that’s why I want to throw out like, consider that we don’t have any solutions for this at the moment, and we can do a hundred things.
415 00:42:54.490 ⇒ 00:43:06.379 Uttam Kumaran: I want us to also think about how like, what are the ways we judge whether our solution is proper, right? Similar to when we push a Dbt model, how do we know that
416 00:43:06.600 ⇒ 00:43:18.140 Uttam Kumaran: it’s it works like we check that the model runs. That’s it, right? We don’t have tests. So similarly, when we solve these problems, what are our tests that these solutions are accurate?
417 00:43:19.660 ⇒ 00:43:25.580 Uttam Kumaran: so I think that’s 1 thing, that as we go through the retros, and as we go through these over the next forever.
418 00:43:25.740 ⇒ 00:43:33.190 Uttam Kumaran: we’ll we’ll start to look at that. But I think these are good enough. These are good problems, like, I think I think the last piece here is.
419 00:43:33.470 ⇒ 00:43:35.240 Uttam Kumaran: I mean, this is roughly like.
420 00:43:35.410 ⇒ 00:43:39.510 Uttam Kumaran: I think again, like, I think this question is.
421 00:43:39.710 ⇒ 00:43:49.200 Uttam Kumaran: there’s a process that integrates these aggregations back into the mart. Basically, like analysts are doing too much modeling
422 00:43:49.390 ⇒ 00:43:53.399 Uttam Kumaran: in the the eye layer. Right? That’s the symptom.
423 00:43:56.180 ⇒ 00:44:05.739 Uttam Kumaran: okay, let’s just talk. Let’s talk through these. Unless does anyone else have any other items we want to add here and again, there’s a hundred 100,000 problems right now, but
424 00:44:06.100 ⇒ 00:44:07.860 Uttam Kumaran: maybe we could just talk through these.
425 00:44:09.200 ⇒ 00:44:09.870 Caio Velasco: Yep.
426 00:44:10.870 ⇒ 00:44:11.670 Uttam Kumaran: Okay.
427 00:44:11.950 ⇒ 00:44:14.334 Uttam Kumaran: So I think that
428 00:44:17.960 ⇒ 00:44:23.060 Uttam Kumaran: this and this, these are sort of related right?
429 00:44:23.840 ⇒ 00:44:27.569 Uttam Kumaran: And then let’s talk about. Let’s talk about like the problem
430 00:44:28.610 ⇒ 00:44:35.504 Uttam Kumaran: associated with it. Right? So I’ll just make this like, so the problem here.
431 00:44:40.940 ⇒ 00:44:47.130 Uttam Kumaran: the problem here is that there are. Yes, I think one. There’s a couple of problems. One is.
432 00:44:47.330 ⇒ 00:44:55.640 Uttam Kumaran: if analysts are doing a lot of modeling in the bi layer, it means the data. Mart is not robust enough. However, I guess demolate. I’ll probably play the
433 00:44:55.810 ⇒ 00:44:57.482 Uttam Kumaran: I’ll play the other
434 00:44:58.010 ⇒ 00:45:09.200 Uttam Kumaran: I’ll play. The other side is, should we allow simple joins in the bi tool simple, join simple filtering.
435 00:45:10.160 ⇒ 00:45:28.670 Demilade Agboola: I think simple join simple filtering. I guess that is somewhat fine, but I think I prefer like generally when when I think of the bi tools modeling. I think of more of like ad hoc, quick analysis. We want to be able to get something out quickly. And we want to be able to understand the scope of what works and what doesn’t work.
436 00:45:29.380 ⇒ 00:45:44.169 Demilade Agboola: But like, I think, more of the mark in Dbt, as like long term sustainable modeling. So how do we want this to operate every single day over the course of the lifespan of this dashboard?
437 00:45:46.440 ⇒ 00:45:54.930 Uttam Kumaran: So I guess. Let me let me put in the problem of the dashboard right? So what happens with the dashboard which which lives.
438 00:45:56.200 ⇒ 00:46:03.450 Uttam Kumaran: you know, for a while it’s not a point in time. It’s maybe at least 3 months, if not longer, it will exist.
439 00:46:04.930 ⇒ 00:46:10.179 Uttam Kumaran: I see that as a different deliverable than like an analysis.
440 00:46:10.310 ⇒ 00:46:11.170 Uttam Kumaran: Right?
441 00:46:11.760 ⇒ 00:46:17.870 Uttam Kumaran: So how can we judge like for use cases like that? How can we judge that
442 00:46:18.410 ⇒ 00:46:29.530 Uttam Kumaran: we’re we’re aware of like the fact that they’re they’re doing modeling there. And it’s fine. I guess what I’m trying to say is, what level of modeling are we okay with in the bi tool. And what are we not?
443 00:46:31.390 ⇒ 00:46:35.999 Uttam Kumaran: to give you? I mean, we all know the examples you can do filtering, you can add dimensions, you can add.
444 00:46:37.095 ⇒ 00:46:41.350 Uttam Kumaran: metrics, there’s degrees of which we’re probably comfortable with.
445 00:46:43.040 ⇒ 00:46:58.599 Demilade Agboola: Yeah, okay. So like, I said, things simple drives are fine. I think once you start adding, like more complicated logic. Or you start adding, you know, just more complicated logic or case when statements things like that like when you’re trying to add like business logic, I think we can move it to Dbt.
446 00:46:59.060 ⇒ 00:46:59.720 Uttam Kumaran: Okay.
447 00:47:01.530 ⇒ 00:47:09.269 Uttam Kumaran: And and and you guys know, if you guys have worked with analysts, they’re always gonna play the other side, right? So I’ll be. I’ll be here. I’ll kind of represent that, and
448 00:47:09.410 ⇒ 00:47:14.345 Uttam Kumaran: maybe we’ll just spend like maybe one or 2 more minutes. We can jump to the other meeting, and we can continue but
449 00:47:14.740 ⇒ 00:47:31.589 Uttam Kumaran: the analysts. And and again, since I see both sides, I’m always pushing both sides to sort of have this conflict, because we’re always gonna have this conflict right? And you guys know our ability to move fast on the analyst side may require them to do some of these things.
450 00:47:31.740 ⇒ 00:47:39.040 Uttam Kumaran: But what is the what is 2 things? One. In the event that something is not urgent. We need to have a process that works for them
451 00:47:39.270 ⇒ 00:47:46.960 Uttam Kumaran: right. This process may work for us. But if you’re a full time analyst, and you’re spending, like.
452 00:47:47.190 ⇒ 00:47:55.679 Uttam Kumaran: you know, a half day waiting or 2 days waiting for stuff to get pushed. And you can’t conduct the analysis for a client, for a meeting that’s happening on Friday.
453 00:47:55.830 ⇒ 00:48:09.499 Uttam Kumaran: That’s the real problem. We’re having. Right. The second piece is, maybe we should have an agreement that, hey? If you do some modeling, you need to have a process by which that modeling gets put back into the warehouse
454 00:48:09.800 ⇒ 00:48:10.590 Uttam Kumaran: right?
455 00:48:10.770 ⇒ 00:48:16.390 Uttam Kumaran: I am more on that side because I’ve never been part of a data team where this hasn’t been a problem.
456 00:48:16.900 ⇒ 00:48:23.510 Uttam Kumaran: and this never. This hasn’t always been a work in progress. Meaning we’re always moving complicated logic into the warehouse.
457 00:48:23.700 ⇒ 00:48:30.140 Uttam Kumaran: If we ask them to wait for us. It may take them 3 days to to do something that may have taken them
458 00:48:30.690 ⇒ 00:48:35.470 Uttam Kumaran: just a little bit of time. So that’s I guess, where I would play the other side.
459 00:48:40.670 ⇒ 00:48:43.679 Demilade Agboola: Oh, yes, I agree with that. I I know that.
460 00:48:43.860 ⇒ 00:48:55.130 Demilade Agboola: you know. That’s part of why I said like fact, and D models are great in the month, for like quick ad hoc analysis, quick ad hoc reports because people need altitudes like asap
461 00:48:56.024 ⇒ 00:48:58.745 Demilade Agboola: but in terms of sustainability.
462 00:48:59.480 ⇒ 00:49:07.210 Demilade Agboola: yeah, that’s always the real question which otherwise that you know, like, I think of moving things back into the math as like a long term play.
463 00:49:07.530 ⇒ 00:49:09.940 Demilade Agboola: How? How so simple is it.
464 00:49:12.230 ⇒ 00:49:17.470 Luke Daque: Yeah, I think that’s true as well to what I think. That’s what we are already currently doing with
465 00:49:17.630 ⇒ 00:49:18.700 Luke Daque: the other
466 00:49:18.910 ⇒ 00:49:24.935 Luke Daque: clients. It’s using real right cause. We don’t do any kind of modeling in the real models, because
467 00:49:25.270 ⇒ 00:49:26.050 Uttam Kumaran: Yes.
468 00:49:26.050 ⇒ 00:49:30.875 Luke Daque: So yeah, cause that’s already, like, yeah, duplication of stuff.
469 00:49:33.880 ⇒ 00:49:36.370 Luke Daque: So yeah, I, I think we should.
470 00:49:37.220 ⇒ 00:49:39.449 Luke Daque: Yeah, it’s it’s a hard problem to solve.
471 00:49:39.850 ⇒ 00:49:46.110 Uttam Kumaran: Let’s jump to the other meeting. Let’s see if we can continue the conversation towards the end of that, if we can get through the updates.
472 00:49:46.460 ⇒ 00:49:49.040 Uttam Kumaran: This is a good conversation. I want to finish this up today.
473 00:49:49.730 ⇒ 00:49:51.639 Luke Daque: Cool. Sounds good to see you. Then. Okay, bye, bye.
474 00:49:51.640 ⇒ 00:49:52.650 Uttam Kumaran: Alright. Talk to you guys. There.