Meeting Title: Urban Stems | Weekly Sync Date: 2025-05-01 Meeting participants: Demilade Agboola, Amber Lin


WEBVTT

1 00:02:16.500 00:02:17.840 Amber Lin: Hi!

2 00:02:32.130 00:02:35.729 Amber Lin: Alright! I’ll share my screen, and

3 00:02:36.590 00:02:40.350 Amber Lin: I just wanna hear your thoughts on this roadmap.

4 00:02:45.850 00:02:47.440 Amber Lin: Alright. Thank you. Now.

5 00:02:49.020 00:02:50.770 Demilade Agboola: Some reason I can’t seem to hear you.

6 00:02:51.590 00:02:52.210 Amber Lin: Yeah.

7 00:02:55.430 00:02:56.470 Amber Lin: Sorry.

8 00:03:05.180 00:03:06.390 Amber Lin: Hello.

9 00:03:06.630 00:03:09.590 Demilade Agboola: Okay, I can hear you. Now, what are you talking for?

10 00:03:10.060 00:03:14.419 Amber Lin: Yeah, I was talking a little bit. Maybe my Internet wasn’t as good.

11 00:03:14.640 00:03:19.699 Demilade Agboola: Yeah, I also, I think you could have been. I was using my headphones, and I switched.

12 00:03:19.700 00:03:23.050 Amber Lin: Oh, I see.

13 00:03:24.845 00:03:32.319 Amber Lin: You know, I was trying to do the roadmap, and I was so lost because I didn’t really understand what wanted.

14 00:03:32.490 00:03:43.490 Amber Lin: and how that related to the how that match the project. So you know, we had a project proposal a long time ago, and I was kind of basing it off of that. Say, Oh, these are the different goals!

15 00:03:43.820 00:03:49.610 Amber Lin: So that was like my mindset before, and then I think he wanted to have like.

16 00:03:50.140 00:03:57.239 Amber Lin: instead of like business questions to structure in terms of, like the data,

17 00:03:59.040 00:04:07.380 Amber Lin: flow and the different tools. So I finally wrapped my head around it. And I kind of just wanted to

18 00:04:07.980 00:04:13.360 Amber Lin: get us as a do you think this is the right way to do it right?

19 00:04:15.340 00:04:16.860 Amber Lin: These are accurate.

20 00:04:18.350 00:04:26.249 Demilade Agboola: I mean, I I have. I I went through it. Bit this morning. I feel like, yeah, it’s definitely on track. I

21 00:04:27.030 00:04:28.880 Demilade Agboola: ultimately feel like

22 00:04:32.760 00:04:38.240 Demilade Agboola: like the phase. 0 is also very important. Like being able to have those conversations with stakeholders and being able to

23 00:04:39.009 00:04:45.459 Demilade Agboola: align on what we need and what that looks like. I feel like we will need to build out

24 00:04:45.740 00:04:50.099 Demilade Agboola: a lot of new models, new infrastructure. We’ll need to work on the

25 00:04:50.300 00:05:07.489 Demilade Agboola: incrementality of the models which you have put there. We also need to work on like the their model runs like the what’s it? How do I explain? Model runs like the time for each like time for each Dbt job. It takes a lot for their jobs, which is kind of what you put there as well.

26 00:05:07.820 00:05:10.128 Amber Lin: Where did I put that?

27 00:05:10.850 00:05:16.739 Demilade Agboola: Do you kind of put it like I think, I added a comment somewhere. So you should kind of see. Get somewhere around that point like.

28 00:05:17.530 00:05:18.940 Amber Lin: Oh! Is it here?

29 00:05:18.940 00:05:22.230 Demilade Agboola: Yeah, so like, so then we move this to.

30 00:05:22.380 00:05:23.560 Amber Lin: Dbt.

31 00:05:24.220 00:05:25.790 Demilade Agboola: Yes, there’s more dbt than.

32 00:05:25.790 00:05:26.170 Amber Lin: Okay.

33 00:05:45.450 00:05:49.100 Amber Lin: Like? Is this also a dbt thing.

34 00:05:49.750 00:05:50.450 Demilade Agboola: Yes.

35 00:05:51.300 00:05:57.140 Amber Lin: Okay, great. Yeah.

36 00:06:05.440 00:06:08.700 Amber Lin: Is this for number 2, or like, here.

37 00:06:09.710 00:06:11.460 Amber Lin: I feel like it was, for here.

38 00:06:11.988 00:06:17.230 Demilade Agboola: Incremental model. I mean, you specifically, said March. So before March, but, like just, generally speaking, like

39 00:06:17.840 00:06:22.620 Demilade Agboola: for everything like we need to speed up the processes for everything just across board.

40 00:06:22.840 00:06:33.850 Amber Lin: Oh, should I leave it there like if you were if you were to write this roadmap like, what would you do? Honestly, I’m just a typewriter for your head.

41 00:06:38.250 00:06:42.949 Demilade Agboola: I think just generally I will see Tristan.

42 00:06:44.720 00:06:49.539 Demilade Agboola: Sorry I was looking at some difficulty jobs. I think what I would say is.

43 00:06:49.710 00:06:52.100 Demilade Agboola: we will need to be able to.

44 00:06:52.770 00:07:00.169 Demilade Agboola: One audits the entire Dbt infrastructure.

45 00:07:00.410 00:07:04.491 Demilade Agboola: get rid of dominance models they love like redundant, like

46 00:07:04.900 00:07:05.510 Amber Lin: Oh!

47 00:07:05.510 00:07:06.660 Demilade Agboola: Necessary models.

48 00:07:07.110 00:07:07.970 Amber Lin: So.

49 00:07:07.970 00:07:09.370 Demilade Agboola: Let’s reduce the Bloatware.

50 00:07:12.140 00:07:18.719 Amber Lin: Audit all Dbt. Models to discover redundancies.

51 00:07:19.320 00:07:22.999 Demilade Agboola: Yes, and get, and so we’ll streamline it. Get rid of all bloatware.

52 00:07:25.320 00:07:26.830 Amber Lin: Sorry. What was that?

53 00:07:27.130 00:07:34.940 Demilade Agboola: I said. Then we will streamline and get rid of all bloats, where, like all blue heads, like models.

54 00:07:35.050 00:07:36.990 Amber Lin: That we don’t need to get rid of them.

55 00:07:42.530 00:07:43.100 Amber Lin: Hmm!

56 00:07:44.290 00:07:49.019 Demilade Agboola: Then, once we free, once we have what we need like, once we know. Okay, so these models are necessary.

57 00:07:49.170 00:07:56.249 Demilade Agboola: We will then at that point need to restructure it. So their models are chunky.

58 00:07:57.600 00:08:03.499 Demilade Agboola: They have models with 600 lines, which is ideally know how you write model models in Dbt.

59 00:08:03.600 00:08:07.109 Demilade Agboola: they have models that are doing too much, basically

60 00:08:07.845 00:08:17.170 Demilade Agboola: at that point we need to restructure it, break it down into bite size pieces. So that’s called modularity. That’s the concept called modularity in software engineering.

61 00:08:18.173 00:08:19.640 Amber Lin: And then.

62 00:08:19.990 00:08:25.019 Demilade Agboola: We will need to then come up with better strategies for how their Dbt jobs run.

63 00:08:26.390 00:08:38.180 Demilade Agboola: So how do we ensure that our model runs. Don’t take 60 min, 40 min, 45 min. How can we get this number down to as low as low a number as possible?

64 00:08:41.020 00:08:42.530 Demilade Agboola: And then

65 00:08:42.750 00:08:52.790 Demilade Agboola: we will kind of do the same thing. But for like dashboards, we will need to audit their dashboards. What dashboards are being used. What decisions are being made with this dashboards.

66 00:08:53.540 00:09:04.790 Demilade Agboola: and what dashboards are you currently not maintaining, but still exist like, what exactly is the point with this, so like we again we cut down stream the the bloats where where necessary.

67 00:09:05.110 00:09:05.690 Demilade Agboola: and.

68 00:09:05.690 00:09:06.410 Amber Lin: Hmm.

69 00:09:07.040 00:09:09.339 Demilade Agboola: At that point. We can also

70 00:09:10.810 00:09:29.330 Demilade Agboola: just redesign whatever dashboards needs to be designed or restructure, whatever dashboards needs to be structured, so that maybe in some cases we realize that we have 2 different dashboards doing similar things. How do we merge them into one? Things like that like, how do we restructure the dashboard so that, like

71 00:09:30.355 00:09:34.570 Demilade Agboola: it’s as useful as possible.

72 00:09:34.780 00:09:40.329 Amber Lin: Oh, build them from from scratch

73 00:09:42.090 00:09:52.259 Amber Lin: like. Why, why do you think there’s a reason that we will kind of edit what they have and not built it from the ground up. Kind of like what you’re doing with the models.

74 00:09:54.650 00:10:08.270 Demilade Agboola: I mean we could. I think it depends on the analyst, on the team and what their vision is for that. So I would say I would say restructure, because I mean there is some utility to it, because I mean, they’re currently using some of them.

75 00:10:08.930 00:10:10.680 Demilade Agboola: The question is

76 00:10:10.780 00:10:15.949 Demilade Agboola: so why, I said, redesign is, for instance, if they need it for certain decisions, but like

77 00:10:16.100 00:10:31.259 Demilade Agboola: they’re only getting the tip of the iceberg, and we potentially add more information in there like, that’s what I meant by redesign in terms of restructuring. Obviously, maybe moving things around or obviously just figuring out like, Hey, we can merge multiple things together and get rid of one dashboard

78 00:10:31.360 00:10:32.740 Demilade Agboola: or through that.

79 00:10:33.360 00:10:34.430 Demilade Agboola: So like

80 00:10:35.320 00:10:44.779 Demilade Agboola: I don’t know if it’s necessarily same thing as like starting from afresh. And just like building again. But that’s also potentially something we we would need to look into.

81 00:10:45.605 00:10:48.080 Demilade Agboola: Like again. This is something

82 00:10:48.590 00:10:53.089 Demilade Agboola: that it depends on the use case like if Looker is working for them

83 00:10:53.250 00:10:59.430 Demilade Agboola: like, look as they what they use for their bi. So if look is working for them, and you are still keeping things in looker.

84 00:10:59.910 00:11:04.720 Demilade Agboola: In some cases I build. Some things are fresh. In some other cases we might, you know, just

85 00:11:05.060 00:11:08.810 Demilade Agboola: modify what exactly is exist on ground.

86 00:11:09.220 00:11:17.190 Demilade Agboola: Now, if look at this for them. And we’re like, okay for your use case in terms of maybe budget. And what? Who needs to see and data governance and data access.

87 00:11:17.350 00:11:25.929 Demilade Agboola: we might decide to switch to a new tool. In that case, we definitely need to, you know, obviously rebuild. So again, that’s part of what the audit would contain.

88 00:11:35.740 00:11:53.390 Demilade Agboola: Also, we will need to do stuff on their documentation. Just so that, like people, it’s clear what’s what is happening. Because sometimes, again, Emily, when not talking to Emily. Emily isn’t exactly, always clear about what’s going on, and it’s not her fault. It’s just like things are not like

89 00:11:54.110 00:11:55.710 Demilade Agboola: properly specified.

90 00:11:56.970 00:11:57.510 Demilade Agboola: Oh.

91 00:12:01.510 00:12:08.719 Amber Lin: Cool and to confirm like is, it’s everything here with the looker stuff, correct.

92 00:12:12.466 00:12:16.049 Demilade Agboola: Yes, logic space. Yes.

93 00:12:16.380 00:12:27.090 Demilade Agboola: no, defined. Well, I think there is a folder structure to be honest, but I don’t think it’s necessary. I mean it exists. I’m sure it’s the most intuitive or the best, the best designed

94 00:12:27.220 00:12:28.780 Demilade Agboola: for the folder structure.

95 00:12:32.440 00:12:34.310 Demilade Agboola: Duplicate dashboards.

96 00:12:36.190 00:12:46.299 Demilade Agboola: I’m not sure, probably, but I haven’t seen any like the duplicate dashboard there probably are, but that there’s so many dashboards, and there’s so much going on. I haven’t actually sat down to like audits that.

97 00:12:46.760 00:12:52.899 Amber Lin: Hmm, that’s okay. I think we can just assume some exist cause they talk about the same thing.

98 00:12:54.850 00:12:55.440 Amber Lin: Oh.

99 00:13:01.170 00:13:03.749 Demilade Agboola: Oh, yeah. So some business logic in

100 00:13:05.100 00:13:10.740 Demilade Agboola: so kind of yeah, I was, gonna say, the logic space in look and is kind of the same thing as business logic in look. Ml.

101 00:13:11.970 00:13:14.610 Amber Lin: Hmm! I see.

102 00:13:14.810 00:13:18.310 Demilade Agboola: Because, like looking up just the language like you’re writing.

103 00:13:18.890 00:13:23.729 Amber Lin: Oh, okay, let me just combine these.

104 00:13:28.380 00:13:35.350 Amber Lin: Row for think this is like, govern, this

105 00:13:48.690 00:13:50.570 Amber Lin: is this

106 00:13:53.770 00:14:01.310 Amber Lin: manual work, is this, do you think that covers everything.

107 00:14:09.880 00:14:15.971 Demilade Agboola: with those sets. Yeah, I think it’s largely comprehensive. Covers a lot.

108 00:14:17.080 00:14:18.680 Amber Lin: Anything. We’re missing.

109 00:14:20.240 00:14:22.850 Demilade Agboola: Nothing obvious that we’re missing.

110 00:14:23.668 00:14:29.259 Demilade Agboola: I said. There’s nothing obvious that we are missing is my my! Can you hear me properly?

111 00:14:29.910 00:14:37.379 Amber Lin: No, yeah, I think cause my Internet broke down. So I’m on my phone data.

112 00:14:37.970 00:14:38.390 Demilade Agboola: Oh!

113 00:14:38.390 00:14:41.250 Amber Lin: Spot. So that’s why it’s it’s me.

114 00:14:43.170 00:14:46.040 Demilade Agboola: I just wanted to be sure, if, like, I need to change myself up.

115 00:14:46.930 00:14:48.610 Amber Lin: No, don’t worry. You’re good.

116 00:14:51.670 00:14:52.310 Amber Lin: Hmm.

117 00:14:52.560 00:14:57.089 Demilade Agboola: Yeah. So I think there’s nothing obvious that we’re missing right now, I think

118 00:14:57.190 00:15:02.370 Demilade Agboola: to be fair. I think the dashboards I haven’t really like dived into the dashboards.

119 00:15:02.980 00:15:11.189 Amber Lin: Okay, that’s okay, like, we’ll just say we need to dive in like that’s, that’s also a good step. What about this part?

120 00:15:11.520 00:15:17.279 Amber Lin: Like we’re using redshift for data. Warehousing.

121 00:15:17.920 00:15:19.050 Amber Lin: Yes, resume too.

122 00:15:20.130 00:15:21.289 Demilade Agboola: Something we need to do

123 00:15:21.400 00:15:26.260 Demilade Agboola: we need to figure out is there like access and grants in the sense that, like

124 00:15:27.140 00:15:29.849 Demilade Agboola: one of the issues we’ve been having and why they’re

125 00:15:30.820 00:15:37.119 Demilade Agboola: jobs have been failing sometimes, like their Dvt jobs have been failing is that you need to like?

126 00:15:37.270 00:15:47.829 Demilade Agboola: I’ll try to. I’ll explain this like, how like you work. So let’s say, we ingest data right? Save through polytomic or state. So this is me explaining. And I’ll kind of go into the issue right now.

127 00:15:47.980 00:15:48.860 Amber Lin: Yeah.

128 00:15:49.380 00:16:00.029 Demilade Agboola: We ingest it. Every polytomic or state, or heave, or whatever it creates, a new table, right? And when it’s new table is created, you need to grant access to who can read the data.

129 00:16:00.480 00:16:04.720 Demilade Agboola: And if access is not granted to the right person or user.

130 00:16:05.410 00:16:07.470 Demilade Agboola: they can’t read the data right.

131 00:16:07.770 00:16:10.769 Demilade Agboola: The problem they were having with their Dvt jobs was that

132 00:16:11.445 00:16:14.840 Demilade Agboola: sometimes, at least with polyatomic data.

133 00:16:15.010 00:16:18.879 Demilade Agboola: Specifically, it would create the table.

134 00:16:19.620 00:16:24.380 Demilade Agboola: But it wouldn’t give access to Dbt, right? So you.

135 00:16:24.830 00:16:30.769 Demilade Agboola: the dB, 2 jobs were failing because of that, because dB, 2, trying to reach on the table can’t access it, and the job will fail.

136 00:16:32.250 00:16:35.570 Demilade Agboola: So now we did a quick fix, which was for the production data.

137 00:16:35.800 00:16:36.970 Demilade Agboola: We

138 00:16:37.502 00:16:50.800 Demilade Agboola: granted Dbt production, something called a super user. So a super user doesn’t need like a super user can access any table basically doesn’t need it doesn’t need specifically granted access to it. However, if that’s a necessary best practice.

139 00:16:50.900 00:16:55.569 Demilade Agboola: So going forward, what we’re doing is for this Dvt staging and Dvt prod users.

140 00:16:56.820 00:16:58.580 Demilade Agboola: We want to create

141 00:16:58.690 00:17:10.949 Demilade Agboola: a flow that every time polytomic, or whatever ingestion tool creates a table at the end of their create table creation. It would run and access grant on DVD. Staging and DVD prod

142 00:17:11.380 00:17:12.599 Demilade Agboola: that we

143 00:17:13.272 00:17:19.050 Demilade Agboola: they would always be able to read the latest data and the freshest data whenever they are doing their runs or tests.

144 00:17:21.880 00:17:25.260 Demilade Agboola: So that’s so, we’re just going to figure out access just generally for

145 00:17:25.736 00:17:36.780 Demilade Agboola: whatever users need to the access. Even Emily has access issues. By the way, this is also this also affects Emily. At times Emily cannot run certain models or certain things because she doesn’t have access.

146 00:17:39.850 00:17:48.430 Demilade Agboola: So we’re setting up the access. And Grant specifically for the redshift would like, just so that that flow is incorporated automatically is, it will be important.

147 00:17:54.450 00:18:04.249 Demilade Agboola: yeah, also, like ingestion, like just kind of setting on ingestion tool, which is the fragmentation portion that you mentioned. So what’s that? One ingestion tool that we will use?

148 00:18:10.140 00:18:18.679 Demilade Agboola: and then we do have schema suppression for Dev and Produness. I’m not sure I’m not sure what that means. Like. Let me let me.

149 00:18:18.680 00:18:23.280 Amber Lin: I don’t know, either. Don’t. I don’t know, either. That’s why I’m asking you.

150 00:18:23.840 00:18:24.500 Demilade Agboola: Oh, cool.

151 00:18:24.960 00:18:30.106 Demilade Agboola: Yes, we have regional portal. We do have their own product thing that that’s fine

152 00:18:30.710 00:18:36.349 Demilade Agboola: concurrent issues in retrofit. That’s just true, we do have models that run at the same time. So we need to like optimize that.

153 00:18:39.320 00:18:50.019 Demilade Agboola: no, clear. Let’s in observability. I mean, I I think there is now. But we we need to. We can improve it. I think that’s we’ve set up some processes, but I think we can improve it.

154 00:18:50.970 00:18:54.130 Demilade Agboola: So that’s why letting an observer, basically, we set up

155 00:18:54.340 00:18:57.000 Demilade Agboola: alerting. And but we can definitely improve

156 00:18:57.190 00:18:59.839 Demilade Agboola: our Etl fields and model freshness.

157 00:19:00.710 00:19:01.690 Amber Lin: Oh.

158 00:19:05.000 00:19:13.040 Amber Lin: is this true, too, like the 1st one, this one large joins and snapshot.

159 00:19:13.530 00:19:17.499 Demilade Agboola: Yeah, I think this is this kind of ties to like the restructuring of

160 00:19:17.863 00:19:23.430 Demilade Agboola: so that perspective, I’m not sure. Let me look at the Runtime. I’m not sure if it’s necessarily the longest for him.

161 00:19:25.020 00:19:32.669 Demilade Agboola: Models, but we do generally have issues with certain models, taking a long time to run

162 00:19:33.460 00:19:34.720 Demilade Agboola: one second.

163 00:19:39.390 00:19:43.389 Demilade Agboola: Where’s this page like the I’m trying to copy the name of the model.

164 00:19:43.920 00:19:48.426 Amber Lin: Yeah, I shared it with you in

165 00:19:49.260 00:19:53.729 Amber Lin: yeah, in our chat. But let me also.

166 00:20:01.750 00:20:06.959 Amber Lin: I’ll send it to you again. And they also sent in urban stems, channel.

167 00:20:09.510 00:20:12.409 Demilade Agboola: Let me see what’s adding comments right now, as we speak.

168 00:20:14.025 00:20:19.560 Amber Lin: Sure, yeah, just whatever anything will be really helpful. Because this is

169 00:20:19.910 00:20:26.880 Amber Lin: this is, as far as I go, technically the help of Gpt. But

170 00:20:27.660 00:20:30.820 Amber Lin: this is where you’d be super helpful.

171 00:20:31.690 00:20:32.849 Demilade Agboola: That’s all. Good.

172 00:20:34.820 00:20:36.260 Amber Lin: Is also commenting.

173 00:20:36.420 00:20:37.670 Demilade Agboola: Yeah, that’s awesome.

174 00:20:41.990 00:20:42.890 Demilade Agboola: Oh.

175 00:21:07.930 00:21:11.180 Demilade Agboola: actually, no. This, the particular model you mentioned doesn’t take too much time

176 00:21:11.500 00:21:13.270 Demilade Agboola: takes about a minute to run.

177 00:21:13.270 00:21:14.110 Amber Lin: Oh, good!

178 00:21:15.710 00:21:17.900 Demilade Agboola: About half a minute to be honest.

179 00:21:18.371 00:21:19.730 Demilade Agboola: So it’s not a struggle.

180 00:21:19.730 00:21:25.640 Amber Lin: Okay. So it’s the redshift stuff. It’s okay. Like doesn’t take too long.

181 00:21:26.490 00:21:36.079 Demilade Agboola: Yeah, so we don’t necessarily have like performance issues with that model. In particular, there are certain models. We do have issues with which we will need to optimize, but not.

182 00:21:36.080 00:21:36.750 Amber Lin: So.

183 00:21:36.980 00:21:37.850 Demilade Agboola: Also not.

184 00:21:38.990 00:21:40.270 Demilade Agboola: Yes, the example.

185 00:21:40.270 00:21:44.470 Amber Lin: Do audit runtime for all models.

186 00:21:45.270 00:21:46.299 Amber Lin: How’s that.

187 00:21:47.670 00:21:48.480 Demilade Agboola: Yeah.

188 00:21:48.480 00:21:58.140 Amber Lin: Some of them are, I think, the ones you did are probably already pretty good. How is this model different from the Dbt model question

189 00:21:58.930 00:22:02.010 Amber Lin: like, there’s so many things called models.

190 00:22:03.870 00:22:06.410 Demilade Agboola: What do you mean? Which models are different from what models.

191 00:22:06.967 00:22:15.799 Amber Lin: Cause, you know, we’re we’re talking about it here, right? We’re talking about performance issues with large drawings and stuff. We’re talking about this right you’re talking about, hey? This one has law

192 00:22:16.350 00:22:18.380 Amber Lin: like long runtime.

193 00:22:18.780 00:22:19.100 Demilade Agboola: Yeah.

194 00:22:19.100 00:22:24.659 Amber Lin: Are we talking still talking about a Dbt model? If so, should I move it up here.

195 00:22:27.550 00:22:34.090 Demilade Agboola: Yeah, it’s still the Dbt models. So the Dbt models end up being like, built into redshift.

196 00:22:34.210 00:22:39.160 Demilade Agboola: So yeah, but like, yeah, we can move. We can make it. You can move it up there.

197 00:22:40.720 00:22:42.350 Amber Lin: Okay.

198 00:22:42.950 00:22:52.890 Amber Lin: But the does. The joins happen in redshift or like in after during the Etl Etl happens before redshift right.

199 00:22:53.930 00:23:00.230 Demilade Agboola: Yes. Well, technically it it depends. So Etl stands for like extract, transform and load right?

200 00:23:00.230 00:23:04.150 Amber Lin: So kind of like within or before.

201 00:23:05.080 00:23:14.400 Demilade Agboola: Yes. So extract extract, you’re extracting the data into the warehouse. That’s what the E, until the transforming actually extracted from different sources.

202 00:23:14.560 00:23:18.640 Demilade Agboola: And then you are transforming data within the warehouse.

203 00:23:18.840 00:23:26.850 Demilade Agboola: and then you’re loading into your like. Your what’s it called your dashboards and stuff.

204 00:23:27.960 00:23:36.379 Demilade Agboola: So yeah, the redshift is like the underlying part of all this entire process. Even what Dbt does Dbt is literally just compiling SQL.

205 00:23:36.520 00:23:46.240 Demilade Agboola: And then ask, telling Redshift to run it. So DVD is not really like track like he’s not really doing any processing. Redshift does all the processing redshift is the wall cost?

206 00:23:46.470 00:23:49.970 Demilade Agboola: Dbt just kind of gets the glory. I don’t want to explain beyond that, but, like.

207 00:23:50.290 00:23:59.300 Amber Lin: I see, because we were on Dvt so many times. It just so seems so important. But everything happens.

208 00:23:59.300 00:24:21.440 Demilade Agboola: Exactly just going like, oh, this! This is what I need you to run. This is what I need you to run like, compile. It compiles the SQL, so run this logic, run this logic, start as a table, start it as a view, start it as an incremental model. But it’s not really actually doing anything in the sense of like building the actual tables. It’s

209 00:24:21.750 00:24:25.500 Demilade Agboola: the redshift instance is, what does all the computing

210 00:24:25.690 00:24:27.980 Demilade Agboola: do you need to kind of just kind of.

211 00:24:28.455 00:24:33.210 Amber Lin: Instructs redshift. So this, this is like a manager ish

212 00:24:33.440 00:24:36.639 Amber Lin: to person that does to something that does the work.

213 00:24:37.190 00:24:38.259 Demilade Agboola: Sort of yes.

214 00:24:38.590 00:24:40.369 Amber Lin: Okay. I see.

215 00:24:40.970 00:24:41.980 Amber Lin: I see.

216 00:24:42.395 00:24:50.900 Demilade Agboola: Let’s just use this as a very high level understanding. But yeah, dbt, just comp, okay.

217 00:24:52.005 00:24:54.600 Amber Lin: That’s helpful. So.

218 00:24:55.090 00:24:57.690 Demilade Agboola: So that’s why some things can overlap right like.

219 00:24:57.690 00:24:59.800 Amber Lin: Yeah, yeah, that makes sense. Now.

220 00:25:00.600 00:25:07.796 Amber Lin: Concurrency. Yeah, I’ll just. I’ll just keep it there. All that old time bigger than

221 00:25:08.660 00:25:19.989 Amber Lin: Is this the right fix plan, or like needs fixes? Are these, are these accurate.

222 00:25:22.366 00:25:25.300 Demilade Agboola: Set of schema 96, product 96. Dev.

223 00:25:26.414 00:25:31.849 Demilade Agboola: Okay, yeah. Those are accurate. This is this thing. Yes.

224 00:25:33.000 00:25:41.299 Demilade Agboola: and then cost tracking, I mean, yeah, we could also add cost tracking as well. Yeah. So I think, yeah, that’s fine, like, not I. I wouldn’t necessarily change anything. There.

225 00:25:42.730 00:25:43.400 Amber Lin: Hmm

226 00:25:48.140 00:25:49.230 Amber Lin: here.

227 00:25:51.780 00:25:53.549 Amber Lin: Where is it?

228 00:26:02.970 00:26:03.970 Amber Lin: Okay.

229 00:26:07.380 00:26:10.480 Amber Lin: why is this? Oh, this isn’t.

230 00:26:11.980 00:26:15.179 Amber Lin: This is the same right like increment.

231 00:26:16.050 00:26:23.090 Amber Lin: is it? Does it? Do you think it should be here like what is Dag.

232 00:26:25.230 00:26:29.020 Demilade Agboola: Oh, so that’s like, directed Acyclic graphs.

233 00:26:29.475 00:26:37.609 Demilade Agboola: So basically, when you build models in Dvt, you start from like, you know, raw staging intermediate mods, right? So that creates like a flow.

234 00:26:38.530 00:26:58.320 Demilade Agboola: and it directs like it ends up, creating like a flow so Dbt knows I need to tackle this model first, st and this model second, and this model 3, rd this model 4, th and then I’ll have my final model, which is the 5th model, which I run something like that like. So it creates a dag. So that’s that’s what’s called a dag like how it flows, how the how Dbt works on the different models, and how it.

235 00:26:58.880 00:27:03.570 Demilade Agboola: if it directs how the models like, how the model should be built out.

236 00:27:04.130 00:27:07.849 Demilade Agboola: So obviously, if something is, you know, slow.

237 00:27:08.490 00:27:12.190 Demilade Agboola: it slows down the overall graph like the overall directed. Exactly graph.

238 00:27:16.310 00:27:22.179 Demilade Agboola: So let’s say we have this. And what would be the impact of lacking like.

239 00:27:22.520 00:27:28.519 Amber Lin: These lacking like incrementally in lacking staging.

240 00:27:28.910 00:27:31.249 Amber Lin: And so what is impact for them.

241 00:27:33.860 00:27:42.790 Demilade Agboola: Impact is if models our low lift like if the models are slow or have low latency.

242 00:27:43.611 00:27:46.070 Demilade Agboola: eventually, what’s gonna happen is.

243 00:27:46.860 00:27:53.250 Demilade Agboola: oh, sorry high latency. I didn’t know latency. If he has high latency. What ends up happening is that

244 00:27:53.990 00:27:55.660 Demilade Agboola: the runs will take longer

245 00:27:58.640 00:28:06.900 Demilade Agboola: and you’re using more resources than you necessarily have to, which is kind of where we. So let me give an example of what what we mean.

246 00:28:07.480 00:28:10.159 Demilade Agboola: If you’re building a table in Dbt.

247 00:28:10.490 00:28:19.500 Demilade Agboola: It literally rebuilds table from scratch every single time dbt run so like it will take the old table, toss it out, and then replace it with a new table.

248 00:28:20.080 00:28:31.039 Demilade Agboola: However, the problem is, there’s some things that are fairly static, like you have transactions. You’ve that have done that you’ve done in 2023, 2024. Those don’t change, you’re not. You’re not going to change that.

249 00:28:31.340 00:28:31.660 Amber Lin: Hmm.

250 00:28:31.860 00:28:44.160 Demilade Agboola: And if you have a lot of transactional orders, you’re literally saying, like, 75% of our table is static. But like we’re tossing out the entire table every single time I’m building out the entire 100, which accounts for maybe

251 00:28:44.830 00:28:55.169 Demilade Agboola: just a little change over the last, like maybe 14 days. Maybe you you still do some things retroactively, but like over the last 14 days, the last one month. That’s the only time that things can actively change.

252 00:28:56.850 00:29:04.549 Demilade Agboola: so incrementality just says, Hey, keep this past this part static and only build the latest updates into this new table.

253 00:29:04.790 00:29:14.079 Demilade Agboola: That’s what makes it faster. That’s kind of how you save the runtime, because you’re not discarding the entire table and rebuilding that fresh every single time. So that’s what a full refresh is so like

254 00:29:15.550 00:29:26.580 Demilade Agboola: incrementality under use means. Models are refreshed by default, so every single time a model is built it would refresh it. It will build it afresh every single time, and sometimes like again. The changes are

255 00:29:26.700 00:29:37.019 Demilade Agboola: only things that happened over the last 2 days last one day, you know, last couple of hours. You don’t need to rebuild the entire table every single time, which is why we say, like incrementality is under use.

256 00:29:37.860 00:29:40.229 Amber Lin: Oh, awesome. Okay.

257 00:29:43.416 00:29:48.250 Amber Lin: Makes sense. Makes sense. Walker.

258 00:29:51.090 00:29:58.910 Amber Lin: Yeah. Actually, let’s look at this. Is this accurate?

259 00:29:59.230 00:30:00.909 Amber Lin: Oh, my bad.

260 00:30:03.950 00:30:05.889 Demilade Agboola: So what is what accurate?

261 00:30:05.890 00:30:14.470 Amber Lin: This part under dbt, the part that kind of needs fixes, I’ll I’ll consolidate these but.

262 00:30:21.630 00:30:25.259 Demilade Agboola: Needs fixes refer to all function functional areas

263 00:30:25.370 00:30:29.580 Demilade Agboola: is a model or structure that is accurate. And Dbt tests.

264 00:30:30.090 00:30:35.679 Demilade Agboola: So add Dv to the test. So like after you put unique, not know accepted values puts.

265 00:30:35.940 00:30:45.919 Demilade Agboola: and also like. So after that bracket, like after the bracket for unique, not know accepted values. The second point activity test. So after the bracket, the idea

266 00:30:46.840 00:30:54.050 Demilade Agboola: add, and no no just put, they just add this and generic and custom test as well.

267 00:30:56.100 00:30:57.719 Demilade Agboola: So the like.

268 00:30:58.470 00:31:06.110 Demilade Agboola: So like unique, not know accepted values at the generic test. They come directly with Dvt, but we’re also trying to also create custom tests as well.

269 00:31:08.660 00:31:09.530 Amber Lin: I see.

270 00:31:13.640 00:31:19.889 Demilade Agboola: So, for instance, this, just like Generic, says, Oh, unique! This column must be unique. Not no, but we can also add custom tests like, oh.

271 00:31:20.130 00:31:29.489 Demilade Agboola: revenue needed should never be less than 100,000, for instance. So like, if it’s less than 100,000 we look into like. So what will happen with our data, not loading things like that, like you can add custom tests and

272 00:31:29.830 00:31:31.619 Demilade Agboola: way. More flexibility to that.

273 00:31:31.770 00:31:38.740 Demilade Agboola: Introduce tags by business domain. Yes, and we build revenue models with clear date logic, or the Diversity delivery date.

274 00:31:39.240 00:31:44.789 Demilade Agboola: I’m not sure if they necessarily, I think they do that. But again, it’s kind of messy how they handle that, to be honest. So

275 00:31:45.740 00:31:49.950 Demilade Agboola: I mean the work layer helps. That’s that’s fine. That’s fine, hey? Delight.

276 00:31:49.950 00:31:50.530 Amber Lin: Okay.

277 00:31:50.780 00:31:53.620 Demilade Agboola: Great exposures for local dashboards. Yes.

278 00:31:54.700 00:31:58.669 Demilade Agboola: rental models, also changing tables. Yes. Okay, I think that. Yeah, that’s fine.

279 00:32:00.350 00:32:04.619 Amber Lin: Sounds good. What’s the snapshot actually.

280 00:32:05.610 00:32:08.049 Demilade Agboola: So if snapshots.

281 00:32:08.350 00:32:16.859 Demilade Agboola: so I don’t necessarily think they overlapping. To be honest. But I will like I will look at like I know that they have multiple snapshots for like inventory. I don’t know what they are. I’m not really like

282 00:32:17.000 00:32:19.469 Demilade Agboola: gone into that to like investigate.

283 00:32:19.810 00:32:20.140 Amber Lin: Hmm.

284 00:32:20.350 00:32:20.980 Demilade Agboola: It’s not the.

285 00:32:20.980 00:32:24.349 Amber Lin: Snapshot mean? Yeah. Sorry you were talking about that.

286 00:32:24.490 00:32:26.495 Demilade Agboola: Yeah. So a snapshot basically means

287 00:32:28.220 00:32:40.879 Demilade Agboola: So some tables are updated. Live. So what I mean by that is, if I had a table that had orders. And that order. That row said, Okay, order. ABC was sent to amber

288 00:32:42.034 00:32:44.630 Demilade Agboola: and the start, what’s the what’s the status?

289 00:32:44.840 00:32:47.270 Demilade Agboola: The status is paid for.

290 00:32:47.620 00:32:58.120 Demilade Agboola: and then maybe the next status is shipped. So like, you know, after you’ve paid for it, the next status would be. It has been shipped. Then the next status it has been delivered right? So like there are 3 different statuses that could occur.

291 00:32:58.440 00:33:17.379 Demilade Agboola: Some people’s tables do not automatically create multiple records of like, Oh, status ABC has been shipped to amber. Has been paid for by amber, then shipped, and then delivered to those 3 different rows, some people’s tables. What what they do instead is, it will show the 1st record as paid for.

292 00:33:17.870 00:33:22.839 Demilade Agboola: and then when you, when it has been shipped it. It goes back to that same row.

293 00:33:23.670 00:33:28.530 Demilade Agboola: removes the the record, paid for, and replaces it with. Now the record shipped.

294 00:33:29.560 00:33:36.060 Demilade Agboola: and then, when this is delivered it goes back to that same row and then removes the record

295 00:33:36.494 00:33:49.740 Demilade Agboola: shipped, and now replaces we delivered. So at all points of time we only have one row. That shows what’s happening. But then there are 3 different statuses. So what the snapshot does is that every time Dbt runs it will look at that table

296 00:33:49.850 00:34:01.770 Demilade Agboola: and take like a picture of that table. And every single time that there’s a change to like a unique role like it goes for that record, ABC. Or before it was paid for. Now it has changed to shipped.

297 00:34:01.920 00:34:14.729 Demilade Agboola: it goes values no longer paid for. The value is now shipped. So now I can, and then, when it changes from shipped to delivered, it takes another snapshot. So now I can have the 3 rows, the 3 different statuses that have changed.

298 00:34:15.199 00:34:33.240 Demilade Agboola: I can look at them, and I can say, Oh, this was only valid. It was in the paid for state for 2 h or 3 h. It was in the shift States for like 5 h, and it wasn’t delivered States from like that date till now. Do you get my point? So like. Now we can start to have an idea of how long it takes for

299 00:34:33.679 00:34:36.689 Demilade Agboola: from payments to shipping, from shipping to.

300 00:34:37.635 00:34:38.580 Amber Lin: Okay.

301 00:34:39.020 00:34:39.850 Demilade Agboola: But before, if we.

302 00:34:39.850 00:34:41.059 Amber Lin: Oh, my goodness!

303 00:34:41.060 00:34:41.690 Demilade Agboola: It, Bruce.

304 00:34:41.690 00:34:43.789 Amber Lin: So, clear. Okay.

305 00:34:43.790 00:34:45.609 Demilade Agboola: Yeah. So if you keep deleting the rules, we don’t.

306 00:34:45.610 00:34:46.090 Demilade Agboola: I see

307 00:34:46.090 00:34:50.250 Demilade Agboola: happening, we just know what the current state is, but we don’t know what the previous state is.

308 00:34:50.409 00:34:54.289 Demilade Agboola: So being able to keep track is way snapshot does. It? Keeps like taking like

309 00:34:54.750 00:34:57.799 Demilade Agboola: what’s happening. So we can see the the changes over time.

310 00:34:59.160 00:35:06.019 Amber Lin: I see, and for them like I don’t know if that’s true, then if we if that’s the case.

311 00:35:07.560 00:35:21.840 Demilade Agboola: Yeah. So I mean they they have snapshot tables, but I don’t like I’m not again. I’m not really gone in to see like what the logic is. What are you taking snapshots of? Which is also part of? Why, like, I also like, you know, we need to co- create new systems for them.

312 00:35:23.270 00:35:38.500 Demilade Agboola: Naming convention. Oh, yeah, we should add that there, by the way, nomenclature like naming, should be clear like sometimes you don’t need to open a model to know which is my problem with the snapshots. You don’t need to open a model to know what

313 00:35:38.620 00:35:41.179 Demilade Agboola: the model is doing or what

314 00:35:41.660 00:35:59.019 Demilade Agboola: the the snapshot is keeping track of, or what you know like it should be clear like you should be able to read the model name and have an idea of what’s going on there. You might not. It might not be like fully explicit, but you should have an idea. I read their model names, and I have no fucking clue. What’s going on?

315 00:36:00.860 00:36:02.870 Demilade Agboola: That’s bad. That’s bad naming.

316 00:36:04.960 00:36:05.860 Amber Lin: Cool.

317 00:36:06.651 00:36:13.549 Amber Lin: I know. Like add, where did he add stuff? I know, he commented. Some new stuff.

318 00:36:14.910 00:36:16.500 Amber Lin: Me load this.

319 00:36:20.153 00:36:24.149 Amber Lin: Where is his comments.

320 00:36:24.910 00:36:31.589 Amber Lin: Yeah, I think I’ll just go. And he added a few. My phone got the notification. Oh, okay.

321 00:36:32.060 00:36:38.750 Amber Lin: sorry, not by June 15th he essentially.

322 00:36:39.580 00:36:42.789 Amber Lin: Can you see his comments? Oh, here we go!

323 00:36:43.070 00:36:49.400 Demilade Agboola: Yeah, I mean, I can’t. I can’t see them like what you’re sharing, but I can see his comments in the Doc directly.

324 00:36:50.270 00:36:50.900 Amber Lin: Yeah.

325 00:37:22.170 00:37:23.410 Amber Lin: Oh.

326 00:37:30.440 00:37:31.400 Amber Lin: okay.

327 00:37:32.220 00:37:41.000 Amber Lin: Cost coding like, what does he mean? Cost? Okay. Cost.

328 00:37:41.570 00:37:43.509 Demilade Agboola: Very good. Customer. Interesting. Oh, okay.

329 00:37:44.620 00:37:47.370 Amber Lin: Look at their licenses, and like Oh.

330 00:37:47.370 00:37:48.300 Demilade Agboola: I mean, it’s great.

331 00:37:48.300 00:37:49.140 Amber Lin: English.

332 00:37:49.140 00:37:52.060 Demilade Agboola: Phase 2 goals rebuild as well.

333 00:37:52.410 00:37:55.109 Demilade Agboola: I mean to be fair, I think what he’s trying to say is like

334 00:37:55.940 00:38:01.580 Demilade Agboola: all these things we’re doing? How do they translate into like a dollar cost amount of savings like?

335 00:38:02.640 00:38:05.040 Demilade Agboola: Because I don’t think I think

336 00:38:05.370 00:38:22.500 Demilade Agboola: indirectly, what we’re doing will save them cost. Because if, for instance, we run less models like by getting rid of the bloated models, we make the runtime shorter. So that means again, the servers are running for less time. All that stuff like it. It does translate to a cost, cutting

337 00:38:23.740 00:38:25.969 Amber Lin: Oh, okay.

338 00:38:25.970 00:38:31.209 Demilade Agboola: I think, need to be more explicit. One about like how these things translate into dollar savings and 2.

339 00:38:31.210 00:38:31.830 Amber Lin: Tomorrow.

340 00:38:31.830 00:38:45.569 Demilade Agboola: Actually also look at like when we’re like saving you all these like different using polytonic stage evil like setting on one tool. You don’t have 3 different contracts you’re running. How do we save cost in that way? Like like? So I think what

341 00:38:45.950 00:38:54.310 Demilade Agboola: potentially like these are all great ideas and all like the great direction. It’s just like, I guess what he’s trying he’s trying to say is like, how do we explicitly call out.

342 00:38:55.965 00:39:00.540 Amber Lin: Like how this is like, how these are saving costs, or how these things save cost.

343 00:39:01.340 00:39:11.350 Amber Lin: Okay, alright sounds good whatever that’s

344 00:39:20.860 00:39:22.400 Amber Lin: sounds good.

345 00:39:23.530 00:39:30.309 Amber Lin: So okay, yeah, that’s good.

346 00:39:30.930 00:39:33.689 Amber Lin: I don’t even I don’t even know

347 00:39:35.700 00:39:43.790 Amber Lin: who assigned all of this. This is tragedy. It’s not probably not gonna be me. But I can.

348 00:39:44.450 00:39:47.159 Amber Lin: I can figure that out.

349 00:39:52.640 00:40:03.430 Amber Lin: Okay, do you have a list of can you like export? A list of look? Our dashboards.

350 00:40:06.530 00:40:08.250 Demilade Agboola: Lists of current. Look at dashboard!

351 00:40:08.740 00:40:16.340 Demilade Agboola: I will look at that. I should be able to do that. Might have to be later in the day. Because, like, like, I’m kind of front loaded today.

352 00:40:17.830 00:40:20.380 Amber Lin: That’s okay. I am like.

353 00:40:22.117 00:40:25.060 Amber Lin: I can see if Utop can give you one.

354 00:40:25.720 00:40:30.559 Amber Lin: and if not, I have some stuff to do with the cost, and then making

355 00:40:30.740 00:40:38.010 Amber Lin: like he wants more of a planned out phase one and phase 2. So I have some set stuff to do.

356 00:40:47.290 00:40:49.650 Amber Lin: Okay, info.

357 00:41:17.990 00:41:20.030 Amber Lin: do. We have stand ups with them.

358 00:41:21.190 00:41:24.479 Demilade Agboola: Yeah, we currently have, like, 15 min stand ups every day.

359 00:41:25.460 00:41:27.279 Amber Lin: Every day. Okay. Daily.

360 00:41:29.040 00:41:31.769 Demilade Agboola: I think it’s not like. Give me one second.

361 00:41:34.310 00:41:40.180 Demilade Agboola: It’s one pm. Et. Like No. 1245 to one Pm. Et.

362 00:41:48.090 00:41:53.490 Amber Lin: Sounds good, wrong, dates.

363 00:41:53.490 00:41:55.880 Demilade Agboola: What time? What’s the time in la right now?

364 00:41:58.880 00:42:01.480 Amber Lin: It’s 7 40 Am.

365 00:42:02.120 00:42:03.909 Demilade Agboola: That’s kind of early for you.

366 00:42:05.231 00:42:09.300 Amber Lin: I. My 1st meetings are 6 30,

367 00:42:10.320 00:42:13.729 Amber Lin: cause I. My team is in Asia.

368 00:42:14.560 00:42:16.600 Demilade Agboola: So I get up

369 00:42:16.880 00:42:21.210 Demilade Agboola: at this point. You should. You should change your name to like early one cause you know you’re.

370 00:42:23.350 00:42:32.140 Amber Lin: I never thought I’ll get up this early, like I used to when I when I was still at school. I get up at like 11 am.

371 00:42:34.070 00:42:43.529 Amber Lin: If cool. Thank you for your help. I wanted to do 30 min, but I guess I

372 00:42:43.690 00:42:52.010 Amber Lin: but I guess we didn’t make it 30 min. But, thank you. This was really really helpful.

373 00:42:52.750 00:42:58.569 Demilade Agboola: Glad to be of help. If you, if you get stuck, shoot me another message, and we’ll point something out.

374 00:42:59.480 00:43:06.610 Amber Lin: Okay, yeah. Sounds good. I’ll try and get the from. If not, I’ll ask you, okay.

375 00:43:06.940 00:43:07.690 Demilade Agboola: Okay.

376 00:43:07.960 00:43:08.910 Amber Lin: Bye-bye.

377 00:43:09.210 00:43:10.040 Demilade Agboola: Bye.