Meeting Title: All Engineering (AI Uses + Workshop) Date: 2025-09-25 Meeting participants: Mustafa Raja, Samuel Roberts, Casie Aviles, Demilade Agboola
WEBVTT
1 00:00:18.540 ⇒ 00:00:19.400 Samuel Roberts: Okay.
2 00:00:21.390 ⇒ 00:00:22.180 Mustafa Raja: A…
3 00:00:26.330 ⇒ 00:00:27.100 Samuel Roberts: See?
4 00:00:28.320 ⇒ 00:00:30.169 Samuel Roberts: The other two join.
5 00:00:34.550 ⇒ 00:00:39.999 Samuel Roberts: So I know we were supposed to be initially, like, doing little presentations and stuff, but that kind of fell off a little bit, because I…
6 00:00:40.410 ⇒ 00:00:46.569 Samuel Roberts: well, messed up the schedule a little bit and lost it. So what I was thinking today is…
7 00:00:47.600 ⇒ 00:00:51.299 Samuel Roberts: Either or both of the other guys are here, is actually using this time
8 00:00:51.580 ⇒ 00:00:58.190 Samuel Roberts: Because one of our goals with the KPIs is to start mapping out or documenting workflows, including engineering.
9 00:00:58.520 ⇒ 00:01:05.310 Samuel Roberts: So I might just… Kinda… Ask them some questions.
10 00:01:05.690 ⇒ 00:01:08.569 Samuel Roberts: Let you guys ask questions.
11 00:01:08.970 ⇒ 00:01:12.320 Samuel Roberts: And… we’ll just have the transcript, potentially.
12 00:01:12.610 ⇒ 00:01:20.770 Samuel Roberts: understand a few more of their processes. I don’t know how much insight you guys have into those already, but I don’t have a ton, so…
13 00:01:22.170 ⇒ 00:01:27.799 Samuel Roberts: The goal is to, like, document and then figure out where we can… add more…
14 00:01:28.360 ⇒ 00:01:30.910 Samuel Roberts: Assistance, whether that’s, you know.
15 00:01:32.000 ⇒ 00:01:40.710 Samuel Roberts: helping out with cursor stuff, or a platform thing, or… some other people might want Slackbots, I don’t know if engineering’s gonna need that as much, but we’ll find out, you know?
16 00:01:40.960 ⇒ 00:01:44.680 Mustafa Raja: Thank you for not.
17 00:01:44.680 ⇒ 00:01:47.010 Samuel Roberts: So… Yeah.
18 00:01:48.360 ⇒ 00:01:56.380 Samuel Roberts: If they don’t join, I might just call it then, because we’ve already talked a few times today. Well, let’s give them…
19 00:01:56.610 ⇒ 00:02:02.859 Mustafa Raja: Well, while we are at it, now that we have the decks.
20 00:02:02.980 ⇒ 00:02:07.279 Mustafa Raja: Yeah. Is there a ticket that we unlocked that I need to work on?
21 00:02:08.199 ⇒ 00:02:13.299 Samuel Roberts: Yeah, there’s definitely something. Let me take a look real quick, I don’t know…
22 00:02:13.300 ⇒ 00:02:15.229 Mustafa Raja: Yeah, that’ll really help.
23 00:02:15.970 ⇒ 00:02:19.480 Samuel Roberts: Yeah, you’re absolutely right, I should have thought of that on the side.
24 00:02:19.480 ⇒ 00:02:22.439 Mustafa Raja: And Mika did send a questionnaire, and…
25 00:02:22.780 ⇒ 00:02:24.010 Samuel Roberts: I saw that, yeah.
26 00:02:24.010 ⇒ 00:02:24.345 Mustafa Raja: break…
27 00:02:26.050 ⇒ 00:02:33.229 Mustafa Raja: I don’t know what the narrative intake is, but the questionnaire… oh, no, this actually is the… both things are the questionnaires. Okay.
28 00:02:33.850 ⇒ 00:02:37.969 Samuel Roberts: Okay, yeah, well, it’s something to at least add to it, you know? .
29 00:02:37.970 ⇒ 00:02:38.290 Mustafa Raja: What’s…
30 00:02:38.290 ⇒ 00:02:39.599 Samuel Roberts: See what we have.
31 00:02:40.660 ⇒ 00:02:52.550 Samuel Roberts: So… Ingest final interlude… decks, text and structure for evals. I think this is the one that…
32 00:02:53.080 ⇒ 00:02:56.289 Samuel Roberts: I’m assigned to right now, but yeah.
33 00:02:57.710 ⇒ 00:03:00.369 Samuel Roberts: I think this is just because we were waiting,
34 00:03:02.330 ⇒ 00:03:04.740 Samuel Roberts: So I think this is it,
35 00:03:05.030 ⇒ 00:03:09.920 Samuel Roberts: This is maybe a little exploratory as well, because… I’m not sure.
36 00:03:10.880 ⇒ 00:03:14.249 Samuel Roberts: It looks like the PDFs aren’t, like, searchable text.
37 00:03:15.170 ⇒ 00:03:15.930 Mustafa Raja: Oh.
38 00:03:16.040 ⇒ 00:03:16.560 Mustafa Raja: That…
39 00:03:16.560 ⇒ 00:03:17.410 Samuel Roberts: So…
40 00:03:17.420 ⇒ 00:03:18.130 Mustafa Raja: I love this.
41 00:03:18.130 ⇒ 00:03:27.440 Samuel Roberts: Yeah, I’m wondering if we just try to do something and see how good the output is, and if not, request they, like, generate ones that are not just images from Figma?
42 00:03:27.510 ⇒ 00:03:29.440 Mustafa Raja: But that’s another step.
43 00:03:29.720 ⇒ 00:03:36.100 Mustafa Raja: And this is for ingest… In just… ingest interlude decks.
44 00:03:36.870 ⇒ 00:03:37.510 Samuel Roberts: Yeah.
45 00:03:38.090 ⇒ 00:03:43.409 Mustafa Raja: So this is converting these PDFs to the Notion doc, right?
46 00:03:44.460 ⇒ 00:03:54.349 Samuel Roberts: Yeah, also the idea is, yeah, that’s one thing we can do. The other thing I’m trying to think of is if we just take the copy and stuff off that, and maybe…
47 00:03:55.360 ⇒ 00:03:58.790 Samuel Roberts: figure out… A way to get closer to that.
48 00:03:58.940 ⇒ 00:04:00.599 Samuel Roberts: Final output from the…
49 00:04:01.130 ⇒ 00:04:01.529 Mustafa Raja: Oh, yeah.
50 00:04:01.530 ⇒ 00:04:02.390 Samuel Roberts: agent.
51 00:04:02.700 ⇒ 00:04:05.499 Samuel Roberts: But we can talk a little bit more about that. It’s a little…
52 00:04:05.500 ⇒ 00:04:06.630 Mustafa Raja: The ticket is…
53 00:04:06.700 ⇒ 00:04:12.019 Samuel Roberts: Decently groomed, but… I think there’s more technical stuff to hammer out, so…
54 00:04:12.620 ⇒ 00:04:16.669 Samuel Roberts: I would say if you could, yeah, think about that a little bit, maybe we can chat later.
55 00:04:17.209 ⇒ 00:04:19.020 Samuel Roberts: Because, yeah, I was a little…
56 00:04:19.640 ⇒ 00:04:25.999 Samuel Roberts: disappointed they weren’t, like, highlightable text, at least, because I thought that’d make it easier to pull out the text, but we’ll see.
57 00:04:26.890 ⇒ 00:04:28.470 Samuel Roberts: Hey, Dimlade.
58 00:04:30.660 ⇒ 00:04:31.680 Demilade Agboola: Hi, Sam.
59 00:04:32.130 ⇒ 00:04:33.870 Demilade Agboola: Hi, Casey. How are you today?
60 00:04:33.870 ⇒ 00:04:35.869 Casie Aviles: I’m pretty good. How’s everyone doing?
61 00:04:36.040 ⇒ 00:04:37.490 Mustafa Raja: Doing good, ding good.
62 00:04:37.490 ⇒ 00:04:38.679 Samuel Roberts: We’re doing alright.
63 00:04:39.160 ⇒ 00:04:42.109 Samuel Roberts: Do you know if Awish is joining?
64 00:04:43.320 ⇒ 00:04:44.830 Demilade Agboola: I have no idea.
65 00:04:45.080 ⇒ 00:05:02.980 Samuel Roberts: Okay, yeah, he didn’t respond on the thing, but, but now you’re here, what I was, telling these guys is that I’ve dropped the ball a little bit on, like, our kind of little presentation walkthrough thing, from the scheduling side. I know we had to cancel a few of these, and you guys can make a few others, so I think it’s just… I need to do a reset on that.
66 00:05:03.310 ⇒ 00:05:05.740 Samuel Roberts: Especially because, like, I had a few more people on the…
67 00:05:06.410 ⇒ 00:05:10.580 Samuel Roberts: rotation initially that are no longer with us, so I want to make sure that I get it.
68 00:05:10.780 ⇒ 00:05:26.179 Samuel Roberts: set up, but one of the other things that would be helpful, probably, is I don’t have a ton of insight into, like, the work you’re doing, and one of our KPIs is to start documenting workflows across different teams.
69 00:05:26.330 ⇒ 00:05:37.210 Samuel Roberts: And so, you know, I’ve had a meeting with Rico about, like, PM and operations stuff. I’m hoping to talk to Justina tomorrow. I think we’re just setting that up. But I figure while we have,
70 00:05:37.820 ⇒ 00:05:38.620 Samuel Roberts: you know.
71 00:05:38.890 ⇒ 00:05:45.780 Samuel Roberts: both of us here, plus Casey Mustafa to, like, also poke and ask questions. I would love to just kind of
72 00:05:46.680 ⇒ 00:05:56.800 Samuel Roberts: digest a little bit of, like, what your daily work is like, what your, like, workflows are like. Again, this is just kind of for us to get an idea of
73 00:05:58.320 ⇒ 00:06:03.570 Samuel Roberts: What you’re currently doing, and then afterwards, we can kind of figure out maybe ways to, like.
74 00:06:03.930 ⇒ 00:06:08.080 Samuel Roberts: add some AI or, whatever it would be.
75 00:06:08.180 ⇒ 00:06:20.089 Samuel Roberts: Or just improve, like, cursor usage, or whatever it is. You know, I don’t want to think too much about potentials at this point, I just want to get an understanding, because I’m not super familiar with, like, the data side.
76 00:06:20.850 ⇒ 00:06:25.910 Samuel Roberts: So, if that’s good with you now, I would love, to just kind of…
77 00:06:26.300 ⇒ 00:06:29.749 Samuel Roberts: casually interview you about your work, and I’ll…
78 00:06:30.070 ⇒ 00:06:34.829 Samuel Roberts: take some notes, and we’ll have the transcript, and then, I just want to kind of get a sense of, like.
79 00:06:36.240 ⇒ 00:06:46.970 Samuel Roberts: how many different workflows you have, like, how different, you know, how varied they are, how many… and then we can dig into each of them. So I’m curious, like, for example, I went through with Rico, and we went through, like, the entire
80 00:06:47.150 ⇒ 00:06:55.240 Samuel Roberts: like, PM cycle of a client, and all the onboarding stuff for new clients, and can he talk me through, like.
81 00:06:55.310 ⇒ 00:07:09.749 Samuel Roberts: what happens, pointed me to some SOPs that, they have that I didn’t realize. But I guess I just kind of want to do that with you now, because we have the time on the calendar, at least, and I’ve been trying to get to some of these and haven’t really had the time yet. So, if you’re good with that, I guess…
82 00:07:09.900 ⇒ 00:07:14.559 Samuel Roberts: Just high level, like… How many different, like, work…
83 00:07:14.710 ⇒ 00:07:22.899 Samuel Roberts: flows do you think you have? Like, I’m sure there’s, like, specific, like, things that happen here and there that you have to do, but, like, what is a couple general workflows you might have?
84 00:07:24.150 ⇒ 00:07:31.670 Demilade Agboola: So general workflows would be, building out new data models,
85 00:07:31.670 ⇒ 00:07:32.910 Samuel Roberts: Right.
86 00:07:32.910 ⇒ 00:07:33.530 Demilade Agboola: mobile.
87 00:07:34.030 ⇒ 00:07:39.289 Demilade Agboola: Debugging and trying to fix existing data flows.
88 00:07:40.040 ⇒ 00:07:45.340 Demilade Agboola: Then… documentation.
89 00:07:45.580 ⇒ 00:07:46.240 Demilade Agboola: of…
90 00:07:46.240 ⇒ 00:07:46.569 Samuel Roberts: Of course.
91 00:07:48.100 ⇒ 00:07:51.279 Demilade Agboola: Either the fixes, or what was just built.
92 00:07:51.670 ⇒ 00:07:57.079 Demilade Agboola: Then we have things like ingesting new data.
93 00:07:57.830 ⇒ 00:08:01.249 Demilade Agboola: So, that’s different from, like, in the sense of, like.
94 00:08:01.470 ⇒ 00:08:08.319 Demilade Agboola: Ingesting is the first process of getting it in before you start to build the model, so ingesting will be the first part.
95 00:08:11.340 ⇒ 00:08:21.100 Demilade Agboola: then… Trying to think of if there’s any other… So,
96 00:08:21.320 ⇒ 00:08:25.950 Demilade Agboola: Probably things like when we have spikes to investigate and compare.
97 00:08:26.520 ⇒ 00:08:27.339 Samuel Roberts: Of course, yeah.
98 00:08:27.730 ⇒ 00:08:31.459 Demilade Agboola: Yeah, the… so when you want to ingest, we use,
99 00:08:31.940 ⇒ 00:08:37.879 Demilade Agboola: Should we use Polyatomic? Shall we use Fivetrans? Shall we use this tool? Shall we use that tool? That’s also the way…
100 00:08:38.260 ⇒ 00:08:43.479 Demilade Agboola: Sort of, like, spikes for that kind of thing, or for this client, should we… what tool…
101 00:08:43.679 ⇒ 00:08:51.200 Demilade Agboola: what warehouse will be the best, or what visualization tool will be the best, I guess. So, I guess that’s another, like, flow.
102 00:08:52.020 ⇒ 00:08:53.620 Demilade Agboola: Cool.
103 00:08:53.880 ⇒ 00:08:59.859 Demilade Agboola: Yeah, I think that’s largely… those would be the… I mean, if anyone comes to mind, I will feel free to mention. Those are, like, the high level.
104 00:08:59.860 ⇒ 00:09:02.989 Samuel Roberts: Yeah, yeah, definitely. No, that’s really good, because,
105 00:09:03.220 ⇒ 00:09:08.760 Samuel Roberts: this is… this is helpful, because I had, like, a general sense of some of that stuff, but I don’t… again, I’m not…
106 00:09:09.110 ⇒ 00:09:17.550 Samuel Roberts: As familiar. So this is, this is helpful. So I guess the general process, maybe, from my understanding. So, like, a new client comes in.
107 00:09:17.710 ⇒ 00:09:21.000 Samuel Roberts: And you probably need to…
108 00:09:21.110 ⇒ 00:09:24.959 Samuel Roberts: ingest the data, so you might do a spike on the tools. Is that what would happen first?
109 00:09:26.200 ⇒ 00:09:27.799 Demilade Agboola: So what would happen first would be…
110 00:09:28.350 ⇒ 00:09:29.249 Samuel Roberts: Sorry, go ahead.
111 00:09:29.700 ⇒ 00:09:41.520 Demilade Agboola: What will happen first is that we’ll need to do an, like, a high-level audit, kind of figure out where they are, because not all clients are the same. Some are… they have the details, for instance, but it’s just a mess.
112 00:09:41.830 ⇒ 00:09:49.480 Demilade Agboola: Some don’t have that data, so it’s about ingesting it and getting it in. So it’s… so that kind of varies per client, but…
113 00:09:49.550 ⇒ 00:09:50.690 Samuel Roberts: Ideally, yeah.
114 00:09:50.760 ⇒ 00:09:56.069 Demilade Agboola: we will need to ingest the data. If they do have the data available, we’ll need to ingest it first step.
115 00:09:56.650 ⇒ 00:10:00.149 Samuel Roberts: Sure, okay, okay. So, I guess walk me through, like, what a…
116 00:10:00.570 ⇒ 00:10:04.309 Samuel Roberts: Workflow for this ingestion is like, because,
117 00:10:04.540 ⇒ 00:10:10.810 Samuel Roberts: So there’s… there’s data somewhere that they have, and you need to kind of pull it into something.
118 00:10:11.640 ⇒ 00:10:15.109 Samuel Roberts: clean it up a bit, I guess? I don’t know, you tell me.
119 00:10:15.290 ⇒ 00:10:18.969 Demilade Agboola: So, to ingest… to ingest data, the first thing would be to…
120 00:10:20.810 ⇒ 00:10:25.130 Demilade Agboola: Go to the source of the data, see what form it exists in.
121 00:10:25.510 ⇒ 00:10:34.170 Demilade Agboola: Two is figure out, like, how you want to connect to that source. Do they have APIs? Do they have…
122 00:10:34.470 ⇒ 00:10:35.850 Demilade Agboola: Do they have, like.
123 00:10:37.280 ⇒ 00:10:45.480 Demilade Agboola: do they have, like, an STPF, or STFP, where you can get it, like, loaded every day? Some also have, like, native connectors.
124 00:10:45.630 ⇒ 00:10:50.889 Demilade Agboola: For instance, if you’re doing something with… Like, in the Google
125 00:10:51.430 ⇒ 00:11:03.569 Demilade Agboola: Google Space, you can easily move it into, like, BigQuery, for instance. So that would be the first thing. You want to be sure that you can get the… you can… you know what the type is.
126 00:11:03.760 ⇒ 00:11:18.590 Demilade Agboola: And you have a way to access it. The next thing would be, once you’ve figured out, okay, we have a way to access it, would be what tool would you use to access it? So, in that case, you’re thinking, is there an easy tool to use?
127 00:11:18.830 ⇒ 00:11:28.839 Demilade Agboola: In terms of, like, technicality, and then you also have to look at the financial aspect of it. Is it cheap to use? So then…
128 00:11:29.050 ⇒ 00:11:43.209 Demilade Agboola: You weigh those, then you decide the tool to use, and then you basically, once you have factored those things in, you basically start, like, the actual process of connecting to the source, scheduling it, like, how often do we need that data, and then…
129 00:11:44.380 ⇒ 00:11:51.969 Demilade Agboola: testing it. You need to test and ensure that the data is actually flowing as it should, and landing in your warehouse as it should.
130 00:11:52.260 ⇒ 00:11:56.940 Demilade Agboola: Once you have those things done, then you can start to think of…
131 00:11:57.350 ⇒ 00:12:02.510 Demilade Agboola: The next part, which is modeling the data, which is where cleaning comes in, or tends to come in.
132 00:12:02.510 ⇒ 00:12:03.460 Samuel Roberts: Okay.
133 00:12:04.440 ⇒ 00:12:07.429 Samuel Roberts: But that’s the modeling process, once the data’s ingested, cool.
134 00:12:08.050 ⇒ 00:12:10.190 Samuel Roberts: Alright, so,
135 00:12:11.550 ⇒ 00:12:24.020 Samuel Roberts: I want to dig in a little bit, but I’m not sure where yet. Maybe let’s just keep… keep there. So that’s ingestion, now I have a decent sense of that. So what goes into the modeling aspect and cleaning, and all this stuff? That would come after this?
136 00:12:24.600 ⇒ 00:12:31.109 Demilade Agboola: So the modeling is usually where we start to use things like a tool like dbt, so we set…
137 00:12:31.330 ⇒ 00:12:35.680 Demilade Agboola: We connect that to the warehouse, And…
138 00:12:36.220 ⇒ 00:12:39.459 Demilade Agboola: So we largely model in a star schema.
139 00:12:40.050 ⇒ 00:12:41.410 Demilade Agboola: That’s the goal.
140 00:12:41.580 ⇒ 00:12:44.589 Demilade Agboola: And then, so your star schema just basically means you have
141 00:12:47.790 ⇒ 00:12:53.180 Demilade Agboola: Sorry, I have a… something in my chest. Yeah, so it starts as if he means that…
142 00:12:53.900 ⇒ 00:13:01.099 Demilade Agboola: Your focus is having some key fact and dimensional models that you can link together to create, like.
143 00:13:02.350 ⇒ 00:13:05.210 Demilade Agboola: more solidifacts, so, like.
144 00:13:05.690 ⇒ 00:13:14.189 Demilade Agboola: A fact model is, for instance, something like… it’s an event that happened, so an order, a transaction, those are facts, so you can have fact orders.
145 00:13:14.410 ⇒ 00:13:19.600 Demilade Agboola: Back to events… fact, transactions.
146 00:13:19.780 ⇒ 00:13:24.720 Demilade Agboola: And then you can have dimensions. So dimensions are the…
147 00:13:25.430 ⇒ 00:13:37.830 Demilade Agboola: details that we add to those orders. So, for instance, a customer would be dimension. The items purchased, or deem items, or deem, products will be…
148 00:13:38.000 ⇒ 00:13:39.300 Demilade Agboola: a dimension.
149 00:13:39.450 ⇒ 00:13:43.550 Demilade Agboola: And so that’s how you can put them together. So, for instance, I want to have a full view of
150 00:13:44.560 ⇒ 00:13:54.770 Demilade Agboola: what was bought, I would want to have my orders with my dimensions, so dimensions, like products put together. Basically, we start
151 00:13:54.770 ⇒ 00:14:09.179 Demilade Agboola: From the beginning, where we take all these things, we start cleaning the data, standardizing data types, handling null values, setting primary keys in some cases, setting up tests to validate those primary keys are,
152 00:14:09.310 ⇒ 00:14:11.470 Demilade Agboola: Are true, are hold to be true.
153 00:14:13.130 ⇒ 00:14:19.900 Demilade Agboola: And dbt comes with a lot of natural tests, like, general, like, generic tests, but you can also do really, really specific tests as well.
154 00:14:19.900 ⇒ 00:14:20.560 Samuel Roberts: Hmm…
155 00:14:20.920 ⇒ 00:14:37.899 Demilade Agboola: and then you put all these things together, and eventually, you should have your data modeled. So, the goal for a good model is it should answer business questions. So, you always want to know what does the business,
156 00:14:38.240 ⇒ 00:14:41.139 Demilade Agboola: What was the business trying to answer, number one?
157 00:14:41.240 ⇒ 00:14:47.819 Demilade Agboola: Number two is the person who’s building the dashboard, what do they need out of the data?
158 00:14:48.100 ⇒ 00:14:50.920 Demilade Agboola: So, sometimes, for instance, if you…
159 00:14:51.130 ⇒ 00:15:06.939 Demilade Agboola: if I go to drill down into data, it, for instance, would tell you… it gives you the level of detail you need to make the data. So if, for instance, I say, I roll up data to the day… every day, we sold maybe $10,000 worth of a product, right?
160 00:15:06.990 ⇒ 00:15:16.169 Demilade Agboola: If I give you just that data of date and amount, date and amount, date and amount, you can’t build a dashboard where you can dig into, where you can see, okay, so what.
161 00:15:16.170 ⇒ 00:15:16.650 Samuel Roberts: It’s had.
162 00:15:16.650 ⇒ 00:15:17.500 Demilade Agboola: the highest.
163 00:15:17.620 ⇒ 00:15:35.800 Demilade Agboola: you know, sales. So being able to understand what is needed of the dashboard, figure out what the sources are, and putting everything together is kind of the modeling process. To Casey’s question about validating the data, so, I mean, obviously, when you’re ingesting the data, you will need to
164 00:15:36.730 ⇒ 00:15:49.040 Demilade Agboola: ensure that the data in your warehouse matches the data in the source, so you will do things… ensure the counts are fine. So, like, do I have the same number of rows? Are the columns the same? That sort of thing.
165 00:15:49.040 ⇒ 00:15:49.930 Samuel Roberts: Hmm.
166 00:15:49.930 ⇒ 00:16:02.139 Demilade Agboola: Obviously, it’s a long-term process, and so for things like that, you would create tests, usually within dbt, or also within your warehouse. So, for instance, in your warehouse, you can put the primary key constraint.
167 00:16:02.290 ⇒ 00:16:10.329 Demilade Agboola: So, for instance, that allows you to know that we should only have one primary key in this entire table. If it fails, then, you know, create an error.
168 00:16:10.570 ⇒ 00:16:17.429 Demilade Agboola: Things like that. So in dbt, you can also create more, like, complex keys, so you can create.
169 00:16:17.430 ⇒ 00:16:18.300 Samuel Roberts: something called.
170 00:16:18.770 ⇒ 00:16:25.970 Demilade Agboola: Where, once you start putting data together, you can realize that, hey.
171 00:16:26.100 ⇒ 00:16:30.860 Demilade Agboola: In a particular table where we’re reporting on…
172 00:16:31.350 ⇒ 00:16:35.989 Demilade Agboola: maybe, like, let’s use the example of when I said the dates and states
173 00:16:36.450 ⇒ 00:16:39.370 Demilade Agboola: revenue, so for every day, and…
174 00:16:39.590 ⇒ 00:16:47.459 Demilade Agboola: you have the revenue for every state. You know that there shouldn’t be… a state should not appear twice on the same day.
175 00:16:47.780 ⇒ 00:16:53.730 Demilade Agboola: like, if I sold something in California and had revenue for, you know, the 24th of…
176 00:16:53.840 ⇒ 00:17:04.080 Demilade Agboola: September 2025, there shouldn’t be another value for California 24th of September 2025. That doesn’t make any sense. So in that case, we’ll create a key
177 00:17:04.099 ⇒ 00:17:18.029 Demilade Agboola: there’s a combination of the date and the state, and say, hey, this is what we call our surrogate key, and that key should always be unique. And so if there’s any, you know, error where those things appear more than once, I’m using… that’s a very simple example there. There are rows where…
178 00:17:18.030 ⇒ 00:17:19.170 Samuel Roberts: Of course, of course.
179 00:17:19.410 ⇒ 00:17:31.699 Demilade Agboola: breaking, like, 10 or 15, you know, columns put together, you know, 5 columns put together, because you want to ensure that the table doesn’t have duplicates.
180 00:17:31.890 ⇒ 00:17:48.159 Demilade Agboola: But yeah, that’s kind of what the model entails, and just trying to ensure that you’re building such a way that it’s easy to, one, debug, and two, build upon, you know, further states, either by yourself or any other person on the team, or even when we leave.
181 00:17:49.830 ⇒ 00:17:52.039 Samuel Roberts: Right, okay, that makes a lot of sense.
182 00:17:55.110 ⇒ 00:18:02.390 Samuel Roberts: So let’s, I think that’s really good. I’m just wondering where to go next with that. I’m thinking…
183 00:18:02.650 ⇒ 00:18:06.670 Samuel Roberts: I would love to learn a little more about, like, dbt specifically.
184 00:18:08.410 ⇒ 00:18:09.500 Demilade Agboola: What?
185 00:18:09.710 ⇒ 00:18:15.690 Samuel Roberts: Yeah, is there, like, a resource or something that I can just familiarize myself with, or do you want to, like, talk through something?
186 00:18:15.690 ⇒ 00:18:23.769 Demilade Agboola: I can go through what dbt is, but then I can just also share, like, some of the things that could help as well. So dbt is basically…
187 00:18:24.040 ⇒ 00:18:29.350 Demilade Agboola: One of my favorite ways to think about DBTs, DBT is SQL on steroids. Okay.
188 00:18:29.710 ⇒ 00:18:36.910 Demilade Agboola: So that’s the first thing. But what dbt does is it tries to apply software engineering practices to
189 00:18:38.080 ⇒ 00:18:53.250 Demilade Agboola: just how people used to write SQL before, and the idea is aim for things like modularity and scalability and, you know, dry processes, so don’t repeat yourself processes. And that allows us to be able to create things like,
190 00:18:54.450 ⇒ 00:18:59.670 Demilade Agboola: how do we build them? So before, prior to that, we would have, like, stored procedures.
191 00:18:59.780 ⇒ 00:19:03.679 Demilade Agboola: And these start procedures will be every morning, or whatever time.
192 00:19:03.800 ⇒ 00:19:08.390 Demilade Agboola: And these stock procedures could be, like, 500 lines long, 1,000 lines long.
193 00:19:08.710 ⇒ 00:19:13.189 Demilade Agboola: And obviously, if it broke, it was always hard to fix, because, like.
194 00:19:13.430 ⇒ 00:19:19.959 Demilade Agboola: It could be any of those lines. So being able to break it down into smaller bits where you can say, hey, this…
195 00:19:20.210 ⇒ 00:19:32.019 Demilade Agboola: is what we call our staging models. These are what we call our intermediate models, and these are what we call our math models. Allows you to, instead of having one big chunk of code, you can break it down to smaller bits.
196 00:19:32.150 ⇒ 00:19:38.480 Demilade Agboola: And that allows you to be able to debug-specific…
197 00:19:38.690 ⇒ 00:19:44.650 Demilade Agboola: bits of code. So instead of having, like, this monolith code, you have, like, smaller bits that you can start to…
198 00:19:45.030 ⇒ 00:19:47.499 Demilade Agboola: attack. So that helps.
199 00:19:47.670 ⇒ 00:19:52.600 Demilade Agboola: Obviously, the advantage… another advantage of that, dbt also does something called references.
200 00:19:52.760 ⇒ 00:19:54.699 Demilade Agboola: So you can refer to…
201 00:19:55.220 ⇒ 00:20:08.419 Demilade Agboola: another model. So, like, instead of having this big code, you break it down to smaller bits, and say, okay, this bit is the first part, this second part refer to the first part, and maybe this third part refer to the second part.
202 00:20:08.420 ⇒ 00:20:17.749 Demilade Agboola: And so now, when you’re done with everything, you have something called a DAG, so you can have a flow of how data should, like, how the processes should work.
203 00:20:17.920 ⇒ 00:20:23.550 Demilade Agboola: And that allows you to schedule, that allows you to be able to say, hey, I can have my…
204 00:20:24.380 ⇒ 00:20:31.700 Demilade Agboola: I can have different things on different schedules quite easily. It allows you to be able to…
205 00:20:32.470 ⇒ 00:20:41.979 Demilade Agboola: Also, ensure that basically… anybody that comes in, oh yeah, and by integrating with Git, my Git pro…
206 00:20:42.780 ⇒ 00:20:55.179 Demilade Agboola: process. It allows you to also ensure that anybody that comes in can see who made those changes, when they made those changes, and ideally, why they made those changes, you know, through the Git message.
207 00:20:55.740 ⇒ 00:21:05.639 Demilade Agboola: again, it adds a level of transparency that, like, a stored procedure would not have, because if I change line 585 in a stored procedure, no one knows why.
208 00:21:05.640 ⇒ 00:21:06.370 Samuel Roberts: And…
209 00:21:06.530 ⇒ 00:21:15.999 Demilade Agboola: reports might start breaking. But, like, now it’s easy. If reports change, or reports start breaking, I just go to the Git history, the PR history, and I can quickly see, oh.
210 00:21:16.620 ⇒ 00:21:26.170 Demilade Agboola: XYZ person made a change yesterday, and since then, reports have been failing, I need to roll this back, or I need to, you know, investigate this and troubleshoot why it broke.
211 00:21:26.600 ⇒ 00:21:46.020 Demilade Agboola: You know, so that’s basically high level what dbt entails. You’re just bringing software engineering practices into data to allow for better debugging and for better transparency, as well as just, you know, a better flow to how things… flow and organization to how things have been.
212 00:21:46.920 ⇒ 00:21:49.270 Samuel Roberts: Yeah, that makes a lot of sense,
213 00:21:50.190 ⇒ 00:21:57.789 Samuel Roberts: I… I can totally understand that now, the way you phrased it, like, the… applying the software practices. Okay. Cool.
214 00:21:58.680 ⇒ 00:22:04.909 Samuel Roberts: Yeah… Is there anything else that I should be asking about that I haven’t yet? Like,
215 00:22:05.240 ⇒ 00:22:14.179 Samuel Roberts: something we just haven’t mentioned that I’m like, oh, that you would say, like, oh, you should know this kind of thing for the… or we covered, like, the broad stuff, and then we’re getting more specific.
216 00:22:14.390 ⇒ 00:22:15.899 Samuel Roberts: Is there anything else outside this?
217 00:22:17.080 ⇒ 00:22:22.309 Demilade Agboola: Right off the top of my head, I would say the…
218 00:22:24.230 ⇒ 00:22:30.599 Demilade Agboola: I mean, we use dbt, just so you… dbt is a tool, and it’s like a,
219 00:22:30.900 ⇒ 00:22:32.869 Demilade Agboola: a SQL compiler.
220 00:22:33.000 ⇒ 00:22:35.980 Demilade Agboola: But there are others, you know, we don’t use them.
221 00:22:36.390 ⇒ 00:22:36.970 Demilade Agboola: Gotcha.
222 00:22:36.970 ⇒ 00:22:38.029 Samuel Roberts: Okay, of course.
223 00:22:38.970 ⇒ 00:22:47.849 Demilade Agboola: So… while we might be dbt-specific right now, it’s always possible whatever solutions we prefer might…
224 00:22:47.960 ⇒ 00:22:52.919 Demilade Agboola: Switch to whatever tool we might hop onto eventually, if we do ever hop on another tool.
225 00:22:53.230 ⇒ 00:22:54.670 Demilade Agboola: Cause I know there’s another one.
226 00:22:54.870 ⇒ 00:22:56.870 Demilade Agboola: Or mesh, that’s another one.
227 00:22:57.330 ⇒ 00:23:00.060 Demilade Agboola: So yeah, there are the tools.
228 00:23:00.250 ⇒ 00:23:03.370 Demilade Agboola: And so… yeah, that’s something that…
229 00:23:03.620 ⇒ 00:23:09.650 Demilade Agboola: Like, like I said, I know about, but, like, you might not know if dbt is just what you get used to.
230 00:23:10.850 ⇒ 00:23:13.419 Samuel Roberts: Okay. No, this is, this is good.
231 00:23:13.820 ⇒ 00:23:17.889 Samuel Roberts: Definitely helps me start understanding the process, so…
232 00:23:18.640 ⇒ 00:23:21.699 Samuel Roberts: Casey Mustafa, you got any other questions that pop to mind?
233 00:23:25.660 ⇒ 00:23:28.720 Casie Aviles: I think that was… Definitely…
234 00:23:29.160 ⇒ 00:23:37.559 Casie Aviles: I learned a lot there, because I didn’t realize some of the stuff, existing work that I’ve been doing, has also been, you know.
235 00:23:37.720 ⇒ 00:23:39.889 Casie Aviles: Kind of data engineering already.
236 00:23:41.000 ⇒ 00:23:45.130 Casie Aviles: So that’s why I was asking about the validation part, like.
237 00:23:45.330 ⇒ 00:23:52.140 Casie Aviles: you know, like, pulling in data from APIs, and then scheduling it on Dagster, I think that’s also kind of
238 00:23:52.370 ⇒ 00:23:55.410 Casie Aviles: Similar to what Demilade has been sharing already.
239 00:23:56.700 ⇒ 00:23:59.040 Casie Aviles: So, yeah, yeah.
240 00:23:59.190 ⇒ 00:24:00.969 Casie Aviles: That was definitely helpful.
241 00:24:01.910 ⇒ 00:24:02.570 Samuel Roberts: Cool.
242 00:24:04.440 ⇒ 00:24:07.450 Mustafa Raja: Yeah, you know, it really, really was helpful.
243 00:24:08.780 ⇒ 00:24:09.739 Demilade Agboola: Yeah, sounds like…
244 00:24:09.740 ⇒ 00:24:10.500 Samuel Roberts: Alright.
245 00:24:10.690 ⇒ 00:24:14.580 Demilade Agboola: just generally, I was just going to say that it might be helpful for, like.
246 00:24:15.290 ⇒ 00:24:30.879 Demilade Agboola: Because I know, like, as we’re growing, we seem to be working farther and farther away. Maybe on a Friday call, it could just be, like, what does XYZ team do beyond just, like, the high level of they work with data, or they are the AI team.
247 00:24:30.880 ⇒ 00:24:32.000 Samuel Roberts: the airport.
248 00:24:32.000 ⇒ 00:24:32.979 Demilade Agboola: Keep them too.
249 00:24:33.740 ⇒ 00:24:40.470 Samuel Roberts: Yeah, because there’s definitely overlaps, and then you should identify those and work together when they’re, like… you know.
250 00:24:41.100 ⇒ 00:24:45.790 Samuel Roberts: When there’s a problem somewhere, like, you might be the best person to talk to instead of me, kind of thing.
251 00:24:45.960 ⇒ 00:24:52.939 Samuel Roberts: But we wouldn’t realize that if we haven’t had, like, a little session like this, or just an overview kind of thing. That’s not a bad idea.
252 00:24:53.610 ⇒ 00:24:54.790 Demilade Agboola: Yeah, yeah.
253 00:24:55.710 ⇒ 00:25:05.529 Samuel Roberts: Well, that’s kind of… yeah, I mean, at least for these two teams, for, like, all of engineering in general, like, this is what this meeting’s for, so I think this was a kind of a successful one. I really appreciate you,
254 00:25:06.440 ⇒ 00:25:12.259 Samuel Roberts: you know, answering all these questions for me, and helping me understand… helping us understand it.
255 00:25:12.910 ⇒ 00:25:14.109 Samuel Roberts: I think that…
256 00:25:14.660 ⇒ 00:25:23.280 Samuel Roberts: I’m good at this point, then, unless y’all have something else you want to talk about as a full engineering team or anything right now? Or almost full, missing a leash, but…
257 00:25:23.900 ⇒ 00:25:27.259 Demilade Agboola: I just wanted to add, since we’re talking flows and things.
258 00:25:27.260 ⇒ 00:25:27.600 Samuel Roberts: Yeah.
259 00:25:27.600 ⇒ 00:25:30.790 Demilade Agboola: Another flow that comes up is…
260 00:25:31.110 ⇒ 00:25:35.169 Demilade Agboola: Just the whole debugging thing. Dashboard breaks…
261 00:25:35.170 ⇒ 00:25:35.910 Samuel Roberts: Sure.
262 00:25:36.640 ⇒ 00:25:44.889 Demilade Agboola: Analyst has no idea why, so you kind of have to debug. So that process is… you start going back… well, I usually start from the…
263 00:25:45.260 ⇒ 00:25:51.320 Demilade Agboola: the dashboard backwards into… Into the source.
264 00:25:51.450 ⇒ 00:26:08.179 Demilade Agboola: So if, for instance, it’s broken going into the dashboard, then it’s something within dbt, for instance. If everything in dbt, like the models, because, you know, like I said, you build from staging, and you build intermediate on your mods, so you kind of work your way backwards.
265 00:26:09.310 ⇒ 00:26:12.980 Demilade Agboola: So, sometimes what I just basically do is, if I have an issue.
266 00:26:13.310 ⇒ 00:26:20.200 Demilade Agboola: if it’s going… if what’s leaving dbt is bad, I hop, check what the source is. It was coming into dbt.
267 00:26:20.580 ⇒ 00:26:22.179 Demilade Agboola: If that’s bad as well.
268 00:26:22.720 ⇒ 00:26:30.130 Demilade Agboola: So, for instance, like, it could be a freshness thing, right? Like, so we’ve not had data updated in the last
269 00:26:30.400 ⇒ 00:26:32.000 Demilade Agboola: 10 hours, right?
270 00:26:32.240 ⇒ 00:26:33.760 Samuel Roberts: if I check…
271 00:26:34.160 ⇒ 00:26:39.849 Demilade Agboola: our model, the final model that fits the dashboard, and I see, hey, yeah, that is true.
272 00:26:39.950 ⇒ 00:26:41.320 Demilade Agboola: The last updated…
273 00:26:41.420 ⇒ 00:26:53.450 Demilade Agboola: time was 10 hours ago, and then I go to the sources, and I check, and I see the last updated time was 10 hours ago, then I know it’s the sources I need to, like, work on.
274 00:26:53.510 ⇒ 00:27:03.559 Demilade Agboola: However, if I check the last… if I check the sources and I say the last updated was, say, an hour ago, then I know something is going on within dbt.
275 00:27:03.900 ⇒ 00:27:06.639 Demilade Agboola: That’s preventing us from having the latest data.
276 00:27:07.890 ⇒ 00:27:08.680 Demilade Agboola: Excellent.
277 00:27:08.680 ⇒ 00:27:10.140 Samuel Roberts: That’s…
278 00:27:10.780 ⇒ 00:27:19.830 Demilade Agboola: So that’s kind of like… so debugging is kind of like that. You kind of might have to go through and read through. In some cases, some hardware cases can take you hours, but you might have…
279 00:27:19.830 ⇒ 00:27:20.190 Samuel Roberts: Excuse me.
280 00:27:21.170 ⇒ 00:27:26.600 Demilade Agboola: hundreds of lines, and just trying to figure out, like, at what point did something break?
281 00:27:26.890 ⇒ 00:27:45.290 Demilade Agboola: at what point did… sometimes it’s not… nothing you could control. The data type had changed upstream, from the source, and that data type has just thrown a spanner in the works and everything. Sometimes think models are failing, and you didn’t have alerts to note that they were failing, and it was the business…
282 00:27:45.660 ⇒ 00:27:46.610 Demilade Agboola: Hold on first.
283 00:27:47.280 ⇒ 00:27:53.939 Demilade Agboola: and you need to push a fix for that, and maybe add constraints to certain things.
284 00:27:54.700 ⇒ 00:28:03.229 Demilade Agboola: yeah, there are a number of reasons why these things can happen, but yeah, that’s… that’s a huge… that can be huge, but, like, literally yesterday, I… I probably spent, like.
285 00:28:04.560 ⇒ 00:28:06.780 Demilade Agboola: 4 hours, debugging yesterday.
286 00:28:06.960 ⇒ 00:28:09.579 Samuel Roberts: Oof, yeah, that’s a… that’s a session.
287 00:28:10.020 ⇒ 00:28:10.679 Demilade Agboola: Yeah, so…
288 00:28:10.680 ⇒ 00:28:27.630 Samuel Roberts: Yeah, debugging is a really good one to bring up, because that is something that we all do in different ways, but the patterns are probably very similar. Like, the way you described it is very similar to how I would go about debugging, like, a front-end to back-end issue if something’s not displaying right on the portal… on the platform.
289 00:28:27.650 ⇒ 00:28:35.590 Samuel Roberts: You know, I start where I know it’s broken, and then work my way backwards till I find this thing that broke it, 100% how I do it.
290 00:28:35.610 ⇒ 00:28:47.840 Samuel Roberts: And sometimes, you know, there’s different ways to go about it, like isolating things and doing all kinds of stuff, but the… the way you described it is at least how I do it, a lot of the time. So I love… that’s… that’s really good to hear, because I’m like.
291 00:28:48.230 ⇒ 00:28:51.889 Samuel Roberts: Oh yeah, we’re all doing very similar things in different forms.
292 00:28:51.890 ⇒ 00:28:53.360 Demilade Agboola: Yeah. Yeah.
293 00:28:53.790 ⇒ 00:28:55.539 Samuel Roberts: Good insight there.
294 00:28:55.720 ⇒ 00:28:56.650 Samuel Roberts: Alright.
295 00:28:57.660 ⇒ 00:29:04.050 Demilade Agboola: Alright then, so I just… I said, let me, let me share that as well, since we’re right here.
296 00:29:04.630 ⇒ 00:29:12.340 Samuel Roberts: Totally, totally. And, yeah, I’m gonna kind of process this a little bit. We’ll have the transcript, I took some notes, and then I’m gonna maybe, like.
297 00:29:13.170 ⇒ 00:29:24.950 Samuel Roberts: I don’t know exactly the final form this is gonna take, but I will probably put something together that’s, like, my understanding of things, and then the workflows kind of, like, generally, like you described it, and then some broken down steps, and, like.
298 00:29:25.080 ⇒ 00:29:35.949 Samuel Roberts: we can talk again and make sure that I’m not missing things, or if I am, you know, what other workflows, because we’re trying to hit, like, a certain percentage of workflows documented, which means
299 00:29:36.310 ⇒ 00:29:42.020 Samuel Roberts: Knowing, I guess, all the workflows as best you can, right? Or knowing all the workflows that exist as best you can.
300 00:29:42.140 ⇒ 00:29:44.920 Samuel Roberts: I think the goal was, like, focusing on,
301 00:29:45.430 ⇒ 00:29:53.630 Samuel Roberts: PM and sales, because we know, like, there’s some places we can really, you know, improve margins or revenue,
302 00:29:54.210 ⇒ 00:30:03.580 Samuel Roberts: And then engineering, I think we were looking to document, like, half, and that is, you know, across both teams. So, this at least gives me a good start for that, and I really appreciate it.
303 00:30:04.870 ⇒ 00:30:05.730 Demilade Agboola: No problem.
304 00:30:05.730 ⇒ 00:30:08.619 Samuel Roberts: I think besides that, we’re good. Thank you guys for the time.
305 00:30:09.790 ⇒ 00:30:10.870 Casie Aviles: Thank you.
306 00:30:11.600 ⇒ 00:30:13.109 Samuel Roberts: Alrighty. Have a good one, y’all.
307 00:30:13.650 ⇒ 00:30:14.400 Mustafa Raja: Thank you.