Meeting Title: Magic Spoon Data Pipeline Sync Date: 2026-01-09 Meeting participants: Demilade Agboola, Mary Burke, Ashwini Sharma, Michael Thorson, Justin Tabarini
WEBVTT
1 00:00:11.480 ⇒ 00:00:12.379 Mary Burke: Hey guys!
2 00:00:13.690 ⇒ 00:00:14.620 Ashwini Sharma: Hello.
3 00:00:14.900 ⇒ 00:00:15.950 Demilade Agboola: Hi, Michael.
4 00:00:16.059 ⇒ 00:00:17.210 Demilade Agboola: How’s everyone?
5 00:00:20.260 ⇒ 00:00:22.139 Michael Thorson: Pretty good over here. How about you guys?
6 00:00:22.520 ⇒ 00:00:25.120 Demilade Agboola: Pretty good, pretty good. Weekends are around the corner, so…
7 00:00:26.200 ⇒ 00:00:27.040 Michael Thorson: Bingo.
8 00:00:27.040 ⇒ 00:00:27.880 Demilade Agboola: Yeah.
9 00:00:30.710 ⇒ 00:00:37.000 Demilade Agboola: So, before I go into it, because I have slides prepared, did you have anything you wanted to bring up?
10 00:00:37.170 ⇒ 00:00:39.730 Demilade Agboola: Anything that you wanted to touch base on?
11 00:00:41.110 ⇒ 00:00:44.600 Mary Burke: Nothing for me outside of what you’ve already outlined.
12 00:00:44.600 ⇒ 00:00:45.250 Demilade Agboola: Okay.
13 00:00:46.290 ⇒ 00:00:48.340 Demilade Agboola: Same with you, okay. Alright then, so let me show you.
14 00:00:49.280 ⇒ 00:00:52.329 Demilade Agboola: Let me jump in, and we can kind of just share updates.
15 00:00:52.840 ⇒ 00:00:56.870 Demilade Agboola: So… First things first…
16 00:00:58.060 ⇒ 00:01:03.100 Demilade Agboola: So these are our product notes that still have not changed, so we know where we’re going to at the end of this project.
17 00:01:03.340 ⇒ 00:01:11.279 Demilade Agboola: So, the agenda is still, like, the Spins API update, as well as the long-running models updates.
18 00:01:11.540 ⇒ 00:01:16.559 Demilade Agboola: So, the Space API update, I would like to just point out that we’ve been able to do…
19 00:01:16.730 ⇒ 00:01:18.920 Mary Burke: Spence API Discovery.
20 00:01:19.300 ⇒ 00:01:23.600 Demilade Agboola: as well as the pipeline implementation. So I’ll hand it over to,
21 00:01:24.120 ⇒ 00:01:29.519 Demilade Agboola: Ashwini right now, so he can kind of explain what we have so far and what our current blocker is.
22 00:01:30.810 ⇒ 00:01:41.550 Ashwini Sharma: Yeah, so basically, I have a pipeline code that is ready with me, and it kind of, you know, fits into the overall framework of other pipelines that you have deployed in Prefect.
23 00:01:41.810 ⇒ 00:01:53.539 Ashwini Sharma: I won’t be able to deploy it in Perfect Cloud until we have certain keys available over there, and once they are available, I’ll be able to deploy it, but essentially what it does is, kind of.
24 00:01:53.680 ⇒ 00:01:58.620 Ashwini Sharma: It loops over the reporting levels, and for each level, it fetches all the data from
25 00:01:58.750 ⇒ 00:02:00.940 Ashwini Sharma: The latest date till…
26 00:02:01.090 ⇒ 00:02:14.579 Ashwini Sharma: 2022nd, 2022, year 20… the first, whatever is available in the entire system, right? And it writes into a table that I’ve created, which is… which contains, weekly data.
27 00:02:14.970 ⇒ 00:02:23.989 Ashwini Sharma: For, you know, the entire time frame. And, when we do the incremental based on certain cadence, the data… data release cadence.
28 00:02:24.200 ⇒ 00:02:30.180 Ashwini Sharma: It will just do an incremental update for up to… 26 weeks.
29 00:02:30.690 ⇒ 00:02:33.610 Ashwini Sharma: Which roughly translates to 6 months.
30 00:02:37.160 ⇒ 00:02:49.409 Michael Thorson: Great, that sounds good, Ashwini. I’ll take a look at that table that you, like, seeded into the warehouse, and I’ll… I’ll just, like, double check that the fields are looking pretty good to start with.
31 00:02:50.340 ⇒ 00:02:51.370 Ashwini Sharma: Sure, yeah.
32 00:02:51.710 ⇒ 00:03:03.220 Michael Thorson: Yeah, and aware of the prefix blocker as well. Again, just waiting on, owner access for our account, so we can start adding in those, keys to the prefixed environment.
33 00:03:04.980 ⇒ 00:03:23.180 Ashwini Sharma: Sure. And I did some basic QA on the stuff that we had discussed earlier, where we were trying to compare, you know, data from different timeframes, and it kind of matches, right? So when you sum up 4 weeks, and then compare it with the 4-week timeframe period, it kind of matches just…
34 00:03:23.330 ⇒ 00:03:27.670 Ashwini Sharma: Aggregation of, individual elements.
35 00:03:28.120 ⇒ 00:03:42.069 Michael Thorson: Yeah. I have another table. I was gonna talk to you about that, soon, but you’re… you’re moving pretty quick. I do have a table that’s in the warehouse that’s aggregated up to the, like, four, like, the 52-week level, and, like, the… like, the…
36 00:03:42.730 ⇒ 00:03:52.320 Michael Thorson: pre-filtered, like, timeframes that we discussed, Ashmini, so maybe we have, like, use that as our QA target, like, a known good file in warehouse, and then just, like.
37 00:03:52.320 ⇒ 00:03:52.710 Ashwini Sharma: Yeah.
38 00:03:52.710 ⇒ 00:03:57.989 Michael Thorson: Yeah, exactly. So I’ll send you that table name, so you can take a look at it, but it should have.
39 00:03:57.990 ⇒ 00:03:58.760 Ashwini Sharma: Oh, sure.
40 00:03:58.760 ⇒ 00:04:00.950 Michael Thorson: It’s 8-10 is when…
41 00:04:01.050 ⇒ 00:04:05.959 Michael Thorson: it was polled, and I have all time frames from 4 weeks up to 52 weeks.
42 00:04:06.320 ⇒ 00:04:07.780 Ashwini Sharma: Okay, alright.
43 00:04:10.170 ⇒ 00:04:23.080 Michael Thorson: Cool. Yeah, so I’ll make some notes about the filtering there, and add that to our, kind of, like, master Excel, just so you, like, know what we’re looking at there. It’s not all the data, it’s just, two subcategories, I think, so… I’ll let you know what the filters were exactly.
44 00:04:23.610 ⇒ 00:04:24.590 Ashwini Sharma: Okay, cool.
45 00:04:26.530 ⇒ 00:04:29.500 Demilade Agboola: Okay, sounds good, sounds like,
46 00:04:30.190 ⇒ 00:04:35.649 Demilade Agboola: Right now, the next step will be verifying, like, this, which would be the verification of the…
47 00:04:35.820 ⇒ 00:04:39.260 Demilade Agboola: Data itself, as well as also…
48 00:04:39.370 ⇒ 00:04:44.620 Demilade Agboola: setting up the implementation of, like, the Prefect pipeline itself, once we get the keys.
49 00:04:44.720 ⇒ 00:04:50.809 Demilade Agboola: So we’re in a good spot now, and once you are able to help us get the keys, we’ll move on to the next stage.
50 00:04:54.970 ⇒ 00:05:06.140 Demilade Agboola: So, alright, so in terms of long-running models, I have been looking around at them, and just a couple of things to note. The current refresh cadence in dbt for lack of a refresh is currently about two and a half hours.
51 00:05:07.380 ⇒ 00:05:24.149 Demilade Agboola: which isn’t the worst, I have definitely seen worse, generally speaking. And it’s fine, in the sense of you can handle ingestion, and then also, handle your dbt runs prior to, like, the business team waking up and needing any data, basically.
52 00:05:24.660 ⇒ 00:05:33.199 Demilade Agboola: I would also look… want to say, like, the ingestion runtime. I’m not sure where I’ll be able to find that, but the idea is, like, how long does it take to ingest your data sources?
53 00:05:33.420 ⇒ 00:05:40.690 Demilade Agboola: directly into Redshift, and that would also help me understand, like, the amount of time
54 00:05:40.900 ⇒ 00:05:54.839 Demilade Agboola: that, like, the entire data run takes… so, like, if the ingestion takes, like, an hour, for instance, and then dbt takes, like, two and a half hours, we know that we’re talking about two… three and a half hours, 4 hours for a total.
55 00:05:54.950 ⇒ 00:05:56.339 Demilade Agboola: Overnight job.
56 00:05:57.180 ⇒ 00:06:15.989 Mary Burke: I believe, that our ingestion runtime is about 90 minutes, and that happens every 4 hours for some data sources. And again, I could be misspeaking there if that’s just limited to the one data source that we… I was speaking about with, our current partner in particular, but,
57 00:06:16.180 ⇒ 00:06:21.389 Mary Burke: My understanding is that they were talking about the entire, entire model being 90 minutes.
58 00:06:21.710 ⇒ 00:06:34.419 Demilade Agboola: Okay. Alright, that’s fair. If you’d like me to look into that, just point me in that direction. I would also look into that and just get an idea of, like, what your night runs, like, look like, and how long it will take for…
59 00:06:34.640 ⇒ 00:06:36.990 Demilade Agboola: Like, just the entire system to run.
60 00:06:37.700 ⇒ 00:06:38.749 Demilade Agboola: What that looks like.
61 00:06:39.210 ⇒ 00:06:44.149 Demilade Agboola: But yeah, one of the things I just mentioned… okay, I could go about to say something.
62 00:06:44.480 ⇒ 00:06:46.439 Michael Thorson: I was gonna say something,
63 00:06:46.560 ⇒ 00:06:51.120 Michael Thorson: Do you have a summary of which the six main running models are? Curious.
64 00:06:51.120 ⇒ 00:06:51.650 Demilade Agboola: They’re… they’re.
65 00:06:51.650 ⇒ 00:06:52.550 Michael Thorson: Later on, alright, cool.
66 00:06:52.550 ⇒ 00:06:54.270 Demilade Agboola: Getting the slides.
67 00:06:54.440 ⇒ 00:07:02.999 Michael Thorson: Cool. Okay, cool. Yeah, because I totally… there could be an opportunity to, like, cherry-pick a couple, kind of, core models that we do want to run a little bit quicker. Yep.
68 00:07:03.150 ⇒ 00:07:04.669 Michael Thorson: Okay, cool. Thanks.
69 00:07:05.030 ⇒ 00:07:06.490 Demilade Agboola: Alright, so, like…
70 00:07:06.920 ⇒ 00:07:22.500 Demilade Agboola: like I said, with the two and a half hours, it’s not the worst, like, it does work, but the idea is scalability. As you want to keep growing and adding more business concepts into your dbt layer, you don’t want to get constrained with time, so if we can make it fast now.
71 00:07:22.690 ⇒ 00:07:34.530 Demilade Agboola: If we can get down from 24 hours to, like, an hour and a half, or maybe even an hour, if we’re being really optimistic, that gives us extra room for more models that we can add to the infrastructure as we build and scale.
72 00:07:34.770 ⇒ 00:07:36.260 Demilade Agboola: So…
73 00:07:36.850 ⇒ 00:07:44.080 Demilade Agboola: I also mentioned that minus the 6, we can also have optimization across, so, for instance, a model that runs in 5 minutes, or 6 minutes.
74 00:07:44.200 ⇒ 00:07:59.659 Demilade Agboola: might not be long-running in that sense, but if we can get it down to 2 minutes across multiple models, we can start to have, like, a knock-on effect that can create, like, a 15 minute extra or 20 minutes extra from those ones themselves. But the 6 models in particular are this.
75 00:08:00.000 ⇒ 00:08:07.970 Demilade Agboola: So, one is the manual spend daily trading revenue tax model. So, this one is the process model, takes about 44 minutes to run.
76 00:08:08.150 ⇒ 00:08:19.549 Demilade Agboola: I’ve looked into it, there’s a lot of, like, lead lag going on in there, and I feel like we can split that into smaller models. The smaller models themselves tend to take… will take less time to run than having everything
77 00:08:19.770 ⇒ 00:08:30.210 Demilade Agboola: happening at once. It’s, like, storing different computes, like… you’re basically using your RAM to, like, store different computes, and holding different bits of information until you’re finally able to do everything at once.
78 00:08:30.630 ⇒ 00:08:33.179 Demilade Agboola: And then you can make incremental, because, like.
79 00:08:33.610 ⇒ 00:08:45.589 Demilade Agboola: the base model is, like, the Google Analytics Raw Daily Events Daily, right? We don’t need to… it’s a table that we’re currently using. We don’t need to currently load the, like.
80 00:08:45.710 ⇒ 00:08:58.520 Demilade Agboola: things that happened 2 weeks ago are still the same events, right? We don’t need to load those events every single day. We can create a window for, like, the last 7 days if we feel like things change, but if we know things don’t change, we can actually just do
81 00:08:58.520 ⇒ 00:09:07.030 Demilade Agboola: Like, the last, you know, 24 hours, or just ensure that the keys that are already in the table do not get duplicated, and we can just have an incremental
82 00:09:07.030 ⇒ 00:09:15.800 Demilade Agboola: build based off that. That should save us time, so that whatever we’re doing is literally just based off the fresh data coming into the system. So that’s the…
83 00:09:16.080 ⇒ 00:09:23.670 Demilade Agboola: First model. The second model is this model, which is the Google Analytics Raw Events Daily Mat. Takes about 40 minutes to run.
84 00:09:24.140 ⇒ 00:09:27.209 Demilade Agboola: Again, part of why I suggested it gets in the catalogue.
85 00:09:27.360 ⇒ 00:09:30.750 Demilade Agboola: I’m setting that up, which is part of why that’s what the dbt theme is.
86 00:09:30.970 ⇒ 00:09:36.240 Demilade Agboola: I’m saying about for these things, because I went through, like, 5 to 10 different runs.
87 00:09:36.530 ⇒ 00:09:53.839 Demilade Agboola: And I could just see what the range was. Obviously, sometimes it runs, like, a minute or two more, a minute or two less, you know, but, like, with the catalog, you’ll be able to see the average over time, so that gives us more accurate representation, and that’s ideally what we want.
88 00:09:53.910 ⇒ 00:10:02.040 Demilade Agboola: But, yes, it takes about 40 minutes, and literally what this is, is a select on another long-running model. So…
89 00:10:02.920 ⇒ 00:10:07.589 Demilade Agboola: we’re selecting this. Do we need to select this? We can actually think of…
90 00:10:07.780 ⇒ 00:10:22.420 Demilade Agboola: if we do need to select it, because we’re like… because literally, there’s no new column being added to this. It’s literally just a selector of that. So, do we need to have it? If we don’t, we could knock this, delete this, lose the time, the 40 minutes that it currently takes?
91 00:10:22.520 ⇒ 00:10:32.629 Demilade Agboola: And if we do feel like we need to have it, we can just still make it incremental, find the unique keys, find the timestamp, event timestamp, and build on top of that, those timestamps.
92 00:10:32.850 ⇒ 00:10:36.990 Demilade Agboola: Yeah, this is the model that it builds off from.
93 00:10:37.380 ⇒ 00:10:39.549 Demilade Agboola: Takes about 20 minutes to run.
94 00:10:40.150 ⇒ 00:10:44.149 Demilade Agboola: Again, it’s also built off the Raw Events Daily model.
95 00:10:45.080 ⇒ 00:11:03.289 Demilade Agboola: And it’s also enriched with some other transaction tables and some other tables, so what we could just do here again, like I said, we just need to enrich the new events that are coming in. We don’t need to keep enriching everything all time, and have… keep going through that process. That saves us some time as well.
96 00:11:04.060 ⇒ 00:11:09.650 Demilade Agboola: This took about 20 minutes as well, so here it’s the Shopify US line items property.
97 00:11:09.910 ⇒ 00:11:28.109 Demilade Agboola: So in here, we can also look at incrementality, find what the event timestamps are, look… does it look back? Does data update over time? If it doesn’t, then we know that once it’s in our database and it becomes historic data, we don’t need to refresh it, we can just keep looking forward.
98 00:11:28.240 ⇒ 00:11:32.260 Demilade Agboola: Two is there’s a cross join in there,
99 00:11:32.420 ⇒ 00:11:40.650 Demilade Agboola: And we will just need to just look at the entire efficiency of that. As we know, like, cross-joins will explode data.
100 00:11:40.760 ⇒ 00:11:44.209 Demilade Agboola: If there’s any way we can get that
101 00:11:45.060 ⇒ 00:11:54.609 Demilade Agboola: same result or same outfields without going through the cross-train route, we should try and explore that, and that would be something we can look ahead. Again, like I said, if I had…
102 00:11:55.080 ⇒ 00:12:10.899 Demilade Agboola: write access, I will be playing around with these things and just kind of seeing what the runtimes look like, and it would have, like… I would be able to say, hey, I tried this, this, this, this is what the runtime looks like right now, versus what it was before. So that’s something we look forward to getting with, like, the dbt write access.
103 00:12:12.240 ⇒ 00:12:21.209 Demilade Agboola: Also, this as well, there isn’t really much, like, going on here, it’s a regular model, so it’s just the same old concept. We will look at the incremental build.
104 00:12:21.400 ⇒ 00:12:38.099 Demilade Agboola: Look at whether we can… if, again, it looks back into time, if it doesn’t, cool. If… if it… if it looks back into time, we would have to find out the window, so we can say, oh, only refresh things over the last, say, 7 days, or the last, you know, 14 days. Every other thing keeps static.
105 00:12:38.110 ⇒ 00:12:41.900 Demilade Agboola: So there are ways around, like, different ways we can build incrementally.
106 00:12:42.590 ⇒ 00:12:46.029 Demilade Agboola: This, again, is the Google Analytics raw.
107 00:12:46.340 ⇒ 00:12:50.050 Demilade Agboola: events daily, it’s the base model. There’s also…
108 00:12:50.260 ⇒ 00:12:57.329 Demilade Agboola: enrichments going on there. And there’s also a filtering technique happening there, so we’re filtering where…
109 00:12:57.590 ⇒ 00:13:05.799 Demilade Agboola: I can’t remember the exact name of the event, but we’re focusing on a particular event. But it only happens at the end, so maybe we can make it happen at the beginning, so we don’t have to…
110 00:13:06.140 ⇒ 00:13:18.800 Demilade Agboola: join data that we don’t eventually need as an output. So, that’s part of what I meant by looking to better filtering. So the idea is, if you have… if I have a table, this table, and I join to another table.
111 00:13:19.110 ⇒ 00:13:24.269 Demilade Agboola: And then it’s only when I get the output that I’m filtering on the condition that I need.
112 00:13:24.440 ⇒ 00:13:36.249 Demilade Agboola: perhaps we could just filter from the first. We create a CTE, or create a model, filter it out, and so we have… we’re joining on smaller values and smaller number of rows versus, you know, what we currently have there.
113 00:13:36.700 ⇒ 00:13:40.460 Demilade Agboola: So yeah, these are the 6 models,
114 00:13:40.830 ⇒ 00:13:46.260 Demilade Agboola: I don’t know if you suspected any of them, or if you have any thoughts on…
115 00:13:46.460 ⇒ 00:13:48.560 Demilade Agboola: how I’m looking to optimize them.
116 00:13:49.890 ⇒ 00:14:05.769 Mary Burke: I can’t speak to the Google Analytics pieces as much, but just thinking through that… the Business Central General Ledger Entries one, which I think was on there for about 15 minutes, I am pretty confident that does currently pull incrementally, so I’m not sure if you are…
117 00:14:05.770 ⇒ 00:14:16.820 Mary Burke: Were you able to, like, have insight into the frequency with which it is pulling, or, like, what that time frame is? And that might just be me not understanding how much access you have.
118 00:14:17.360 ⇒ 00:14:17.900 Demilade Agboola: Oh, scope.
119 00:14:17.900 ⇒ 00:14:30.249 Michael Thorson: Oh, wait, I think, to clear that up too, I think the API calls are incremental, so, like, we’re only adding new data, but I think what you’re saying, Denalon, is that the dbt is building the full table every time.
120 00:14:30.250 ⇒ 00:14:31.730 Demilade Agboola: Exactly, so it’s this…
121 00:14:31.730 ⇒ 00:14:32.340 Mary Burke: Okay.
122 00:14:32.340 ⇒ 00:14:38.589 Demilade Agboola: So, like, we can be refreshing the raw data incrementally, but excuse me one second, this is the…
123 00:14:39.460 ⇒ 00:14:41.209 Demilade Agboola: Let me find it within GitHub.
124 00:14:43.300 ⇒ 00:14:48.280 Demilade Agboola: Alright, so… This is the base table itself.
125 00:14:48.700 ⇒ 00:15:00.859 Demilade Agboola: And so what’s happening here is, you can see it’s metarized as a table, and it’s just pointing directly to the incremental… what I believe is the incremental table in the warehouse. So every day, it’s just refreshing the same table.
126 00:15:00.860 ⇒ 00:15:01.719 Mary Burke: Got it, okay.
127 00:15:01.720 ⇒ 00:15:11.029 Demilade Agboola: If we can make it incremental, we don’t need to update… you don’t need to put, the same data from yesterday, 2 weeks ago, three weeks ago, 2 months ago.
128 00:15:11.230 ⇒ 00:15:14.740 Demilade Agboola: it’s already there, we don’t need to keep doing that. So that’s where I’m looking at, like.
129 00:15:14.930 ⇒ 00:15:21.360 Demilade Agboola: Let’s try and make it, like, not a table, but instead an incremental build right there.
130 00:15:21.670 ⇒ 00:15:24.140 Mary Burke: Got it, that makes sense. Thank you for walking through that.
131 00:15:24.820 ⇒ 00:15:25.730 Demilade Agboola: Okay…
132 00:15:26.340 ⇒ 00:15:29.819 Michael Thorson: Yeah, and confirming the GA4 stuff is, like, something…
133 00:15:30.000 ⇒ 00:15:43.049 Michael Thorson: we just… we became aware of was causing some problems with runtime just recently. So, like, you’re spot on there that, like, there’s probably some optimization. Jt, I don’t know if we got that, but one of the joins was on to the daily…
134 00:15:43.220 ⇒ 00:15:49.609 Michael Thorson: spend tracking. It was, like, pretty heavily slowed down because of… GA4 joins.
135 00:15:50.000 ⇒ 00:15:51.070 Justin Tabarini: Interesting.
136 00:15:51.650 ⇒ 00:15:52.230 Michael Thorson: Yeah.
137 00:15:52.770 ⇒ 00:15:55.369 Michael Thorson: Okay. Yeah. That’s an opportunity.
138 00:15:55.370 ⇒ 00:15:57.109 Justin Tabarini: At a high frequency.
139 00:15:58.510 ⇒ 00:16:02.370 Justin Tabarini: So it’s definitely something we can… Optimize, then.
140 00:16:03.400 ⇒ 00:16:04.920 Demilade Agboola: Yeah, so…
141 00:16:05.250 ⇒ 00:16:13.949 Demilade Agboola: I have, like, so I’ve listed all the… I don’t know if, Justin saw the full presentation, but, like, I listed out the main culprits in terms of runtime.
142 00:16:14.240 ⇒ 00:16:30.239 Demilade Agboola: And what we can look at in terms of optimizing it. I would play around with it a bit more, but I don’t have right access, so it’s just basically… I’m just using, like, reading the code and seeing where, like, there are potential inefficiencies and what we can do to improve them, so that’s kind of what’s going on here.
143 00:16:30.490 ⇒ 00:16:35.859 Demilade Agboola: I will share this like I did with the performer presentation at the end of this, so, like, if you want to look at it.
144 00:16:35.950 ⇒ 00:16:55.950 Demilade Agboola: Go to the models themselves, have any, like, look through, see if there’s anything you could spot, and feel like, hey, why are we doing this? Maybe you can also apply some business context to it, and be like, I don’t think we need this, or this is unnecessary. We can actually get rid of a join, we can get rid of this model entirely, like, you know, that sort of context would also be very useful.
145 00:16:56.110 ⇒ 00:16:58.430 Demilade Agboola: For any optimization.
146 00:17:01.560 ⇒ 00:17:08.679 Demilade Agboola: Okay, so we’ve done… gone through this. And also, another thing I mentioned earlier is
147 00:17:09.300 ⇒ 00:17:12.680 Demilade Agboola: Beyond the long-running models, because again, we have a ton of models.
148 00:17:12.920 ⇒ 00:17:17.230 Demilade Agboola: There’s some models that run for, like, 6 minutes, you know.
149 00:17:17.359 ⇒ 00:17:19.559 Demilade Agboola: 5 minutes here.
150 00:17:19.569 ⇒ 00:17:38.070 Demilade Agboola: they might not be long-running in that sense, but if we can save 2 minutes from one model here, 2 minutes from another model here, 3 minutes from the model here, they stack on… it can all stack on itself, and eventually we can get, like, 15 minutes, 20 minutes, just from increment… just from finding ways to improve the efficiency across board.
151 00:17:38.210 ⇒ 00:17:41.189 Demilade Agboola: And that’s aside from the long-running models that we currently have.
152 00:17:41.910 ⇒ 00:17:47.840 Demilade Agboola: Alright, so the next steps… Is we would want to…
153 00:17:48.120 ⇒ 00:17:56.089 Demilade Agboola: complete the data audit, that’s one. So that will just be the full concept of everything we’ve been talking about so far in a…
154 00:17:56.530 ⇒ 00:18:10.340 Demilade Agboola: an audio document that’s currently in progress. I will be… it’s currently in progress, and I’ll be sharing within the team… my team internally for reviews and feedback, and then I’ll share it with the Magic Spoon team next week.
155 00:18:10.610 ⇒ 00:18:14.910 Demilade Agboola: Also, yeah, we would want to kick off the data mat.
156 00:18:15.170 ⇒ 00:18:18.290 Demilade Agboola: But obviously, we need the data in…
157 00:18:18.720 ⇒ 00:18:22.800 Demilade Agboola: we need data landed in Redshift, and we need to be able to create our perfect
158 00:18:22.950 ⇒ 00:18:28.530 Demilade Agboola: Pipeline, so that would require us getting access to
159 00:18:28.900 ⇒ 00:18:32.410 Demilade Agboola: The perfect keys and, like, being able to do that.
160 00:18:33.470 ⇒ 00:18:42.439 Demilade Agboola: And also, like, scoping for February, I’m not sure if this is something you still want, we could always take it off if that’s not something you want, but would also like to look at, like, the new connector build.
161 00:18:42.610 ⇒ 00:18:44.410 Demilade Agboola: That we can start to get to.
162 00:18:44.540 ⇒ 00:18:48.639 Demilade Agboola: If there are anyone specifically you want that are not here, please let us know.
163 00:18:49.110 ⇒ 00:18:52.790 Demilade Agboola: And the idea is we would want to be able to see how we can,
164 00:18:53.150 ⇒ 00:18:59.459 Demilade Agboola: Find opportunities for you to be able to, build out some new business insights.
165 00:19:01.220 ⇒ 00:19:17.829 Mary Burke: Got it, yeah, I think, just kind of echoing from our conversation earlier this week, for February, I think our focus is mostly going to be on the… just the overall partner transition, and making sure that we can help get you guys up to speed on our entire infrastructure, and,
166 00:19:18.160 ⇒ 00:19:27.569 Mary Burke: Just implementing some of the suggestions that we anticipate you guys will have from the overall data audit and, going through the projects that we have.
167 00:19:27.920 ⇒ 00:19:33.170 Mary Burke: in the queue, or that have been sitting in the queue with our existing partners. So I think that’ll be,
168 00:19:33.720 ⇒ 00:19:39.369 Mary Burke: probably our biggest focus for, for February, in addition to all of the MMM stuff.
169 00:19:40.030 ⇒ 00:19:40.949 Demilade Agboola: Oh, okay, alright.
170 00:19:40.950 ⇒ 00:19:49.909 Mary Burke: We will, and we’ll push on our existing partner to, as anonymously as possible, get you guys that access there. We know that that’s been a blocker, so we are…
171 00:19:50.090 ⇒ 00:19:51.960 Mary Burke: Well, we’ll poke them again about it today.
172 00:19:52.250 ⇒ 00:19:53.100 Demilade Agboola: Okay.
173 00:19:53.430 ⇒ 00:19:59.260 Demilade Agboola: I mean, I think, just generally speaking, we’ve tried to make things work with what we’ve… the access we’ve had.
174 00:20:00.160 ⇒ 00:20:09.669 Demilade Agboola: Obviously, we would like to make things work even better, so access will take us to that next level, but we do understand that listings take some time, so we’re here, and once we get it.
175 00:20:09.990 ⇒ 00:20:12.720 Demilade Agboola: We go, like, you know, we don’t really waste time with…
176 00:20:13.010 ⇒ 00:20:18.830 Demilade Agboola: with things that we need to do. So, yeah, we’ll just keep pushing this. We’ll get the data audit to you by next week.
177 00:20:18.930 ⇒ 00:20:33.319 Demilade Agboola: And just let you know, like, hey, here’s the… here are things that we believe we can make more efficient. Here are the current… the current setup, and things that we can do for, like, documentation, or, like, testing, and just ensuring that, like.
178 00:20:33.420 ⇒ 00:20:40.170 Demilade Agboola: What you get is good data as well, good, reliable data, as fast as possible.
179 00:20:42.260 ⇒ 00:20:44.619 Mary Burke: Great, we are in a line with that mission.
180 00:20:44.830 ⇒ 00:20:45.800 Demilade Agboola: Oh, okay.
181 00:20:45.800 ⇒ 00:20:47.260 Justin Tabarini: Yeah, and
182 00:20:47.620 ⇒ 00:20:54.500 Justin Tabarini: I think, talking to the, like, February stuff, and kind of, like, later this month, I think…
183 00:20:54.860 ⇒ 00:20:57.330 Justin Tabarini: You know, as we’re still working through the SPINS data.
184 00:20:57.460 ⇒ 00:21:12.659 Justin Tabarini: I hope by mid to end of next week, I’ll have a, final format that I will, like, have for the MMM data transfer to go to our partner, and I’m gonna, like, list the marts that we want to be using to find that data from.
185 00:21:14.420 ⇒ 00:21:26.679 Justin Tabarini: So, yeah, basically, summary is, I think that stuff, there’s additional data sources that we probably would want to have, but don’t have, that might be good to partner with for February. So…
186 00:21:27.200 ⇒ 00:21:27.830 Demilade Agboola: Okay.
187 00:21:28.270 ⇒ 00:21:30.449 Justin Tabarini: thinking about that stuff would be.
188 00:21:31.480 ⇒ 00:21:34.899 Justin Tabarini: I think we should focus on, yeah, MMM and partner transition.
189 00:21:36.150 ⇒ 00:21:50.149 Demilade Agboola: Alright, sounds good. Like, we’re here. If you have any change in scope, do let us know. We will, you know, be able to look at it internally, get back to you on, like, how that would look like. But, like, right now, I think we’re all in sync.
190 00:21:51.580 ⇒ 00:22:00.790 Mary Burke: Great, agreed. Yeah, I think our… our next to-dos are really pushing to get you guys that access, and then, like Michael mentioned earlier, just passing over the,
191 00:22:01.030 ⇒ 00:22:09.370 Mary Burke: that spins table that’s in the warehouse, and making sure that, Ashwini, you can QA against that, since he’s done a lot of legwork there to make sure that that matches our manual pulls.
192 00:22:09.880 ⇒ 00:22:10.510 Ashwini Sharma: Sure.
193 00:22:11.880 ⇒ 00:22:16.600 Demilade Agboola: Okay. All right then. Does anyone have any feedback or any questions about anything?
194 00:22:18.570 ⇒ 00:22:21.200 Mary Burke: No, I think, oh, go ahead, Michael.
195 00:22:21.200 ⇒ 00:22:25.750 Michael Thorson: Oh, I was just gonna say, yeah, I don’t think so, but again, feel free to tap any of those pieces.
196 00:22:25.750 ⇒ 00:22:26.140 Mary Burke: Yeah.
197 00:22:26.140 ⇒ 00:22:42.170 Michael Thorson: set up 30 minutes here and there, like, we have some bandwidth put aside to just help you out, whether that’s getting the dbt model, like, walking through the warehouse, or, like, on the pipeline spins data QA side. Just tap us, should be able to help, or find someone to help.
198 00:22:42.780 ⇒ 00:22:44.180 Demilade Agboola: Okay.
199 00:22:44.310 ⇒ 00:22:53.340 Demilade Agboola: I just wanted to say that I will potentially make the Tuesday meetings Friday meetings, if that’s fine with everyone, partly because, like.
200 00:22:53.570 ⇒ 00:22:57.889 Demilade Agboola: Tuesday… this is fri- Friday, I’m not sure if we’ll get… if we get access on Monday.
201 00:22:58.130 ⇒ 00:23:13.310 Demilade Agboola: I’m not sure if we’ll have enough to give you feedback on Tuesday, and also, like, Friday’s just the end of the week. We can just talk about what happened that week, and, like, let you know, what we did that week, what we’re looking forward to doing the following week, and that is… it creates a cycle in which we can just keep going that way.
202 00:23:14.560 ⇒ 00:23:30.950 Mary Burke: Sure, I think, in general that should work, but if we have any, any out-of-offices or anything, we can handle that over Slack. I’m just thinking, thinking ahead for next week that we might have some out-of-offices with the holiday weekend, but we’ll make sure to communicate that so we, we can get everyone, on the same call.
203 00:23:31.110 ⇒ 00:23:35.579 Demilade Agboola: Okay, sounds good. Like, if there’s a week where Friday doesn’t work, we can always make it Thursday, like, it’s not…
204 00:23:35.580 ⇒ 00:23:36.840 Mary Burke: Cool. Yeah, sounds good.
205 00:23:36.840 ⇒ 00:23:42.159 Demilade Agboola: We’re flexible in that sense. It doesn’t necessarily always have to be a Friday, especially if there might be a holiday or, you know…
206 00:23:42.530 ⇒ 00:23:46.820 Demilade Agboola: We would just want to be able to have, like, a… An end-of-week update, basically.
207 00:23:46.820 ⇒ 00:23:54.120 Mary Burke: Yeah, I’ll take a look at calendars, too, and just send you what it looks like we’ll have recurring Thursday or Friday availability.
208 00:23:54.120 ⇒ 00:24:03.579 Demilade Agboola: Alright, sounds good, so I will send them, like, daily update, like I do, just let you know what’s going on, and you can just reply with the availability that works for the team.
209 00:24:03.950 ⇒ 00:24:05.049 Mary Burke: Awesome, sounds great.
210 00:24:05.050 ⇒ 00:24:06.180 Demilade Agboola: Alright, Ben.
211 00:24:06.380 ⇒ 00:24:10.470 Demilade Agboola: If we don’t have any other questions or feedback, I guess we can call it a day.
212 00:24:11.690 ⇒ 00:24:13.139 Mary Burke: Sounds good, thank you both.
213 00:24:13.140 ⇒ 00:24:14.040 Demilade Agboola: Alright, thank you very much.
214 00:24:14.040 ⇒ 00:24:15.230 Ashwini Sharma: Thank you, thank you.