Meeting Title: Magic Spoon Biweekly sync Date: 2026-04-17 Meeting participants: Demilade Agboola, Michael Thorson, Mary Burke
WEBVTT
1 00:00:11.100 ⇒ 00:00:11.940 Michael Thorson: Hey.
2 00:00:12.550 ⇒ 00:00:13.900 Michael Thorson: How’s it going, Demi?
3 00:00:17.810 ⇒ 00:00:18.980 Demilade Agboola: Pretty good, how are you?
4 00:00:20.800 ⇒ 00:00:26.179 Michael Thorson: Solid! It’s pretty hot in New York, so I’m just baking in my apartment… for once.
5 00:00:26.530 ⇒ 00:00:28.370 Demilade Agboola: How hot, how hot are we talking?
6 00:00:28.800 ⇒ 00:00:37.270 Michael Thorson: Not that hot, I think it’s just, like, it’s not quite summer yet, so, like, it’s a bit of a surprise. I think it’s, like, 80, 75, 80 Fahrenheit today?
7 00:00:37.270 ⇒ 00:00:41.840 Demilade Agboola: Okay, that is hot for, like, April, to be honest.
8 00:00:41.840 ⇒ 00:00:42.410 Michael Thorson: that.
9 00:00:42.770 ⇒ 00:00:44.620 Michael Thorson: Not, not acclimated yet.
10 00:00:49.270 ⇒ 00:00:50.230 Demilade Agboola: Hey guys!
11 00:00:51.150 ⇒ 00:00:51.920 Michael Thorson: Hey, man.
12 00:00:54.380 ⇒ 00:00:58.660 Demilade Agboola: Alright, how’s everyone doing? Are we expecting anyone else? Like, JT…
13 00:00:59.950 ⇒ 00:01:02.950 Michael Thorson: JT can’t make it today, so it might just be us three.
14 00:01:02.950 ⇒ 00:01:05.060 Demilade Agboola: Alright, sounds good.
15 00:01:05.690 ⇒ 00:01:12.319 Demilade Agboola: I mean, I think this week, instead of, like, how we typically would have slides, and it just seems, like, slidey.
16 00:01:12.490 ⇒ 00:01:15.259 Demilade Agboola: I think I’ll make it a bit more…
17 00:01:15.520 ⇒ 00:01:21.439 Demilade Agboola: relational. So high level, this past two weeks, couple of things that we’ve been working on.
18 00:01:21.560 ⇒ 00:01:31.310 Demilade Agboola: So we’ve set up channels, like, in Slack, so we can get alerts when the Perfect loads fail.
19 00:01:31.430 ⇒ 00:01:46.020 Demilade Agboola: So in Slack, we have that channel, so now whenever, like, those, like, flows fail, we’re, like, notified, and we’re able to action it even faster, rather than, like, having to go in, Prefect, and look at it every single time.
20 00:01:46.230 ⇒ 00:01:49.450 Demilade Agboola: Appears Mary’s frozen.
21 00:01:53.060 ⇒ 00:01:53.740 Michael Thorson: Heavy?
22 00:01:53.740 ⇒ 00:01:54.660 Demilade Agboola: Yeah, I think so.
23 00:01:55.250 ⇒ 00:01:55.709 Michael Thorson: Oh, yeah.
24 00:01:55.710 ⇒ 00:02:00.630 Demilade Agboola: Yeah, I was gonna say, I love that she’s very still.
25 00:02:02.120 ⇒ 00:02:03.529 Demilade Agboola: Should we give her a second?
26 00:02:03.930 ⇒ 00:02:07.210 Michael Thorson: Maybe she can hear. We can keep cruising.
27 00:02:07.210 ⇒ 00:02:08.419 Demilade Agboola: Oh, okay, alright then.
28 00:02:09.190 ⇒ 00:02:17.740 Demilade Agboola: So that was something we’ve done. Obviously, with regards to that, there have been some maintenance work on, like, flows that have broken.
29 00:02:17.870 ⇒ 00:02:20.280 Demilade Agboola: As well, over the last, like, 2 weeks.
30 00:02:20.480 ⇒ 00:02:24.190 Demilade Agboola: We’ve done some work on the, like.
31 00:02:24.940 ⇒ 00:02:30.290 Demilade Agboola: JT’s stuff regarding, like, loading in spend for…
32 00:02:30.750 ⇒ 00:02:33.360 Demilade Agboola: including in Spain for, like, the MMA Marts.
33 00:02:35.530 ⇒ 00:02:51.349 Demilade Agboola: I noticed now that… I’ve noticed that, like, the prod run is failing, but that’s because it’s referring to old… the old name, and Ashwini has changed the name to Marketing Insights, so I’m going to… and I know… I see that has been merged as well, so this should fix that on the next run.
34 00:02:52.050 ⇒ 00:03:04.060 Demilade Agboola: And ultimately, we’ve also been able to start working on, like, the April spend load. I noticed when he said he thought he had done it, but apparently there was some issues with that.
35 00:03:04.170 ⇒ 00:03:09.430 Demilade Agboola: And I’ve talked to him about it, and they said that we should figure out a way to…
36 00:03:09.570 ⇒ 00:03:10.579 Demilade Agboola: Flag it.
37 00:03:11.180 ⇒ 00:03:13.780 Demilade Agboola: Basically…
38 00:03:13.980 ⇒ 00:03:30.299 Demilade Agboola: He expected it to have loaded, the job ran, but it didn’t fully load, and so for that, we will be just setting up, like, alerts, just so that we’re fully aware, and can fully, like, verify that the numbers are in there as expected.
39 00:03:31.880 ⇒ 00:03:35.699 Mary Burke: Sorry, guys, my computer just restart, so sorry I missed that.
40 00:03:35.700 ⇒ 00:03:36.160 Demilade Agboola: I don’t know what happened
41 00:03:36.810 ⇒ 00:03:47.440 Demilade Agboola: Yeah, I literally just was looking at the screen, and I was like, I think Mary’s frozen, because either that, or you were very still, like…
42 00:03:47.440 ⇒ 00:03:50.630 Mary Burke: Nope, nope, I’m back. My bad.
43 00:03:51.380 ⇒ 00:04:04.320 Demilade Agboola: So just high-level recap, basically I just walked through the things we’ve done over the last couple weeks. So, we created Slack, a Slack channel where we’ve been able to hone down all the failed jobs to only the important ones that Michael flagged.
44 00:04:04.320 ⇒ 00:04:11.529 Demilade Agboola: And so, on that Slack channel, when any of those reports, many of those flows, perfect flows, fail, we immediately see it.
45 00:04:11.530 ⇒ 00:04:13.750 Demilade Agboola: And we’re able to action it much faster.
46 00:04:13.760 ⇒ 00:04:15.970 Demilade Agboola: We’ve also been able to,
47 00:04:16.130 ⇒ 00:04:18.950 Demilade Agboola: do stuff around, like, MMM, because JT asked.
48 00:04:19.490 ⇒ 00:04:30.370 Demilade Agboola: loading some spend, we’ve done that as well. As well as, you know, just obviously tackling when things… when the perfect flows fail.
49 00:04:30.780 ⇒ 00:04:33.890 Demilade Agboola: And I was just also mentioning to,
50 00:04:34.050 ⇒ 00:04:37.750 Demilade Agboola: Michael, just now, that we’ve been working on the April span loading.
51 00:04:37.930 ⇒ 00:04:48.939 Demilade Agboola: But there were some issues with that, and that’s what Ashwin is currently working on now, so the idea is, before close of week, we can load that in and have that available, in the table.
52 00:04:50.000 ⇒ 00:05:01.700 Michael Thorson: Yeah, I’d love to see a root cause for that as well, because I did see the prefect ran, as intended last week. Okay. So, like, the script to run, but I don’t understand why…
53 00:05:01.980 ⇒ 00:05:05.240 Michael Thorson: It didn’t pull the most recent spins.
54 00:05:05.770 ⇒ 00:05:12.739 Michael Thorson: like, delivery, so I would love to get just, like, a root cause so we can understand this pipeline a little bit more, and just, like, understand the levers behind the scenes.
55 00:05:12.740 ⇒ 00:05:18.229 Demilade Agboola: Okay. Alright, so what I’ll do is… so Ashwini’s currently working on the fixture as we speak.
56 00:05:18.390 ⇒ 00:05:34.299 Demilade Agboola: Once he’s done with the fix, I would ask him to, like, update us on the fix, as well as also be able to, just give us, like, a proper root cause analysis of what went wrong, and how we are mitigating it in future instances, because
57 00:05:34.890 ⇒ 00:05:40.629 Demilade Agboola: I mean, it’s great to be able to figure out why it went wrong now, but obviously, in the future, we want to be one step ahead of it all.
58 00:05:41.400 ⇒ 00:05:57.040 Michael Thorson: Yeah, that’s fine, and it just… I’m… I haven’t taken a look at his code for, like, what he’s… like, how he’s calling the Spins API, so… just trying to, like, understand it before digging in and reviewing it more directly. Okay. Cool. Yeah, that’d be… that’d be helpful on the data engineering side.
59 00:05:57.670 ⇒ 00:05:58.280 Demilade Agboola: Yeah.
60 00:05:58.280 ⇒ 00:06:04.399 Mary Burke: And then, Michael, I just have a question. I know that we, asked Ugo from Spins to make sure that we, like.
61 00:06:04.970 ⇒ 00:06:13.689 Mary Burke: just have the capacity to ping the API more times. I don’t know if that was just for the initial load, or if that’s steady state. I don’t know if that comes into play at all.
62 00:06:14.130 ⇒ 00:06:16.529 Michael Thorson: Oh, it’s just for the backfill. I think we should be okay.
63 00:06:16.690 ⇒ 00:06:17.260 Mary Burke: Okay.
64 00:06:19.100 ⇒ 00:06:38.200 Michael Thorson: Yeah, and then, Demi, are you up to speed on what the state of the, like, current Market Insights table will look like? I just wanted to, like, make sure I’m understanding, like, exactly what’s being dumped into that, data table. Do you know, do you know, kind of, the parameters of…
65 00:06:38.230 ⇒ 00:06:41.149 Michael Thorson: What’s being loaded into that, like, one…
66 00:06:41.150 ⇒ 00:06:44.410 Demilade Agboola: So I don’t know the full parameters in terms of, like.
67 00:06:44.690 ⇒ 00:07:02.079 Demilade Agboola: columns and what the filters are, but I do know that we’re trying to make it the source, so instead of having, like, the, like, March, you know, table, market site underscore March, blah blah blah, we’re gonna make this the primary table in which we can start to run the transformations against and spins can load into.
68 00:07:02.710 ⇒ 00:07:05.079 Michael Thorson: Hmm. And then, in terms of…
69 00:07:06.440 ⇒ 00:07:19.269 Michael Thorson: because there’s two major, like, timeframes that we’re looking at, or there’s two kind of categories. You know, we’re moving from, like, one-week time frame up to 52-week time frame. Yeah. For that new end date, so…
70 00:07:19.430 ⇒ 00:07:24.559 Michael Thorson: Just curious if that’s all being loaded correctly, because…
71 00:07:25.600 ⇒ 00:07:29.080 Michael Thorson: the… they’re slightly different polls, I guess. It’s like the…
72 00:07:29.760 ⇒ 00:07:40.509 Michael Thorson: at least the way I understand it is, like, there’s the higher time frames, like 4 weeks, 12 weeks, etc, which are all backed off of the SPIN’s data relief date, so the end date will all be kind of, like.
73 00:07:40.930 ⇒ 00:07:44.480 Michael Thorson: The same, or, like, the next incremental month?
74 00:07:44.610 ⇒ 00:07:45.400 Demilade Agboola: Okay.
75 00:07:45.590 ⇒ 00:07:50.840 Michael Thorson: But the… the other… the one-week timeframe should be, like, a continuous…
76 00:07:51.160 ⇒ 00:07:52.170 Demilade Agboola: Yeah.
77 00:07:52.170 ⇒ 00:07:56.170 Michael Thorson: model. Are you seeing that when you’re working in the data?
78 00:07:56.720 ⇒ 00:08:02.060 Demilade Agboola: So when I’m working in data, I am seeing that for, what we have so far.
79 00:08:02.270 ⇒ 00:08:05.240 Demilade Agboola: I will need to confirm that on, like.
80 00:08:05.380 ⇒ 00:08:09.720 Demilade Agboola: the newer, like, magic insights table.
81 00:08:10.050 ⇒ 00:08:10.410 Michael Thorson: Yeah.
82 00:08:10.410 ⇒ 00:08:16.080 Demilade Agboola: And so I’ll let you know if that also lines up with what our expectations are.
83 00:08:16.690 ⇒ 00:08:23.109 Michael Thorson: Yeah, that’s what I really want to understand, market, and I can, like, pull that up as well. I just want to understand, like, because…
84 00:08:23.380 ⇒ 00:08:30.400 Michael Thorson: when I was looking at the API documentation, there was an API for, like, Market Insights, which is, like, the higher timeframes, like, pre-aggregated.
85 00:08:30.400 ⇒ 00:08:31.010 Demilade Agboola: Okay.
86 00:08:31.010 ⇒ 00:08:34.179 Michael Thorson: And then there was, like, a trended…
87 00:08:34.630 ⇒ 00:08:42.199 Michael Thorson: trended API endpoint, to, to, like, to pull any kind of, like, endpoint or, like, end date.
88 00:08:42.460 ⇒ 00:08:57.009 Michael Thorson: Which is, like, probably more effective for the one-week timeframe, but, like, open to whatever, like, whatever was successful on Ashwini’s side, I just want to, like, close the loop there to make sure we’re pulling a continuous, like, one-week dataset, and then also pulling, like, a monthly
89 00:08:57.490 ⇒ 00:08:59.870 Michael Thorson: Of the aggregate timeframes.
90 00:08:59.870 ⇒ 00:09:17.190 Demilade Agboola: Okay, yeah, it sounds… sounds good. I feel like I’ll definitely get back to Ashwini, and we’ll get back to you on this, just so we’re able to clarify, like, everything, and allay any, like, worries that… or fears that my… anyone might have about the data. Yeah.
91 00:09:17.960 ⇒ 00:09:27.569 Michael Thorson: Yeah, that’s fine by me, and yeah, just report back exactly what’s in that Market Insights table, like, what’s kind of down there. And then I was also curious, like, in terms of…
92 00:09:28.030 ⇒ 00:09:34.809 Michael Thorson: Incremental modeling there, that I do want to understand, like.
93 00:09:36.190 ⇒ 00:09:45.639 Michael Thorson: how that’s being updated. So for example, if we want to see, like, the one week of sales in, like, March 1st.
94 00:09:45.960 ⇒ 00:10:04.969 Michael Thorson: There’s obviously… that data will probably exist in, like, the most recent poll, and the poll in March, for example. So, I just, like, can you double check, too, if he’s doing that incremental, like, deduping, or if you are in the model? I just want to make sure we’re filtering out that model as close to base as possible, so we’re not, like, duplicate reporting.
95 00:10:04.970 ⇒ 00:10:05.970 Michael Thorson: The subject did.
96 00:10:06.330 ⇒ 00:10:13.910 Demilade Agboola: Yeah, definitely, we won’t want to, like, inflate, numbers by double counting, so we’ll def… I’ll look into that.
97 00:10:14.260 ⇒ 00:10:19.750 Demilade Agboola: Both from what I see in the data, as well as, what…
98 00:10:20.760 ⇒ 00:10:26.159 Demilade Agboola: Ashwini is doing in Prefect, so that allows us to be able to have, like, some level of…
99 00:10:26.540 ⇒ 00:10:30.119 Demilade Agboola: certainty in the values that we’re sending into Omni.
100 00:10:33.070 ⇒ 00:10:44.699 Michael Thorson: Yeah, that sounds… okay, that sounds good, and would love to see that, just, like, via Slack, or if y’all want to set up a walkthrough, like, code dev session to, like, walk through it together, if you’re seeing any, like, weird anomalies, or…
101 00:10:44.760 ⇒ 00:10:58.550 Michael Thorson: If it… if it is kind of, like, nuanced. Just want to get up to speed, and just make sure, like, we… like, JT and I are, like, as in the weeds on the model as possible, since we’re starting to work on our end as well, in, like, Omni setting these up.
102 00:10:59.250 ⇒ 00:11:04.389 Demilade Agboola: Alright, sounds good. What work are you doing in Omni in regards, like, this being theater so far?
103 00:11:07.140 ⇒ 00:11:10.659 Michael Thorson: We just stood up some, like, base… some, like.
104 00:11:11.540 ⇒ 00:11:14.260 Michael Thorson: Base views, like, nothing crazy.
105 00:11:14.260 ⇒ 00:11:14.859 Demilade Agboola: Oh, okay.
106 00:11:15.190 ⇒ 00:11:16.829 Michael Thorson: Nothing to report back on, so…
107 00:11:17.330 ⇒ 00:11:28.109 Demilade Agboola: Alright, yeah, so just, like, always, like, as always, just let us know what’s going on there, and if there’s anything we can do to, improve that process, or improve the quality of data, or, like…
108 00:11:28.730 ⇒ 00:11:38.379 Demilade Agboola: Any transformations we can do to make that data even more useful than it currently is, just let us know. We’re always available to work through that and support you in that way possible.
109 00:11:38.910 ⇒ 00:11:56.930 Michael Thorson: Yeah, the problem right now is the data that we’re seeing is not what we expected, and there’s this kind of misalignment of what we thought was in warehouse versus what actually is. So the first step is, like, that’s why I wanted to tie off on exactly what is being dumped into the Market Insights raw table. Okay.
110 00:11:56.930 ⇒ 00:12:01.399 Michael Thorson: The second piece of feedback, which I think you fixed in the most recent merge, was…
111 00:12:01.720 ⇒ 00:12:06.289 Michael Thorson: The initial push, this is, like, more of a flick of feedback, is…
112 00:12:06.410 ⇒ 00:12:22.540 Michael Thorson: Yeah, none of the variable types were cast, so this is throwing a ton of issues. Like, by the time data was trying, like, read in Omni, all the variables were assumed as, like, var car… var cars, like, just, like, strings, basically, so…
113 00:12:22.690 ⇒ 00:12:28.240 Michael Thorson: I think just wanted to, like, double down on that, like, the point of, like, our data stack, too, is…
114 00:12:28.240 ⇒ 00:12:45.940 Michael Thorson: base table is, like, the raw source of truth data should be, like, cleaned and, like, unfiltered, so that’s kind of, like, the intent that we’re looking for. All data should be, like, forced cast into, like, the expected outcomes. Okay. So, like, things looked like they were working on the second pass.
115 00:12:46.130 ⇒ 00:12:53.250 Michael Thorson: As you, like, submitted the merge, I haven’t actually, like, imported a lot of that data into Omni, but just want to, like, clear that up, like…
116 00:12:53.250 ⇒ 00:12:53.680 Demilade Agboola: Yeah.
117 00:12:53.680 ⇒ 00:12:58.610 Michael Thorson: It’s super important to have, like, a just consistent,
118 00:12:59.370 ⇒ 00:13:12.650 Michael Thorson: workflow in our existing data models, so if you could just, like, kind of, like, whenever you’re building a new base table, or, like, building marts, just, like, kind of parallel what we’ve done in the past for redevelop purposes, that would be super, super helpful.
119 00:13:12.880 ⇒ 00:13:22.949 Demilade Agboola: Gotcha. Yeah, feedback noted, we’ll take that on point. So normally, yeah, we do typecast that, and yeah, just kind of…
120 00:13:23.660 ⇒ 00:13:39.470 Demilade Agboola: it just kind of slipped by, in the sense of, like, because I was trying to just get it out, and trying to, like, see that it worked. I got the raw tables, tested it, it worked, and… but, like, obviously, like, Omni, had its issues with it, so, like, yeah, just going forward.
121 00:13:39.500 ⇒ 00:13:48.140 Demilade Agboola: super strict on, like, typecasting in the base layer, ensuring that, like, no columns will be Google and cast. So…
122 00:13:48.530 ⇒ 00:13:51.079 Demilade Agboola: That would be the new standard in that regard.
123 00:13:52.850 ⇒ 00:13:55.390 Michael Thorson: And then… oh yeah, sorry, you were saying?
124 00:13:55.390 ⇒ 00:13:59.080 Demilade Agboola: Yeah, and also, like, for, like, Spins data, given that,
125 00:14:01.440 ⇒ 00:14:06.729 Demilade Agboola: the data has, like, a similar, like, structure. We’ve created, like, a macro forecasting.
126 00:14:06.880 ⇒ 00:14:22.249 Demilade Agboola: So now you can just kind of use the macro, and it will cast in the right and appropriate type, and so if we need to make any changes as well to casting, just also let me know, and I can go into the macro and make that change, and it will apply across all the, like, spin models at once, so…
127 00:14:23.870 ⇒ 00:14:41.280 Michael Thorson: Cool. Yeah, that… I saw the macro, really liked that. I think the first, like, push, or the first, like, pull request was no casting, so just, like, it spent… like, I took another hour to kind of figure out what was going on, which is just, like, yeah, just, like, a note for the future, as we’re, like, moving towards…
128 00:14:41.510 ⇒ 00:14:44.079 Michael Thorson: More of a relationship there. And then,
129 00:14:44.230 ⇒ 00:14:49.890 Michael Thorson: One thing that I also discussed with JT, and this is net new to me, is we were just discussing
130 00:14:50.040 ⇒ 00:14:54.290 Michael Thorson: spins… Modeling architecture.
131 00:14:54.290 ⇒ 00:14:54.810 Demilade Agboola: Okay.
132 00:14:54.810 ⇒ 00:15:00.460 Michael Thorson: And I think… This is… this is… this might be annoying, but it’s…
133 00:15:00.610 ⇒ 00:15:12.900 Michael Thorson: I think we kind of want to move, actually, like, all your filtering that, like, I think this is on me, like, I was saying, like, oh, like, filter at the base model. Yeah. So filter by, like, the product level and all that jazz.
134 00:15:12.900 ⇒ 00:15:13.330 Demilade Agboola: Yeah.
135 00:15:13.330 ⇒ 00:15:21.390 Michael Thorson: I think what we would prefer to do, actually, is have one base table that’s just all, like, Spins Market Insights.
136 00:15:21.620 ⇒ 00:15:23.589 Michael Thorson: No filtering whatsoever.
137 00:15:23.590 ⇒ 00:15:23.930 Demilade Agboola: Okay.
138 00:15:23.930 ⇒ 00:15:42.909 Michael Thorson: And… and just, like, well, yeah, just have that kind of, like, base table with cleaned and classed variables as our only spins base table. And then, actually, I would love to move the, like, the work you’ve done, actually, to the… the output schema. So it’s, like, outputs, spins.
139 00:15:43.020 ⇒ 00:15:52.629 Michael Thorson: And do all the filtering for those models actually in the marts themselves. Would we be able to move that downstream a bit, or would you recommend against that?
140 00:15:53.430 ⇒ 00:16:12.770 Demilade Agboola: I think it depends on what you want to do. I think that’s… I was actually going to ask, like, what… how do you want to utilize it? If you’re gonna use the tables as they are, like, the base tables as they are, like, and you just need to have it in the outspace model for, like, consistency purposes, that’s fine.
141 00:16:13.040 ⇒ 00:16:29.299 Demilade Agboola: But if you want to be able to utilize them, like, downstream, so you want to make, like, certain aggregations, you want to join them certain tables, I would suggest having them as they are as base tables, because now it’s base of the different set of splits.
142 00:16:29.600 ⇒ 00:16:40.780 Demilade Agboola: And now, if you’re going to be joining, like, different splits to different sources, or different splits to different, like, tables, it’s much easier to make those, like, data manipulations downstream.
143 00:16:41.340 ⇒ 00:16:47.840 Demilade Agboola: And also, subsequently, if you do need any of those tables,
144 00:16:48.200 ⇒ 00:16:51.879 Demilade Agboola: Downstream, it’s much easier to be able to utilize, like.
145 00:16:52.300 ⇒ 00:16:55.079 Demilade Agboola: A version of it that already exists.
146 00:16:55.280 ⇒ 00:16:55.810 Demilade Agboola: Oh.
147 00:16:55.810 ⇒ 00:16:56.410 Michael Thorson: Hmm.
148 00:16:56.410 ⇒ 00:17:04.890 Demilade Agboola: And you can just say, hey, this is the table I’m trying to join to this, or to Shopify data, or to whatever data, and utilize that there.
149 00:17:06.170 ⇒ 00:17:06.589 Michael Thorson: Hmm.
150 00:17:06.599 ⇒ 00:17:13.499 Demilade Agboola: I think, also, what we could also do is we can also have, like, a base layer of, like, the entire clean spins data.
151 00:17:13.589 ⇒ 00:17:30.539 Demilade Agboola: try and have, like, an intermediate model of, like, the different, like, models that exist as they are now in some intermediate form, and that still allows us to utilize it in the output model, like, combined with, like, Shopify data, or we can also just have it
152 00:17:30.759 ⇒ 00:17:31.589 Demilade Agboola: You know…
153 00:17:32.079 ⇒ 00:17:48.449 Demilade Agboola: pushed forward as that. So it just… it depends on what the use case is, and I think that’s the most important thing. If they’re just going to be used as they are, like, your first suggestion works perfectly, like, there’s no point doing too much. Just have the base table, and then filter it down, and utilize it in Omni.
154 00:17:48.609 ⇒ 00:17:58.049 Demilade Agboola: But if we need to do way more, or if they’re, like, aggregations and cuts will like to do, and joints will like to do, yeah, I would suggest that we factor that in.
155 00:17:58.209 ⇒ 00:18:00.739 Demilade Agboola: Before just moving that to the output layer.
156 00:18:01.930 ⇒ 00:18:06.589 Michael Thorson: Yeah, that’s… that’s a good call-out, and I think… I think I agree with you, and I think there’s…
157 00:18:06.910 ⇒ 00:18:16.019 Michael Thorson: Primary, our user use cases will be, to your point, filtered down pretty tightly before we do those joins, so maybe you’re right, we keep exactly what you’ve done to date.
158 00:18:16.110 ⇒ 00:18:29.470 Michael Thorson: And I think my… my ask, then, is we do… we also create kind of, like, a mark… like a spins base, just, like, clean table. Okay. So when we do need to, like, utilize that, we can.
159 00:18:29.540 ⇒ 00:18:34.750 Demilade Agboola: Okay, and what would the use case… what would the use case of, like, a full spin space be? Like…
160 00:18:35.620 ⇒ 00:18:50.470 Michael Thorson: just… just to… to have a table that’s in the warehouse, that everything’s clean, so as JT and I are also doing development, we can kind of use that as our jumping-off point for future development. And, like, mostly for, like, yeah, like, internal development and tests, so…
161 00:18:50.650 ⇒ 00:18:51.950 Demilade Agboola: Okay. Alright.
162 00:18:51.950 ⇒ 00:18:52.330 Michael Thorson: Yep.
163 00:18:52.330 ⇒ 00:19:08.820 Demilade Agboola: Alright, sounds good then. Okay, so I will make that as well. So basically, just so we’re clear, that’s basically everything, so all the different weeks, like, levels of time granularity in one table updated as the new data comes in, right?
164 00:19:08.820 ⇒ 00:19:12.370 Michael Thorson: Just whatever the, whatever’s in the Market Insights table.
165 00:19:12.550 ⇒ 00:19:17.790 Michael Thorson: and then, like, variable casting to make sure that’s pretty quick. And then the one…
166 00:19:18.160 ⇒ 00:19:25.759 Michael Thorson: The one thing is the, probably add a window function, just to, ensure we’re referencing the most recent,
167 00:19:26.110 ⇒ 00:19:27.950 Michael Thorson: Data, data point.
168 00:19:27.950 ⇒ 00:19:28.940 Demilade Agboola: Alright, gotcha.
169 00:19:28.940 ⇒ 00:19:31.309 Michael Thorson: So we’re not having, like, duplicating data, exactly.
170 00:19:31.310 ⇒ 00:19:36.619 Demilade Agboola: Yeah, yeah, yeah, basically like a dedupe to the latest possible, or the most recent possible data.
171 00:19:37.060 ⇒ 00:19:53.380 Michael Thorson: Yeah, that’ll be really grounding. I think JTC and I, especially since the data’s in kind of a strange state, like, we just want to make sure we’re, like, looking, because we have, like, 6 models now, it’s like, I just want to have one that’s kind of, like, we’re leaning into for, like, validation and test, and then we’re like, okay, now the other 6 models will be good to go.
172 00:19:53.630 ⇒ 00:19:57.559 Demilade Agboola: Alright, sounds good. So, I will definitely work on that model,
173 00:19:57.740 ⇒ 00:20:07.389 Demilade Agboola: and look at the Windows functions for that. I would also look at the other models, and if they also need Windows functions, so, like, we can dedupe that as well.
174 00:20:07.720 ⇒ 00:20:08.160 Michael Thorson: Cool.
175 00:20:08.160 ⇒ 00:20:10.150 Demilade Agboola: And then I’ll make those changes.
176 00:20:10.910 ⇒ 00:20:15.000 Michael Thorson: Amazing. Yeah, that would be super, super helpful. Cool. That’s… that’s…
177 00:20:15.420 ⇒ 00:20:20.259 Michael Thorson: From the Market Insights spins conversation, I guess. Thanks for…
178 00:20:20.470 ⇒ 00:20:25.999 Michael Thorson: Opening that up. Did you have any questions there, Mary? Sorry, I just kinda took the reins for a second.
179 00:20:26.330 ⇒ 00:20:38.299 Mary Burke: No, I don’t think so, but I haven’t really, like, played around with that data, but I think the logic of having just everything in their base to build off of for our team is… that makes sense to me.
180 00:20:38.650 ⇒ 00:20:41.709 Mary Burke: But I trust you, you’ve been closer to it.
181 00:20:43.400 ⇒ 00:20:47.970 Demilade Agboola: Yeah, that is a good call-out, and we’ll definitely get that in.
182 00:20:48.090 ⇒ 00:20:53.450 Demilade Agboola: Just curious, does anyone have any, like, feedback on the, engagement so far? Like.
183 00:20:53.680 ⇒ 00:20:57.639 Demilade Agboola: Does anyone have any things we’d rather, like, us focus on, or…
184 00:20:57.940 ⇒ 00:21:02.189 Demilade Agboola: Potentially, just things that could be handled differently, you know, that sort of thing.
185 00:21:03.940 ⇒ 00:21:05.360 Michael Thorson: Yeah, I think…
186 00:21:05.930 ⇒ 00:21:20.359 Michael Thorson: The spins, I think we’re just, like, kind of in the, like, in the iteration loop for the spins, because, like, JTC and I are the only people actually in the data. Okay. So him and I both independently got in and, like, stood up those models in Omni this week.
187 00:21:20.360 ⇒ 00:21:29.599 Michael Thorson: That’s where we uncovered a lot of the casting issues, and just, like, some of the, like, modeling issues, and then also noticed the lack of data from the most recent run.
188 00:21:29.600 ⇒ 00:21:30.210 Demilade Agboola: Okay.
189 00:21:30.210 ⇒ 00:21:40.870 Michael Thorson: Yeah, those are, like, the top priorities, is just, like, one, just making… well, the casting should be done, and then two, making sure, like, the most recent load is in, so we can QA, correctly.
190 00:21:40.870 ⇒ 00:21:41.850 Demilade Agboola: Okay.
191 00:21:41.850 ⇒ 00:21:43.030 Michael Thorson: Yeah.
192 00:21:43.030 ⇒ 00:21:47.160 Demilade Agboola: I was looking, on the, like, most recent run, what dates…
193 00:21:47.460 ⇒ 00:21:54.830 Demilade Agboola: do we get the most recent runs? Are they hard-coded? Are they, like… is there a timeline we can use so that we can just kind of…
194 00:21:54.950 ⇒ 00:22:07.979 Demilade Agboola: Have those checks in places, so that we can say, hey, if it’s 48 hours after the expected run, and it hasn’t loaded into the… we haven’t loaded the new data, we need to flag it and go into it.
195 00:22:08.660 ⇒ 00:22:13.570 Michael Thorson: Yeah, sure. There’s, in that main, spins,
196 00:22:14.340 ⇒ 00:22:23.640 Michael Thorson: In our main, kind of like, spins working document, there’s a data release schedule, which I’ve shared with Adweeney, and I think he built his…
197 00:22:24.070 ⇒ 00:22:30.759 Michael Thorson: Cadence, like, his, prefect orchestration cadence in alignment, but if you want to be aware of that.
198 00:22:30.760 ⇒ 00:22:36.470 Demilade Agboola: I think I’ve seen it. It’s the data release schedule, ETL cadence.
199 00:22:36.790 ⇒ 00:22:37.989 Michael Thorson: Yep, that’s the one.
200 00:22:37.990 ⇒ 00:22:39.040 Demilade Agboola: Okay, gotcha.
201 00:22:39.780 ⇒ 00:22:49.599 Michael Thorson: So the most recent release was 4.6, and that’s the one that, like, we’re still not seeing in Warehouse, but there was… there was a run, so that’s why I’m a little confused.
202 00:22:49.600 ⇒ 00:22:51.570 Demilade Agboola: Okay, alright, so…
203 00:22:51.680 ⇒ 00:23:00.450 Demilade Agboola: I will personally just try and get on top of this, so we have, like, a flag, so I think, obviously, it takes time for the runs to, like, reflect. The runs do take
204 00:23:00.700 ⇒ 00:23:02.000 Demilade Agboola: Quite a bit of time.
205 00:23:02.140 ⇒ 00:23:11.279 Demilade Agboola: So potentially what I’ll just do is we’ll have a flag, so that if it’s, you know, 48 hours after the proposed release date.
206 00:23:11.410 ⇒ 00:23:27.149 Demilade Agboola: and it’s still not loaded in the data, we will have to just kind of, like, go into that. It will flag it, we have a flag, and we’ll manually go in there. So that way, we’re always kind of on top of it, and we don’t need to, like, be responding or reacting to that difference.
207 00:23:28.440 ⇒ 00:23:43.190 Michael Thorson: Yeah, that’s huge. That’s definitely number one from our side. And then number two is honestly, like, standing up that, since we’re still in the validation phase, that spins deduped base model. Yeah.
208 00:23:43.190 ⇒ 00:23:43.800 Demilade Agboola: Okay.
209 00:23:44.750 ⇒ 00:23:56.570 Michael Thorson: Yeah, those are, like, our top priorities, so that, like, your models will be, like, we’re confident that, like, whatever’s in warehouse is finally, like, good to go, and then we can help with the validation as we’re building, kind of, our Omnibus.
210 00:23:56.800 ⇒ 00:23:58.090 Demilade Agboola: Alright, sounds good.
211 00:23:59.160 ⇒ 00:24:02.599 Demilade Agboola: I know Mero’s about to say something, I’m not sure.
212 00:24:05.260 ⇒ 00:24:15.569 Mary Burke: No, I was gonna say that I… I don’t have a ton of, feedback. I feel like things have been moving pretty smoothly. I know we’ve had some things come up with backfills and the MMM stuff,
213 00:24:15.570 ⇒ 00:24:15.910 Demilade Agboola: Yeah.
214 00:24:15.910 ⇒ 00:24:19.410 Mary Burke: I think you guys have been responsive and on top of it, but if there’s ever any…
215 00:24:19.520 ⇒ 00:24:28.120 Mary Burke: confusion about priorities or what’s, in the queue for this week. I think we can cover it on these meetings or just over Slack, too, to make sure everyone’s on the same page.
216 00:24:30.290 ⇒ 00:24:35.670 Demilade Agboola: Okay, alright, in that case, I…
217 00:24:35.980 ⇒ 00:24:46.699 Demilade Agboola: We’ll ensure that we continue moving at same pace and same cadence, but also just, like, tighten up certain, like, loose ends and ensure that those things are moved out the way.
218 00:24:48.160 ⇒ 00:24:59.089 Demilade Agboola: If at any point in time, Mary, Michael, you have any, like, questions about stuff, please feel free to message me, and just let me, like, know what’s going on, and I’ll just be on top of it, okay?
219 00:24:59.600 ⇒ 00:25:00.090 Michael Thorson: Cool.
220 00:25:00.090 ⇒ 00:25:00.660 Demilade Agboola: Alright.
221 00:25:00.660 ⇒ 00:25:01.450 Mary Burke: Awesome.
222 00:25:03.170 ⇒ 00:25:04.050 Demilade Agboola: Alright.
223 00:25:04.610 ⇒ 00:25:11.439 Demilade Agboola: In that case, I think if we don’t have anything else, we can, you know, get, like, 25 minutes back or 20 minutes back.
224 00:25:13.360 ⇒ 00:25:17.189 Michael Thorson: Okay. Oh, I had one more question from JT.
225 00:25:17.190 ⇒ 00:25:17.680 Demilade Agboola: Okay.
226 00:25:17.680 ⇒ 00:25:25.880 Michael Thorson: For context, like, we were talking yesterday, and he was working on the… Mmm.
227 00:25:26.310 ⇒ 00:25:28.120 Michael Thorson: Mart, that you helped, I think…
228 00:25:28.530 ⇒ 00:25:36.650 Michael Thorson: I’m just, like, making sure I understand the workflow. It sounds like you seeded some files into the warehouse, and then, like, updated the MMM data package.
229 00:25:36.830 ⇒ 00:25:39.669 Michael Thorson: Just… that we were, like, gonna send over to our partner.
230 00:25:39.790 ⇒ 00:25:42.260 Michael Thorson: Was that this week, or was that last week?
231 00:25:42.450 ⇒ 00:25:45.679 Demilade Agboola: A bit of both. I did some for last week, I did some this week.
232 00:25:45.820 ⇒ 00:26:00.329 Demilade Agboola: Cool. Based off of, like, different spends, so some were, like, linear spend, programmatic spend. I think this week was, like, off-site spend. I might have mixed that up, but, like, yeah, effectively, there were different spends I had to work on over the last, like, 2 weeks.
233 00:26:01.170 ⇒ 00:26:04.310 Michael Thorson: Sweet. And then his question was…
234 00:26:04.430 ⇒ 00:26:14.489 Michael Thorson: Just wanted to clear up, like, what happened. I think when the… when you, like… I think the most recent update there, like, maybe broke the production…
235 00:26:14.760 ⇒ 00:26:19.239 Michael Thorson: like, the dbt production model. Just wanted to understand, like, what happened with that.
236 00:26:19.240 ⇒ 00:26:28.570 Demilade Agboola: Yeah, so the… the most recent update didn’t really actually break the production model, because Ashwini changed the name of the base models.
237 00:26:28.570 ⇒ 00:26:29.590 Michael Thorson: to…
238 00:26:29.590 ⇒ 00:26:31.120 Demilade Agboola: Marketing Insight.
239 00:26:31.350 ⇒ 00:26:35.059 Demilade Agboola: And so that change from Marketing Insights
240 00:26:35.380 ⇒ 00:26:39.199 Demilade Agboola: the previous name, 09, blah blah blah.
241 00:26:39.610 ⇒ 00:26:57.529 Demilade Agboola: so it’s trying to refer to the old models, like, the old model names, but that name doesn’t currently exist anymore. So that’s why I had to do a PR today, where I changed the… what it’s referring to, to the, like, just Marketing Insights. So now, because that exists as a model in
242 00:26:57.820 ⇒ 00:27:05.339 Demilade Agboola: the warehouse, it will be able to refer to it, and the subsequent runs should be fine. But, like, I’m also looking at it right here.
243 00:27:05.650 ⇒ 00:27:11.629 Demilade Agboola: And another thing I was looking at, like, long-term is we’ll also need to start a…
244 00:27:11.760 ⇒ 00:27:26.550 Demilade Agboola: work on getting the model, like, the runs take about 3 hours, potentially, if we can start to, like, chop that down bit by bit, some 30 minutes here, some 20 minutes here, some 5 mains here, like, they start to…
245 00:27:26.800 ⇒ 00:27:31.290 Demilade Agboola: Get the runs down to a time that’s, like, 2 hours, maybe even under.
246 00:27:33.680 ⇒ 00:27:47.309 Demilade Agboola: But yes, that would… that’s effectively why the run broke, and in just terms of, like, being able to be on top of it, I believe the next run should be fine, but I’ll also be looking out for that, and I’ll let you know on the channel that it’s fine.
247 00:27:47.960 ⇒ 00:27:53.320 Michael Thorson: Yeah, and maybe just to ground us, too, I think this is what I meant to bring up.
248 00:27:53.540 ⇒ 00:28:00.559 Michael Thorson: Which is not pulling from the SPINS model, it’s pulling from the linear spin data, which I assume is a seed file.
249 00:28:00.560 ⇒ 00:28:01.260 Demilade Agboola: Yes.
250 00:28:02.050 ⇒ 00:28:02.670 Michael Thorson: So…
251 00:28:02.880 ⇒ 00:28:13.810 Michael Thorson: It looks like JTC had to go in here yesterday and make a change from… it looks like the model was referring to spend underscore date field, which broke… which, like, broke the actual run.
252 00:28:14.160 ⇒ 00:28:14.550 Demilade Agboola: Oh, no.
253 00:28:14.960 ⇒ 00:28:19.319 Michael Thorson: Correct reference was capital date in the new seed file.
254 00:28:19.320 ⇒ 00:28:19.710 Demilade Agboola: Okay.
255 00:28:19.710 ⇒ 00:28:26.480 Michael Thorson: So, did you, like, I guess he saw this because it threw an error during the production run?
256 00:28:26.600 ⇒ 00:28:27.160 Demilade Agboola: Okay.
257 00:28:27.160 ⇒ 00:28:35.289 Michael Thorson: And I was curious, too, are you running, like, before you merge changes, are you able to run tests locally just for that specific…
258 00:28:35.440 ⇒ 00:28:40.360 Michael Thorson: data lineage? Or you… is your… what’s your workflow there, I guess?
259 00:28:40.360 ⇒ 00:28:45.900 Demilade Agboola: Yeah, so workflow that is, I run it, like, locally, so I have the, what’s it called?
260 00:28:46.440 ⇒ 00:28:52.680 Demilade Agboola: the local branch on my computer, so I can, like, run it, get feedback on it.
261 00:28:52.800 ⇒ 00:28:56.160 Demilade Agboola: Surprised?
262 00:28:56.310 ⇒ 00:28:58.460 Demilade Agboola: That didn’t do an error.
263 00:28:58.880 ⇒ 00:29:04.299 Demilade Agboola: So, I will look into that. I have no idea what I think right now.
264 00:29:04.750 ⇒ 00:29:07.810 Demilade Agboola: Okay.
265 00:29:08.530 ⇒ 00:29:16.419 Demilade Agboola: I’ll look into that. I know one of the things we’re also looking at doing is setting up, like, CICD as well, so prior to…
266 00:29:16.690 ⇒ 00:29:22.639 Demilade Agboola: any merge. Like, you can kind of see the effect the merge will have on production as well, so…
267 00:29:22.910 ⇒ 00:29:25.289 Demilade Agboola: That would also help catching things like this.
268 00:29:26.740 ⇒ 00:29:39.059 Michael Thorson: Cool. Yeah, I mean, that was… that was his open question. Like, he pushed his change yesterday, and he was like, can you just check if Fishweenie, or if, Demi is, running, like, testing the full build before merging?
269 00:29:39.370 ⇒ 00:29:43.519 Michael Thorson: So, cool. Maybe it’s, yeah, it could just be a versioning thing, so…
270 00:29:43.520 ⇒ 00:29:44.140 Demilade Agboola: Yeah.
271 00:29:44.530 ⇒ 00:29:47.400 Michael Thorson: Be careful. I guess it’s not comment.
272 00:29:48.190 ⇒ 00:29:54.550 Demilade Agboola: Alright, sounds good. I will definitely look into that, and I will ensure, like, next week, Ashwini does, like, the CI-CD.
273 00:29:54.780 ⇒ 00:30:00.600 Demilade Agboola: Even if it’s just for, like, the models that have been modified, so at least we can kind of get an idea of…
274 00:30:00.930 ⇒ 00:30:03.830 Demilade Agboola: If any of the changes are going to be breaking changes.
275 00:30:04.850 ⇒ 00:30:11.670 Michael Thorson: Cool. Yeah, that was… that was an interesting one, because I was like, I was… I was like, I’m sure Demi saw this break locally, but then maybe there was some…
276 00:30:12.010 ⇒ 00:30:16.880 Michael Thorson: gap between your reference and what JT was seeing on his end, but…
277 00:30:16.880 ⇒ 00:30:17.400 Demilade Agboola: Hmm.
278 00:30:18.000 ⇒ 00:30:22.229 Demilade Agboola: you know, we’ll tighten that loop as well. We’ll ensure we’ll tighten that loop as well.
279 00:30:22.670 ⇒ 00:30:26.719 Michael Thorson: Yeah, cool. Yeah, thanks for just taking a moment on that and being open.
280 00:30:26.720 ⇒ 00:30:27.790 Demilade Agboola: Pleasure.
281 00:30:27.940 ⇒ 00:30:29.049 Demilade Agboola: Sounds good.
282 00:30:31.960 ⇒ 00:30:36.230 Michael Thorson: Any other questions for us? Need any support?
283 00:30:36.580 ⇒ 00:30:44.409 Demilade Agboola: Right now, no, I think just general would just be, like, just ensuring that we tighten up things in terms of, like, the spins data.
284 00:30:44.640 ⇒ 00:30:50.380 Demilade Agboola: And also, just getting back to you on… Like…
285 00:30:51.460 ⇒ 00:30:56.080 Demilade Agboola: What steps were taken to, like, number one, tighten up, like,
286 00:30:56.340 ⇒ 00:31:07.479 Demilade Agboola: the spins data, the issues with prod, in terms of, like, the runs, so feedback on if the runs are working properly now, as well as just, like, anything around
287 00:31:07.730 ⇒ 00:31:11.580 Demilade Agboola: Basically… V…
288 00:31:12.430 ⇒ 00:31:18.550 Demilade Agboola: how we’re gonna load the data in Perfect, through… and then also, like, finally, the… trying to remember…
289 00:31:19.130 ⇒ 00:31:22.010 Demilade Agboola: Something’s slipping… something has to the top of my mind.
290 00:31:22.350 ⇒ 00:31:23.610 Demilade Agboola: Give me one second.
291 00:31:25.550 ⇒ 00:31:39.539 Demilade Agboola: Oh, yes, the CICD. Yeah, I know that’s something I was thinking of. So yes, we also want to think of, like, the CICD, and just how we plan to be able to bridge that gap between, like, all pull requests and, like, change… changes that go into production as well, so…
292 00:31:40.120 ⇒ 00:31:40.730 Michael Thorson: Yep.
293 00:31:41.130 ⇒ 00:31:56.559 Michael Thorson: Cool. Yeah, and I think my final request is, just since JT was unavailable today, and we weren’t looking over a deck to share, would you be able to just send the kind of priorities and goals for this upcoming week to the shared Slack channel?
294 00:31:56.980 ⇒ 00:31:58.330 Demilade Agboola: Okay, alright.
295 00:31:58.610 ⇒ 00:31:59.940 Demilade Agboola: Rolder, roller.
296 00:32:00.750 ⇒ 00:32:02.980 Michael Thorson: Great. Yeah, that would be super helpful.
297 00:32:03.930 ⇒ 00:32:04.929 Demilade Agboola: Alright, sounds good.
298 00:32:08.050 ⇒ 00:32:08.640 Michael Thorson: Cool.
299 00:32:09.570 ⇒ 00:32:10.890 Michael Thorson: All good from my end.
300 00:32:11.180 ⇒ 00:32:11.620 Demilade Agboola: Alright.
301 00:32:11.620 ⇒ 00:32:12.960 Mary Burke: Awesome. Thanks, Demi.
302 00:32:12.960 ⇒ 00:32:14.019 Demilade Agboola: Thank you, bye.
303 00:32:14.710 ⇒ 00:32:15.970 Michael Thorson: Bye, guys. Have a good weekend.