Meeting Title: Magic Spoon Biweekly sync Date: 2026-05-01 Meeting participants: Demilade Agboola, Mary Burke, Michael Thorson
WEBVTT
1 00:00:41.040 ⇒ 00:00:42.530 Mary Burke: Hey, Demi, how are you?
2 00:00:44.230 ⇒ 00:00:47.540 Demilade Agboola: Hi, Mary. Good, how are you? Hi, Michael.
3 00:00:47.540 ⇒ 00:00:48.500 Mary Burke: And thanks.
4 00:00:49.080 ⇒ 00:00:52.070 Demilade Agboola: That’s good to hear, that’s good to hear. How’s everyone doing?
5 00:00:54.070 ⇒ 00:00:54.990 Mary Burke: Hanging in.
6 00:00:55.360 ⇒ 00:01:00.150 Demilade Agboola: Nice, nice, nice. Are we waiting for JT, or should we get started?
7 00:01:01.690 ⇒ 00:01:06.390 Michael Thorson: We can get started. I got his side covered. We’ve been working closely this week.
8 00:01:07.290 ⇒ 00:01:10.800 Demilade Agboola: Gotcha. Alright, so let’s get to it.
9 00:01:14.700 ⇒ 00:01:16.480 Demilade Agboola: Okay, so can we see my screen?
10 00:01:18.770 ⇒ 00:01:19.690 Michael Thorson: Yep.
11 00:01:20.300 ⇒ 00:01:29.939 Demilade Agboola: Alright, so just quick overview. I think just generally, Prefect and dbt are in good shape overall. I know there are…
12 00:01:30.850 ⇒ 00:01:40.100 Demilade Agboola: there’s currently some issue with the general ledger flow, and I know I wish… I’m sorry, Ashrini put details up about that in the
13 00:01:40.490 ⇒ 00:01:41.470 Demilade Agboola: Channel?
14 00:01:41.840 ⇒ 00:01:48.880 Demilade Agboola: So, like, the investigation is in progress, and we’ll get to the bottom of that. But just as a general…
15 00:01:49.100 ⇒ 00:01:55.619 Demilade Agboola: Roll, like, things are good, which is kind of, like, ensuring that everything remains on par.
16 00:01:56.610 ⇒ 00:02:04.559 Demilade Agboola: And in terms of, like, things we’d like to just go through, just, like, observability and, like, the business central, the warehouse and modeling.
17 00:02:04.910 ⇒ 00:02:09.989 Demilade Agboola: dbt runtime, and then just, like, the engineering hours over the last, like, 2 weeks.
18 00:02:11.210 ⇒ 00:02:16.060 Demilade Agboola: So, in terms of… Observability.
19 00:02:16.210 ⇒ 00:02:21.970 Demilade Agboola: I think just general… Generally, we’ve been able to see that we’ve… put up…
20 00:02:23.140 ⇒ 00:02:29.569 Demilade Agboola: Our alerting channel, so we’re able to see, like, critical failures or failures that are happening with ingestion.
21 00:02:29.940 ⇒ 00:02:40.199 Demilade Agboola: And that allows us to have, like, that one spot where we can monitor things. I think in terms of things that have failed, we have things around the Business Central, like sales and voice.
22 00:02:40.350 ⇒ 00:02:49.709 Demilade Agboola: And the… the central… business central general ledger entries, as well as some other, like, flows here and there.
23 00:02:49.840 ⇒ 00:02:54.230 Demilade Agboola: But generally, we’ve been able to just, like, do, like, lookbacks,
24 00:02:54.510 ⇒ 00:02:58.129 Demilade Agboola: Load the data, ensure that those data fixes, like, those
25 00:02:58.300 ⇒ 00:03:01.849 Demilade Agboola: Gaps, do not create any data gaps within our system.
26 00:03:02.440 ⇒ 00:03:06.550 Demilade Agboola: And, like I said, with the, general ledger.
27 00:03:06.850 ⇒ 00:03:14.730 Demilade Agboola: We are kind of investigating that flow as well, but just generally, yeah, we’ve been able to fix the errors that have come up for the last two weeks.
28 00:03:15.200 ⇒ 00:03:20.520 Demilade Agboola: And just, like, the next steps is we’ll, like, just, like, classify the retries and, like, real failures.
29 00:03:20.710 ⇒ 00:03:30.760 Demilade Agboola: As well as just, like, shorten any noisy cancel cycles. So, like, figure out, like, why they’re canceling, and just, like, how do we ensure that that doesn’t happen again.
30 00:03:30.940 ⇒ 00:03:33.959 Demilade Agboola: And just, like, continue, like…
31 00:03:34.330 ⇒ 00:03:38.460 Demilade Agboola: handling, especially, like, Business Central seems to be, like, the main problem point.
32 00:03:38.780 ⇒ 00:03:46.920 Demilade Agboola: How do we ensure that that flow remains stable, or gets more stable, with, like, a clearer plan of action whenever they fail?
33 00:03:48.660 ⇒ 00:03:49.330 Michael Thorson: Yeah.
34 00:03:49.480 ⇒ 00:03:50.680 Demilade Agboola: That’ll be at 4.
35 00:03:50.890 ⇒ 00:03:51.800 Demilade Agboola: Any gesture?
36 00:03:52.510 ⇒ 00:04:01.619 Michael Thorson: I have some good context. I was doing some re… like, when we were having issues, I was, like, in the model a lot, and I realized, actually, that, like.
37 00:04:02.100 ⇒ 00:04:07.290 Michael Thorson: We… have you noticed that there’s, sales invoices V1 and V2?
38 00:04:07.290 ⇒ 00:04:07.940 Demilade Agboola: Yes.
39 00:04:08.500 ⇒ 00:04:10.839 Michael Thorson: So, V1 is the one that’s failing, right?
40 00:04:10.840 ⇒ 00:04:11.790 Demilade Agboola: Yep.
41 00:04:12.130 ⇒ 00:04:28.879 Michael Thorson: So, yeah, this is business context. I don’t… I don’t… I think we talked, kind of, about this high level, but now is probably a good time to, like, bring this up as, like, more of a working project, where we did build a purpose-built API connection for some key tables, so if you see a V2 in, like, the dbt model.
42 00:04:28.940 ⇒ 00:04:47.510 Michael Thorson: that’s pulling from a purpose-built API, so it’s not Business Central’s standard API, because this issue was happening historically. Our previous data engineers, like, built it, and we started to test and kind of, like, validate that the data’s flowing in correctly.
43 00:04:47.560 ⇒ 00:04:56.859 Michael Thorson: What I will say is, like, it’s… it’s… I think it was looking pretty good. We didn’t do an all-in, like, field-to-field check, because, like, some…
44 00:04:56.860 ⇒ 00:05:04.420 Mary Burke: They don’t have… yeah, not all the fields in the purpose-built API were available in V1, which is a little bit,
45 00:05:04.610 ⇒ 00:05:07.449 Mary Burke: Not a disaster, but a little bit of an issue.
46 00:05:07.670 ⇒ 00:05:08.430 Demilade Agboola: Oh, okay.
47 00:05:09.040 ⇒ 00:05:18.469 Mary Burke: And we didn’t switch over every table that is available through this API, just we were focusing on the top priorities, but it will be worthwhile to switch over everything.
48 00:05:19.490 ⇒ 00:05:36.700 Michael Thorson: Yeah, just maybe, like, noting that as, like, that’s a project, like, because instead of, like, dumping time into, like, understanding these critical failures, maybe we actually slot in some key tables in the next month as, like, a data modeling project, just start to swap the field names, in the backend model.
49 00:05:36.860 ⇒ 00:05:42.829 Michael Thorson: How do you feel about that, Mary, though? It could be, like, with all the failures, I was like, this just might be the right time.
50 00:05:43.740 ⇒ 00:05:45.419 Mary Burke: Yeah, I think that makes sense.
51 00:05:45.980 ⇒ 00:05:46.589 Michael Thorson: We’re up.
52 00:05:47.070 ⇒ 00:05:51.310 Demilade Agboola: Yeah, I think we can add that into, like, the roadmap, in terms of, like.
53 00:05:51.780 ⇒ 00:06:00.259 Demilade Agboola: we’ll look at the… what’s currently available, and where we can replace the V1 models with the V2 models, and see…
54 00:06:00.720 ⇒ 00:06:08.979 Demilade Agboola: that. Just also, do we have an idea of the cadence that’s just looking perfect, the cadence at which that works, and,
55 00:06:09.770 ⇒ 00:06:14.600 Demilade Agboola: If there have been any errors in terms of, like, freshness on its own.
56 00:06:15.000 ⇒ 00:06:16.349 Demilade Agboola: I feel like…
57 00:06:16.350 ⇒ 00:06:16.880 Michael Thorson: in the V2.
58 00:06:17.020 ⇒ 00:06:35.569 Demilade Agboola: Sounds like we’re more comfortable with it, but I just want us to… I mean, I’ll look into it and just be sure that, you know, there aren’t any, like, lapses in data quality on that end as well, and then if not, we can just, like, swap them out into where, like, there are similar columns, and use that as a replacement of what we currently have.
59 00:06:36.290 ⇒ 00:06:46.000 Michael Thorson: Yeah, and I was gonna say, I think maybe the first step of that project is just kind of, like, doing a revalidation of, like, the… if we’re confident in the V1, just doing, like, a, like.
60 00:06:46.010 ⇒ 00:07:03.299 Michael Thorson: some quality checks and comparing before we make the switch that, you know, like, some amount, like, some quantity across all time, and then, like, certain periods, and just, like, just a general super, super high-level aggregation to be like, okay, did we lose rows? Like, did we, like, is V2 not as,
61 00:07:03.300 ⇒ 00:07:19.489 Michael Thorson: consistent, or is there a breakpoint anywhere in the V2? So we might have to have kind of, like, a validation period, like, a quick high level, just like, is this accurate? And then two, is the actual, like, switching or the handholding to switch the models over to point to V2.
62 00:07:19.910 ⇒ 00:07:22.530 Demilade Agboola: Gotcha, yes, that, that sounds…
63 00:07:23.100 ⇒ 00:07:27.759 Demilade Agboola: Just sounds about right, yeah, just being able to have that validation and use that would be very important.
64 00:07:28.730 ⇒ 00:07:34.349 Michael Thorson: Sweet. Or we could even, like, parallel the other ideas, we could even flip the…
65 00:07:34.830 ⇒ 00:07:38.080 Michael Thorson: We could probably build a new mart.
66 00:07:38.240 ⇒ 00:07:45.919 Michael Thorson: as, like, the V2 mart, you know, and then actually just switch what Omni’s looking at, as long as the field names are, like, one-to-one. That’d probably be, like, the…
67 00:07:46.140 ⇒ 00:07:49.580 Michael Thorson: Cleanest way, in case we need to roll back, you know?
68 00:07:49.800 ⇒ 00:08:00.019 Demilade Agboola: Yeah, I mean, potentially. I think if you do it within DBT, especially if they’re already, like, the flow is already clear, and what, like, V1 is currently going through.
69 00:08:00.660 ⇒ 00:08:01.100 Michael Thorson: Mmm.
70 00:08:01.100 ⇒ 00:08:05.789 Demilade Agboola: then have, like, V2 columns, replace what the V1 columns are.
71 00:08:06.180 ⇒ 00:08:15.639 Demilade Agboola: And so, like, Omni would have that, like, without having to make any major changes in Omni, the new data would just kind of flow in to Omni.
72 00:08:17.300 ⇒ 00:08:18.570 Michael Thorson: Yeah, that makes sense.
73 00:08:21.350 ⇒ 00:08:25.329 Demilade Agboola: So in terms of, like, warehouse and modeling.
74 00:08:28.120 ⇒ 00:08:37.020 Demilade Agboola: So I know, like, I noticed that, Michael and JT have been working on, like, the insights, tables…
75 00:08:37.390 ⇒ 00:08:40.940 Demilade Agboola: So, that was pushed, like, 2 days ago, 3 days ago?
76 00:08:41.350 ⇒ 00:08:49.500 Demilade Agboola: I noticed that there were some, like, naming discrepancies, so that has been fixed. So now the ones are working well.
77 00:08:49.720 ⇒ 00:08:52.810 Demilade Agboola: In terms of, like, just, like, ongoing validation.
78 00:08:53.160 ⇒ 00:08:56.379 Demilade Agboola: Just so that, like, whenever, like, the retail release…
79 00:08:56.500 ⇒ 00:09:06.179 Demilade Agboola: comes out, like, the Spanish data comes out, we’re ensuring that we are always on top of it, and then any, like, where data is, we’re able to find it.
80 00:09:06.310 ⇒ 00:09:10.000 Demilade Agboola: And part of what we’ve done to enable that is the freshness check.
81 00:09:10.150 ⇒ 00:09:14.869 Demilade Agboola: So, based off the retail release date, like the Spain’s release date.
82 00:09:15.050 ⇒ 00:09:25.339 Demilade Agboola: We’ve been able to create a test in dbt such that once, 48 hours have exceeded the release date, and the new data isn’t reflected in there.
83 00:09:25.650 ⇒ 00:09:45.489 Demilade Agboola: we’re able to have that test fail, and we’re able to then know that our spins ingestion didn’t work as it should. So, that’s just, like, part of what we’re just trying to integrate in there to ensure that, we’re always on top of things, and we always can automi- like, automatically know when things are, like, not going the way they should.
84 00:09:46.520 ⇒ 00:09:52.389 Demilade Agboola: Also, part of the fix that was made,
85 00:09:52.590 ⇒ 00:09:54.959 Demilade Agboola: Is that we’ve been able to…
86 00:09:55.220 ⇒ 00:09:58.450 Demilade Agboola: create, so what was happening with the error that I had to fix?
87 00:09:58.610 ⇒ 00:10:05.770 Demilade Agboola: was that… We had… Things for… we had macros for the raw data.
88 00:10:06.490 ⇒ 00:10:12.809 Demilade Agboola: But because the models that were being built were not being built on the raw data, but on, like, the,
89 00:10:14.060 ⇒ 00:10:26.190 Demilade Agboola: based… already transformed data, so the… the model… the column names were not the same anymore, so now there is now a new col… there’s now a new macro that we can use to refer to
90 00:10:26.600 ⇒ 00:10:27.290 Demilade Agboola: the…
91 00:10:27.510 ⇒ 00:10:34.149 Demilade Agboola: The raw data, as well as the base model data, so that that way, we can use them as we please.
92 00:10:34.300 ⇒ 00:10:37.479 Demilade Agboola: I will not run into, like, errors, like, further downstream.
93 00:10:39.900 ⇒ 00:10:50.859 Demilade Agboola: Yeah, so that’s kind of that. And then, yeah, there’s been also, like, introduction of a seed. Like, oh yeah, the seed column rename in prod,
94 00:10:51.380 ⇒ 00:10:55.500 Demilade Agboola: So what happened there was, like, the model that was pushed into production.
95 00:10:55.670 ⇒ 00:11:03.539 Demilade Agboola: didn’t necessarily match… the seed I was pushed into production didn’t necessarily match the seed that was on my local device, so that was.
96 00:11:03.540 ⇒ 00:11:04.140 Michael Thorson: Okay.
97 00:11:05.150 ⇒ 00:11:10.699 Demilade Agboola: you know, so imagine, like, on Pradio, it’s called spend underscore date, but on my local device, it’s date.
98 00:11:10.970 ⇒ 00:11:21.800 Demilade Agboola: So, like, when I was validating on my local device, it was running, but in product, it wasn’t running, so that fix has been made, and the… I replaced the seed on my local device, so everything matches as well.
99 00:11:21.920 ⇒ 00:11:26.059 Demilade Agboola: And I think, like, just in terms of next steps with that, it’s just, like.
100 00:11:26.170 ⇒ 00:11:34.509 Demilade Agboola: Number one is one to, like, just continue to maintain, like, the clean spin space, and ensure that, like, the later screen is as it should be.
101 00:11:34.650 ⇒ 00:11:40.179 Demilade Agboola: Next step will just be watching out for the next release window, and just ensuring that,
102 00:11:40.840 ⇒ 00:11:47.930 Demilade Agboola: Any failures that occur in ingestion, we automatically get to know about it, and we’re on top of things.
103 00:11:48.040 ⇒ 00:11:54.599 Demilade Agboola: And just, like, any potential, like, dbt co-build, if we have any model… modeling details that we want to, like.
104 00:11:54.770 ⇒ 00:12:02.719 Demilade Agboola: Go through, or figure out, or if there are any questions about how we want to, like, expand our current, like, modeling base.
105 00:12:03.420 ⇒ 00:12:11.260 Demilade Agboola: And just, like, just tighter… let’s just tighter reviews on, like, names, seeds, schemas, and just things around that nature.
106 00:12:14.350 ⇒ 00:12:22.740 Michael Thorson: Yeah, I mean, it’s par for the course, like, that’s kind of how I feel like we see things, like, yeah, there’s gonna be, like, I’ve made the same…
107 00:12:23.400 ⇒ 00:12:27.840 Michael Thorson: seed. The same seed issue has happened to me many times, so…
108 00:12:28.200 ⇒ 00:12:40.049 Michael Thorson: Yeah, in terms of just, like, the volume of seed files we’re sending into the warehouse, like, the number of people, I think it’s okay that we’re kind of running into those issues. Sorry for any, like, inefficiencies that caused you, though, that is annoying, so…
109 00:12:40.050 ⇒ 00:12:44.460 Demilade Agboola: Yeah, it’s all good, it’s all good. We’re just…
110 00:12:44.890 ⇒ 00:12:47.979 Demilade Agboola: kind of, like, leads us to the next point that I was actually thinking.
111 00:12:47.980 ⇒ 00:12:48.650 Michael Thorson: you know.
112 00:12:49.190 ⇒ 00:12:53.530 Demilade Agboola: So, a couple of things in terms of, like, dbt runtime.
113 00:12:54.250 ⇒ 00:12:59.490 Demilade Agboola: So, like I mentioned in the… Slack channel?
114 00:13:00.120 ⇒ 00:13:09.480 Demilade Agboola: Our DBT runs are getting close to 4 hours. It’s not exactly 4 yet, it’s like 3 hours 45 minutes, 3 hours 50 minutes, sometimes 3 hours 30 minutes.
115 00:13:09.700 ⇒ 00:13:13.700 Demilade Agboola: But we’re inching there, it’s kind of, like, the general trend.
116 00:13:13.980 ⇒ 00:13:22.499 Demilade Agboola: And so I’m just trying to flag it so that if we can get ahead of it, so we can say, hey, instead of having, like, a 4-hour schedule.
117 00:13:23.090 ⇒ 00:13:25.729 Demilade Agboola: 44 hour, run.
118 00:13:26.260 ⇒ 00:13:33.350 Demilade Agboola: Where, over time, we can start to have this effect, where, like, a run is still in progress, and then another run kicks off.
119 00:13:33.450 ⇒ 00:13:35.710 Demilade Agboola: And then we start having this, like.
120 00:13:36.480 ⇒ 00:13:44.650 Demilade Agboola: things start to append, because, you know, it starts to form a queue, and I don’t think that’s the most efficient way for us to have things running, so…
121 00:13:44.940 ⇒ 00:13:52.490 Demilade Agboola: what I was, like, thinking of, I know, like I kind of mentioned, is just, like, optimizing certain models. So, again, if there are models that take, like, 40 minutes.
122 00:13:52.760 ⇒ 00:13:59.100 Demilade Agboola: 45 minutes, getting it down to 20 minutes, will start to make, like, a difference in how the overall runtime
123 00:13:59.320 ⇒ 00:14:06.130 Demilade Agboola: looks like. But I know, like, Michael has suggested that, you know, we can have some as overnight refreshes.
124 00:14:06.680 ⇒ 00:14:10.939 Demilade Agboola: Versus those that run during the day at a regular sync.
125 00:14:11.250 ⇒ 00:14:19.259 Demilade Agboola: We can also do that as well. It will just be a function of, like, being able to identify what are the overnight refreshes versus what are the
126 00:14:19.410 ⇒ 00:14:20.160 Demilade Agboola: you know.
127 00:14:20.390 ⇒ 00:14:23.379 Demilade Agboola: Every flower refreshes that like we need to do.
128 00:14:25.020 ⇒ 00:14:30.930 Demilade Agboola: And then, another thing, in terms of, like, right now, is just, like, we haven’t set up a CICD.
129 00:14:31.540 ⇒ 00:14:35.869 Demilade Agboola: And so, like, a CICD would just basically be every time we set up a PR,
130 00:14:36.330 ⇒ 00:14:42.200 Demilade Agboola: It will do a run and let us know if there are any breaking changes.
131 00:14:42.500 ⇒ 00:14:51.619 Demilade Agboola: In the PR, so we can easily say, hey, actually, no, this PR will break what we have in production, and that allows us to be a bit,
132 00:14:52.920 ⇒ 00:14:56.969 Demilade Agboola: more proactive rather than reactive with some of our job failures.
133 00:14:57.490 ⇒ 00:15:02.120 Demilade Agboola: Part of what we’ll need to do is, because of the volume of prod.
134 00:15:02.430 ⇒ 00:15:12.950 Demilade Agboola: We will need to have, like, samples of… instead of having the full scale of data that we have in broad, so we’ll have sample data… sample data sizes.
135 00:15:13.080 ⇒ 00:15:19.320 Demilade Agboola: present in the CICD environment, and so that way, we can have it run
136 00:15:19.460 ⇒ 00:15:29.560 Demilade Agboola: On smaller… a smaller volume of data, rather than, you know, the entire, like, volume of data that we have right now, which could take hours in some cases.
137 00:15:30.180 ⇒ 00:15:34.919 Demilade Agboola: And I’m also thinking about how we can integrate that logic also into dev.
138 00:15:35.330 ⇒ 00:15:46.319 Demilade Agboola: Because I’ve also noticed that dev takes a really long time, especially if you’re dealing with, like, high-level models, like your output models. If you’re trying to run the entire lineage, it can take a really long time for it to run.
139 00:15:46.690 ⇒ 00:15:50.839 Demilade Agboola: And so having that ability to… to be able to flag
140 00:15:50.970 ⇒ 00:15:55.889 Demilade Agboola: Any issues that are occurring before the week hits prod would also be very helpful.
141 00:15:56.240 ⇒ 00:16:00.390 Demilade Agboola: So in terms of, like, next steps, yeah, we should try and chip away at, like, the time.
142 00:16:00.560 ⇒ 00:16:07.830 Demilade Agboola: And just have, like, concurrency guardrails. It’s just as well as, like, figuring out, like, standing up the dbt-CI CD.
143 00:16:08.500 ⇒ 00:16:09.070 Michael Thorson: Yep.
144 00:16:09.410 ⇒ 00:16:12.769 Demilade Agboola: So we can start to preview the impact on the production environment.
145 00:16:13.900 ⇒ 00:16:27.959 Michael Thorson: For sure. Aligned with the, runtime concurrency guardrails, couple solutions there is, like, the actual optimization or transitioning certain tables to incremental. Another kind of quick and…
146 00:16:28.220 ⇒ 00:16:46.959 Michael Thorson: easy fix, I guess, is, like, happy to, like, if we just, like, prepare a list of, like, what we actually need to rebuild during the daytime, and then what models are actually, like, lower priority, which can, like, wait a full day. It could be, like, the quickest and kind of dirtiest, and honestly, like, a good structure long-term to actually, like, have
147 00:16:46.980 ⇒ 00:17:00.960 Michael Thorson: daytime, like, like, you know, every 3-hour runs, and then also have just, like, a batch that runs in the middle of the night for less important tables. Because, like, a lot of the tables actually are not very used in Omni, so…
148 00:17:00.980 ⇒ 00:17:17.580 Michael Thorson: Yeah, if you want to pull just, like, honestly, a CS… or, like, an Excel sheet or Google Sheet of, like, all the model names, I can just go tag, like, what can be overnight, as one opportunity. I think that would, like, that’s kind of how we work in Business Central as well, like, there’s certain kind of, like, high-priority jobs.
149 00:17:17.589 ⇒ 00:17:17.949 Demilade Agboola: Okay.
150 00:17:17.950 ⇒ 00:17:24.320 Michael Thorson: Mostly related to, like, D2C, Shopify, like, active stream data, and then there’s a lot of, kind of, batch-based
151 00:17:24.520 ⇒ 00:17:25.990 Michael Thorson: data.
152 00:17:26.170 ⇒ 00:17:28.270 Michael Thorson: Pipelines as well, so…
153 00:17:28.480 ⇒ 00:17:29.630 Demilade Agboola: And…
154 00:17:29.630 ⇒ 00:17:34.120 Michael Thorson: Cool. Yeah, so would that be helpful, or would you prefer a different strategy?
155 00:17:34.650 ⇒ 00:17:41.170 Demilade Agboola: Yeah, I think what we could do is I will send you a list of all outfuse models that I can see within dbt.
156 00:17:41.470 ⇒ 00:17:41.820 Michael Thorson: Yep.
157 00:17:41.820 ⇒ 00:17:48.590 Demilade Agboola: Mark which outfit models you want, like, running frequently, which ones you want running, like, overnight.
158 00:17:48.750 ⇒ 00:17:51.520 Demilade Agboola: And then, that way, I can start to tag
159 00:17:52.570 ⇒ 00:17:57.570 Demilade Agboola: the MARP models as an overnight slash business hours tag.
160 00:17:57.730 ⇒ 00:18:04.389 Demilade Agboola: And so what I will do is, the business model tags, or the business hour tags, I can…
161 00:18:05.890 ⇒ 00:18:11.689 Demilade Agboola: them would be in the lineage. Everything upstream of that, as well as the models themselves.
162 00:18:11.910 ⇒ 00:18:12.370 Michael Thorson: Oh, no.
163 00:18:12.760 ⇒ 00:18:18.290 Demilade Agboola: Every, like, 4 hours during business hours, and then we’ll have an overnight refresh.
164 00:18:19.430 ⇒ 00:18:24.259 Michael Thorson: Nice. That sounds great. And then, as far as the tags go in dbt,
165 00:18:24.370 ⇒ 00:18:29.140 Michael Thorson: Is… this is more like a down-the-road roadmap idea, but if…
166 00:18:29.700 ⇒ 00:18:48.460 Michael Thorson: Can you trigger, like, via API to, like, dbt Cloud, can you trigger, like, like, all, models with a specific tag from, like, API? Like, if we tag every model to do with, like, pacing, for example, which is, like, a report flow, could we trigger that with, like, an API call?
167 00:18:48.620 ⇒ 00:18:55.059 Demilade Agboola: Yeah, so basically, like, once you have the tags, everything you could do, like a dbt run, for instance.
168 00:18:55.160 ⇒ 00:19:01.459 Demilade Agboola: now, like, you can just add, hey, dbt run, select tag, and then put the tag name.
169 00:19:01.750 ⇒ 00:19:02.159 Michael Thorson: you know, I think.
170 00:19:02.160 ⇒ 00:19:04.660 Demilade Agboola: Instead of, like, running every single model, goes.
171 00:19:04.660 ⇒ 00:19:05.150 Michael Thorson: Good.
172 00:19:05.150 ⇒ 00:19:09.960 Demilade Agboola: out only these tags, the models with these tags. And you can also start to add, like, more…
173 00:19:10.300 ⇒ 00:19:15.959 Demilade Agboola: like, complexity in a nice way, in the sense of, like, I can say, hey, do plus
174 00:19:16.270 ⇒ 00:19:19.380 Demilade Agboola: The tag name, and so it says everything upstream of that tag.
175 00:19:19.690 ⇒ 00:19:21.950 Demilade Agboola: Yeah. That’s kind of what I was kind of thinking.
176 00:19:22.060 ⇒ 00:19:39.540 Demilade Agboola: So if I say every single output model with this tag, every 3 or plus of that, it would run everything upstream of those models that… so you won’t be wasting resources building out unnecessary, models. So every model that runs leads to a direct output that we need during business hours.
177 00:19:40.300 ⇒ 00:19:46.660 Michael Thorson: Perfect. Then I think, yeah, then with this, this is a great, like, turning point. I’ve been thinking about this all week, but, like.
178 00:19:46.660 ⇒ 00:20:09.790 Michael Thorson: with this CSV, or, like, this, Gsheet, if you just want to add, like, all the marts you’re seeing and make sure we’re, like, aligned there, I can add… first layer of tags is, like, overnight versus, like, daytime, like, regular schedule, and then there’s… I’ll probably add in some other tags, just, like, some business work streams. Yeah. So that down the road, we can use that for, like, building some internal products, and just making sure, like, Mary’s empowered to trigger a refresh.
179 00:20:09.920 ⇒ 00:20:16.089 Michael Thorson: Or even, like, downstream users. There’s so many times, like, at TAP where it’s like, is this live? And I would love to just…
180 00:20:16.090 ⇒ 00:20:17.850 Mary Burke: We’d have no idea. Yeah.
181 00:20:17.850 ⇒ 00:20:20.350 Michael Thorson: Give people button. Awesome.
182 00:20:20.350 ⇒ 00:20:28.850 Demilade Agboola: Okay, and also, another, like, thing is, like, the way it’s set up right now is using the default, like, scheduled every 4-hour run.
183 00:20:29.050 ⇒ 00:20:44.789 Demilade Agboola: But we can start to actually do, like, cron jobs, and we can say, hey, actually, we don’t need it every 4 hours, so, like, we can literally create a business hour, cron job. So every hour, every, like, 4 hours between, like, a certain time frame.
184 00:20:44.790 ⇒ 00:20:51.260 Demilade Agboola: or a certain number, like, days. So, like, say, for instance, on Monday to Friday, between, say.
185 00:20:52.070 ⇒ 00:20:59.649 Demilade Agboola: 7AM ET and 7pm ET, or thereabout, like, we only need these runs every 4 hours for the business hour jobs.
186 00:21:00.010 ⇒ 00:21:01.050 Michael Thorson: And then…
187 00:21:01.050 ⇒ 00:21:06.080 Demilade Agboola: once 7PM ET hits, We might have, like, one overnight refresh.
188 00:21:06.380 ⇒ 00:21:13.039 Demilade Agboola: And then once 7AM ET hits the next day, we have the 4-hour jobs that run. So, like, effectively, we can start to create, like.
189 00:21:13.790 ⇒ 00:21:25.829 Demilade Agboola: buckets of jobs, so we don’t necessarily have a round-the-clock for, every 4 hours we’re running the same models every single time. We can actually start to put a bit more thought into how we want it to run.
190 00:21:26.530 ⇒ 00:21:44.929 Michael Thorson: That sounds great. Yeah, then let’s just start a source of truth, so you, like, we can build upon this kind of, like, first pass. We’ll start with, yeah, the overnight versus daytime, and then we’ll add some, like, additional tags, and then if there’s anything that you recommend, feel free to, like, add a field in the, like, a column name, or, like, some tags are like, oh, like, these are useful.
191 00:21:44.930 ⇒ 00:21:49.420 Michael Thorson: And I’m happy to, like, apply them to the right models,
192 00:21:49.420 ⇒ 00:21:52.149 Michael Thorson: Yeah, this’ll be super, super great, I think.
193 00:21:52.390 ⇒ 00:22:00.580 Demilade Agboola: Alright, sounds good, then. Alright, so the final thing is, like, just hours.
194 00:22:03.500 ⇒ 00:22:09.650 Demilade Agboola: So, like, in prior week, we spent about 15 hours, there’s quite a bit of, like, debugging. Yep.
195 00:22:09.900 ⇒ 00:22:12.079 Demilade Agboola: This week we’re around 10.
196 00:22:13.340 ⇒ 00:22:18.570 Demilade Agboola: And so, basically, I see you’re squinting, I think the font is really small.
197 00:22:18.920 ⇒ 00:22:21.639 Michael Thorson: I know, I zoomed in, we’re good now.
198 00:22:23.810 ⇒ 00:22:29.099 Demilade Agboola: So yeah, we have things around, like, reverting, like, the last bins, load changes, we’ve had to…
199 00:22:29.300 ⇒ 00:22:32.220 Demilade Agboola: Make changes to some of the spins.
200 00:22:32.430 ⇒ 00:22:34.570 Demilade Agboola: setups.
201 00:22:35.150 ⇒ 00:22:37.520 Demilade Agboola: done the test coverage for SpinsFlow.
202 00:22:37.860 ⇒ 00:22:41.570 Demilade Agboola: Then we had to, like, monitor the perfect runs.
203 00:22:41.700 ⇒ 00:22:46.519 Demilade Agboola: And then, just, like, things around triaging, failures and backfills.
204 00:22:46.770 ⇒ 00:22:50.729 Demilade Agboola: And then, also, we had to backfill two flows this week.
205 00:22:51.190 ⇒ 00:23:03.670 Demilade Agboola: and then just, like, things around, like, dbt fixes and, all of that. I think part of what would accelerate certain processes will be things around,
206 00:23:04.750 ⇒ 00:23:10.419 Demilade Agboola: like, I know for dbt especially, if we’re able to get the dev into, like, a faster
207 00:23:11.490 ⇒ 00:23:14.270 Demilade Agboola: Runtime, especially once we’re dealing with certain models.
208 00:23:14.390 ⇒ 00:23:21.999 Demilade Agboola: That would speed up the process, because at that point, it doesn’t take as long for the models to run, like, able to get, like, faster.
209 00:23:22.350 ⇒ 00:23:24.600 Demilade Agboola: feedback on the changes that I made.
210 00:23:25.820 ⇒ 00:23:30.659 Demilade Agboola: And then once those changes are made, we are able to, like, you know, make those modifications off of that.
211 00:23:30.930 ⇒ 00:23:38.710 Demilade Agboola: And then, yeah, I think, basically, the next steps would just be around, like, adjusting priorities.
212 00:23:39.480 ⇒ 00:23:42.699 Demilade Agboola: Whenever there are any, like, changes in…
213 00:23:43.110 ⇒ 00:23:46.720 Demilade Agboola: What we feel is important, as well as just, like, you know, the…
214 00:23:47.160 ⇒ 00:23:52.209 Demilade Agboola: Let you know the hours and the priorities, basically, as the engagement works… as the engagement skills.
215 00:23:53.700 ⇒ 00:24:03.970 Michael Thorson: Yeah, I think this makes a lot of sense, just, like, I think, looking forward as well, we will have some, like, we’ve been working closely with our… with Heather, like, our internal team on…
216 00:24:04.110 ⇒ 00:24:09.180 Michael Thorson: This spins current marts, and there’s a couple,
217 00:24:09.640 ⇒ 00:24:25.169 Michael Thorson: new queries we actually need to make on the prefix side, so just, like, a look ahead for next week. I’ve been chatting with, Ashwini on the side, just kind of, like, prepping him for, like, what we’ll probably need for next week. Still kind of getting an understanding of, like.
218 00:24:25.270 ⇒ 00:24:34.020 Michael Thorson: how we’re gonna, like, expand the filters and then do a backfill. So, like, probably be focused on spins from the prefix side again.
219 00:24:34.130 ⇒ 00:24:37.230 Michael Thorson: Just calling that out for this team.
220 00:24:37.530 ⇒ 00:24:43.070 Michael Thorson: And then, on the DBC side, Demi, like, what is your workflow, I guess, for tests?
221 00:24:43.320 ⇒ 00:24:52.070 Michael Thorson: I’m interested that you’re talking… that you say, like, if we split apart the dbt run, or that we optimize the runtime.
222 00:24:52.250 ⇒ 00:24:59.769 Michael Thorson: that… that would… that would, like, improve your efficiency. Like, what’s your kind of, like, test workflow before you merge a change? Are you running the full…
223 00:24:59.900 ⇒ 00:25:04.329 Michael Thorson: Model in cloud environment, or are you just running, like, locally?
224 00:25:04.330 ⇒ 00:25:15.449 Demilade Agboola: Yeah, I’m running locally, so I run the lineage of whatever changes were made. So, like, dbt allows you to be able to say, hey, select the models.
225 00:25:15.920 ⇒ 00:25:16.460 Michael Thorson: Good.
226 00:25:16.620 ⇒ 00:25:19.869 Demilade Agboola: And then either on, like, downstream, upstream, or just the model itself.
227 00:25:20.420 ⇒ 00:25:25.250 Demilade Agboola: I think ultimately, especially within the spins workflow, the models are pretty heavy.
228 00:25:25.440 ⇒ 00:25:32.239 Demilade Agboola: And so… and so any, like, change to the Spins workflow does take…
229 00:25:32.470 ⇒ 00:25:34.700 Demilade Agboola: A tremendous amount of time for it to run.
230 00:25:35.240 ⇒ 00:25:35.940 Michael Thorson: Hmm.
231 00:25:36.650 ⇒ 00:25:40.379 Demilade Agboola: And so ultimately, yeah, just being able to, like…
232 00:25:40.550 ⇒ 00:25:56.819 Demilade Agboola: have dev environments where instead of having, like, millions of rows, we can just have, say, a thousand rows, or, you know, 10,000 rows. So it’ll be, like, dummy sample data from, like, the, from the API, or, like, from the initial, like, table in prod.
233 00:25:57.000 ⇒ 00:26:15.089 Demilade Agboola: That way, what we’re just doing is we’re getting a feel of, like, if the change is going to be a breaking change or not, rather than, you know, us really trying to rebuild every single thing from scratch, because, you know, we just need to. I think what the goal here isn’t necessarily always, like.
234 00:26:15.490 ⇒ 00:26:20.210 Demilade Agboola: Perfect. It should just be, like, hey, does this change work in production?
235 00:26:20.320 ⇒ 00:26:23.149 Demilade Agboola: And then we can have that, you know, rollout.
236 00:26:23.380 ⇒ 00:26:26.189 Demilade Agboola: Which is kind of also what I want to do for CICD as well.
237 00:26:27.260 ⇒ 00:26:35.680 Michael Thorson: Yeah, that’s… that’s a super… I hit that as well, like, I would try to run some changes, and it takes… it was, like, a 30-minute runtime for the full…
238 00:26:35.850 ⇒ 00:26:47.839 Michael Thorson: base to Mart, so I feel you on that. So yeah, if you… I think that’s a great priority, is, like, focus, like, if you’re working with the Spins model, like, focus on just that, like, test…
239 00:26:48.510 ⇒ 00:26:54.719 Michael Thorson: like, test dataset, I guess, and then we can work out a workflow so we can work more effectively with this model.
240 00:26:55.020 ⇒ 00:26:55.960 Demilade Agboola: Exactly.
241 00:26:56.100 ⇒ 00:26:58.010 Demilade Agboola: So that, that’s that.
242 00:26:58.450 ⇒ 00:27:00.869 Demilade Agboola: Yeah, so that’s it in terms of, like.
243 00:27:01.190 ⇒ 00:27:06.960 Demilade Agboola: things wanted to talk about. And…
244 00:27:07.430 ⇒ 00:27:13.119 Demilade Agboola: I guess just, you know, same old general questions. Do we have any, like, feedback on the engagement so far?
245 00:27:13.280 ⇒ 00:27:20.209 Demilade Agboola: Are there any things that were like, hey, this could… this could have been better, or this would be nice to have? And just, like.
246 00:27:20.460 ⇒ 00:27:21.730 Demilade Agboola: Just general feedback.
247 00:27:25.640 ⇒ 00:27:44.379 Michael Thorson: From my side, I think, no, like, doing okay, just kind of want to understand, like, priorities for the week. I think it was… it’s really useful to know, kind of, like, what, like, the breakdown of tasks is for the week, and just kind of, like, who owns what. I think some of this, like, CI… or some of this, like.
248 00:27:44.430 ⇒ 00:27:51.039 Michael Thorson: test plus the tagging structure for the dbt runs is gonna be, like a…
249 00:27:51.080 ⇒ 00:28:07.579 Michael Thorson: first step to, like, a much bigger unlock from the data, like, engineering side, so… Okay. Yeah, like, would be super stoked if that’s kind of a priority on your side for the week, and then Ashwini will kind of, like, work through the iteration cycle for, just, like, improving the spins API calls.
250 00:28:07.850 ⇒ 00:28:09.440 Demilade Agboola: Okay, alright, sounds good.
251 00:28:10.110 ⇒ 00:28:31.910 Mary Burke: Yeah, Demi, I know we had talked, previously about just sending out, at the beginning of the week, just aligning on what those priorities are and where the focuses are, and then if things shift during the week, like, things come up where we need to backfill, just communicating that in that Slack channel for, just for the two of us, and then also JT and the other… the entire team to have… have visibility there, so we could get back in the habit of.
252 00:28:31.910 ⇒ 00:28:34.410 Mary Burke: Doing that in the Slack channel, that would be awesome.
253 00:28:34.410 ⇒ 00:28:42.400 Demilade Agboola: Okay, sounds good, Ben. Communicate the priorities for the week and, like, just our allocations to those priorities.
254 00:28:42.540 ⇒ 00:28:46.330 Demilade Agboola: And then, yeah, as the week goes on, if there.
255 00:28:46.960 ⇒ 00:28:58.280 Demilade Agboola: that break or any things that change, we’ll also try and keep that. I mean, now that you’re also in the, like, alerting channel, you can also see if things need to… if priorities need to shift based off of things breaking.
256 00:28:59.880 ⇒ 00:29:03.110 Mary Burke: Yeah, a lot of alerts come in there, so it’s hard to know what’s…
257 00:29:03.110 ⇒ 00:29:06.620 Demilade Agboola: And trust me, that’s the sanitized version. Like, when we initially set it up.
258 00:29:07.200 ⇒ 00:29:07.670 Mary Burke: Yeah.
259 00:29:07.670 ⇒ 00:29:13.469 Demilade Agboola: I had to ask Ashwini, like, okay, we need to, like, actually clean this up. That’s why we asked you, like, what are the important runs.
260 00:29:13.470 ⇒ 00:29:14.060 Mary Burke: Yeah.
261 00:29:14.060 ⇒ 00:29:19.359 Demilade Agboola: Because there were stuff that were breaking down, like, okay, yeah, this will be too… this will be, like, really overwhelming, but you’ll…
262 00:29:19.460 ⇒ 00:29:23.779 Demilade Agboola: your Slack would just keep going ding, ding, ding, ding, ding, so we’re like, alright, let’s get it to a point where it’s.
263 00:29:23.780 ⇒ 00:29:26.099 Mary Burke: It’s scary that that’s the pared-down version, then.
264 00:29:26.100 ⇒ 00:29:38.619 Demilade Agboola: But yeah, I think the idea is just we want to get it to a point where it’s stable, and we can feel, like, really confident in the data that’s there. So part of, like, even what we’re going to be looking at next week would just be, like.
265 00:29:39.110 ⇒ 00:29:41.289 Demilade Agboola: What’s happening with the runs,
266 00:29:41.430 ⇒ 00:29:52.139 Demilade Agboola: Ideally, I want to get to the point where we don’t have the same failures occurring multiple times, so if we can figure out, like, okay, this is what happened, okay, how do we get ahead of it next time?
267 00:29:52.280 ⇒ 00:30:03.720 Demilade Agboola: And so, like, we want to have, like, new errors, quote-unquote, like, so we can say, hey, this is software, we’re just learning to, like, make our… the pipeline and the system more robust, so we can handle any,
268 00:30:03.940 ⇒ 00:30:09.330 Demilade Agboola: change, so we get… the idea is to get to a point where your Slack doesn’t have to keep going off. You just go, oh…
269 00:30:09.540 ⇒ 00:30:14.889 Demilade Agboola: it’s been here for, like, 2 weeks, not, like, we’ve not heard any alerts, or we’ve not seen any alerts. That will be the goal.
270 00:30:18.680 ⇒ 00:30:24.579 Demilade Agboola: Also, in terms of, like, triaging and just being ahead of things, like, do you feel comfortable with how things are going?
271 00:30:27.020 ⇒ 00:30:29.620 Michael Thorson: Honestly, yeah, I think…
272 00:30:29.800 ⇒ 00:30:50.710 Michael Thorson: Yeah, I think it’s… this week was really, like, helpful, and I think we’re still finding our footing in terms of, like, that alert system, and just kind of, like, getting the coverage. The good news is we have too many eyes on it. Like, I… I ended up running a backfill just because, like, on my own machine, because we were, like, kind of falling behind on a certain… a couple tables.
273 00:30:50.710 ⇒ 00:31:02.989 Michael Thorson: I think… still kind of figuring out the right workflow there for, like, okay, like, what time of day will Ashwini hop on the backfill from prefig versus what time of day, because he’s working different hours, should we hop in there?
274 00:31:03.170 ⇒ 00:31:10.380 Michael Thorson: Or, like, yeah, and also just, like, the SLA for, like, what… how many… It’s… that could be…
275 00:31:10.490 ⇒ 00:31:28.979 Michael Thorson: maybe on the dbt test side, where it’s like, really understanding, like, the SLA and, like, the look-back period for all the models, I think we’re all kind of, like, getting our… our grip around that as well, so that we know, like, if General Ledger fails for 12 hours straight, for example, it’s like, oh, that’s actually requiring a backfill. I think just, like.
276 00:31:28.980 ⇒ 00:31:47.539 Michael Thorson: starting to get familiar around those, like, key tables we’ve kind of centered around, like, general ledger, invoices, shipments, etc, like, Business Central mostly, is just, like, maybe having… I don’t know, like, we can add, like, something pinned to the channel as just, like, a cheat sheet, so that we can all be like, oh, like, the last successful run was, you know.
277 00:31:47.550 ⇒ 00:31:57.629 Michael Thorson: within 6 hours, like, it’s something we don’t have to worry about. I think that’s kind of the place I would love us to get to, and it’s totally fine, we’re not there yet. I think we’re… we’re kind of in the learning curve of, like.
278 00:31:57.950 ⇒ 00:32:07.319 Michael Thorson: the whole team, like, the three of us, plus Ashwini, getting on top of that, so… Yeah, that’s just kind of where I’m thinking, is just, like, knowing…
279 00:32:07.680 ⇒ 00:32:19.139 Michael Thorson: as an organization, like a team, like, what failures actually are, dangerous? And, like, being really, like, crystal clear with that alert system. So it’s not, like, to your point, like, a lot of noise.
280 00:32:19.710 ⇒ 00:32:26.009 Demilade Agboola: Yeah, no, that’s fair, that’s fair. I think… I also agree with that. I know, like, ultimately,
281 00:32:26.070 ⇒ 00:32:34.520 Demilade Agboola: some alerts, again, prior to when you guys joined the channel, I know there were some alerts that, like, will pop up, and I’ll just ask Ashwini if, like.
282 00:32:34.580 ⇒ 00:32:53.219 Demilade Agboola: is this the first one? And he’s like, no, like, it is the first one. And I’ll say, okay, like, watch out for subsequent runs, and let me know if they run, like, if there’s an issue. Sometimes subsequent runs will be fine, so it’s like, okay, cool, you know, it was just one hour missed out of, like, you know, the hourly thing. So just being able to have that sort of clarity on, like.
283 00:32:53.310 ⇒ 00:33:01.490 Demilade Agboola: Different models, the cadence, at what point it becomes, like, critical to, trigger a backfill?
284 00:33:01.750 ⇒ 00:33:21.579 Demilade Agboola: And at what point, like, because again, not every single failure is critical. Sometimes it’s just a one-off that we are fine having, and sometimes it’s like, oh, if there are two of these, or if there are three of these in a row, that means it’s gone for 6 hours, for instance, without a refresh, and this needs to trigger a backfill. So just having the rules around that will be…
285 00:33:21.760 ⇒ 00:33:24.320 Demilade Agboola: Stuff we’ll look into and just document as well.
286 00:33:25.660 ⇒ 00:33:26.800 Michael Thorson: Yeah, cool.
287 00:33:27.060 ⇒ 00:33:38.290 Michael Thorson: Yep, and that’s… yeah, it’s just kind of general, that’s not really, like, a feedback on any quality of work. I think it’s just, like, let’s keep improving, and I think we’ll just be able to, you know, work well as a team together. Even better.
288 00:33:38.290 ⇒ 00:33:38.820 Demilade Agboola: market.
289 00:33:38.980 ⇒ 00:33:41.399 Demilade Agboola: Art sounds good, Ben.
290 00:33:42.060 ⇒ 00:33:48.160 Demilade Agboola: Yeah, other than that, I don’t have any other thing from my side. I don’t know if either of you do.
291 00:33:51.060 ⇒ 00:33:51.920 Michael Thorson: All good.
292 00:33:52.250 ⇒ 00:33:52.750 Demilade Agboola: Okay.
293 00:33:52.750 ⇒ 00:33:53.430 Mary Burke: Great.
294 00:33:53.430 ⇒ 00:33:54.340 Demilade Agboola: Alright, then.
295 00:33:54.440 ⇒ 00:34:01.550 Demilade Agboola: Mary, I’ll probably reach out to you early next week, just so I can have a conversation with you, just about, like.
296 00:34:01.980 ⇒ 00:34:05.500 Demilade Agboola: The project as a whole, and just, like, what next steps look like as well.
297 00:34:06.320 ⇒ 00:34:07.239 Mary Burke: Okay, sounds good.
298 00:34:07.240 ⇒ 00:34:08.010 Demilade Agboola: Alright, then.
299 00:34:08.730 ⇒ 00:34:10.070 Mary Burke: Alright, thanks, Demi.
300 00:34:10.070 ⇒ 00:34:11.570 Demilade Agboola: Alright, have a good weekend, everyone.
301 00:34:12.130 ⇒ 00:34:12.480 Mary Burke: Me too.
302 00:34:12.830 ⇒ 00:34:13.420 Demilade Agboola: Bye.