Meeting Title: Urban Stems Data Monitoring Demo Date: 2025-08-21 Meeting participants: Perry’s Fellow Note Taker, Demilade Agboola, Ian Biles, Jesse Sawa-an, pk.arthur, Alex K, Emily Giant, Santiago Posso, Stephanie Plaza
WEBVTT
1 00:01:34.250 ⇒ 00:01:36.220 Alex K: What up, y’all?
2 00:01:38.690 ⇒ 00:01:39.800 Emily Giant: Hello!
3 00:01:41.710 ⇒ 00:01:42.380 Demilade Agboola: Boom.
4 00:01:43.960 ⇒ 00:01:48.080 Emily Giant: I think Utam is presenting today, if I’m not mistaken.
5 00:01:49.110 ⇒ 00:01:51.400 Alex K: Yes, he is.
6 00:01:52.340 ⇒ 00:01:56.969 Demilade Agboola: So I did message him. I know you had a meeting right before.
7 00:01:58.580 ⇒ 00:02:01.150 Demilade Agboola: I just want to get confirmation.
8 00:02:01.420 ⇒ 00:02:02.150 Emily Giant: Yeah.
9 00:02:06.330 ⇒ 00:02:10.939 Emily Giant: And then Felipe… Is not able to join.
10 00:02:21.090 ⇒ 00:02:21.840 Demilade Agboola: Okay.
11 00:02:23.120 ⇒ 00:02:28.689 Emily Giant: Oh, Santi’s back! Welcome back! Or you may have been back for a while, I just thought you were also gone for a while.
12 00:02:28.870 ⇒ 00:02:37.200 Santiago Posso: Yes, I’m very glad to be back. I’m a little bit lost of what are we doing here, but I will be, like, keeping on posted.
13 00:02:38.980 ⇒ 00:02:57.190 Emily Giant: You may be more lost after this meeting, because I think that, today Utam’s gonna demo, like, a future state thing, that’s, we’ve been having a lot of, like, refresh failures lately, and, so he was gonna demo a program that, he’s been working on building out that will, like.
14 00:02:58.020 ⇒ 00:03:06.489 Emily Giant: get us proactively in front of any of these failures, and help us deploy things faster. And I would do the demo if I had.
15 00:03:06.490 ⇒ 00:03:07.090 Demilade Agboola: I will.
16 00:03:07.090 ⇒ 00:03:07.650 Emily Giant: Excellent.
17 00:03:07.650 ⇒ 00:03:08.290 Demilade Agboola: Back to it.
18 00:03:08.290 ⇒ 00:03:08.640 Emily Giant: But….
19 00:03:08.640 ⇒ 00:03:12.700 Demilade Agboola: No, I could just start it now, to be fair, I have access to it, so if…
20 00:03:12.960 ⇒ 00:03:17.719 Demilade Agboola: Because he hasn’t yet responded, and since I have access, might as well just start.
21 00:03:19.270 ⇒ 00:03:19.870 Emily Giant: Sounds good.
22 00:03:19.870 ⇒ 00:03:22.860 Alex K: As they say, the only way to begin is to begin, right?
23 00:03:24.440 ⇒ 00:03:29.189 Demilade Agboola: Okay… Alright, so… can you see my screen?
24 00:03:29.350 ⇒ 00:03:30.050 Emily Giant: Yes.
25 00:03:31.060 ⇒ 00:03:37.030 Demilade Agboola: Great. … So this tool is called Metaplane.
26 00:03:37.800 ⇒ 00:03:45.939 Demilade Agboola: And it’s very useful for… monitoring different parts of the data architecture.
27 00:03:46.890 ⇒ 00:03:52.860 Demilade Agboola: Very less so the BI, but more the tables that come before that.
28 00:03:53.060 ⇒ 00:03:57.360 Demilade Agboola: So we have things in Redshift where we can…
29 00:03:57.650 ⇒ 00:04:01.049 Demilade Agboola: See everything that’s happening across the different schemas.
30 00:04:01.280 ⇒ 00:04:07.219 Demilade Agboola: We can prioritize certain schemas above other schemas for, like, tests.
31 00:04:07.460 ⇒ 00:04:12.789 Demilade Agboola: So, for instance, we know the tables that power a lot of the dashboards.
32 00:04:13.100 ⇒ 00:04:19.779 Demilade Agboola: And for them, we can create monitors that allow us to know when things happen within that.
33 00:04:20.100 ⇒ 00:04:24.720 Demilade Agboola: So, for an example would be the column count, or certain
34 00:04:24.920 ⇒ 00:04:36.540 Demilade Agboola: of certain, … the columns should remain consistent. The number of rows in certain columns should increase at a consistent rate. So, for instance, if you had
35 00:04:36.670 ⇒ 00:04:38.090 Demilade Agboola: You know, a million.
36 00:04:38.790 ⇒ 00:04:41.239 Demilade Agboola: Rows in a table for orders.
37 00:04:41.380 ⇒ 00:04:45.770 Demilade Agboola: If the next day you have $500,000, there’s a problem.
38 00:04:45.890 ⇒ 00:04:53.909 Demilade Agboola: Or if the rate at which it’s increasing is not consistent. So if it’s been increasing at around, say, 200,000 rows every day.
39 00:04:54.060 ⇒ 00:05:00.320 Demilade Agboola: And then, on one day it increases by 800,000 rows, it will flag it as an, like, there’ll be…
40 00:05:00.470 ⇒ 00:05:02.899 Demilade Agboola: A notification that something is wrong.
41 00:05:03.030 ⇒ 00:05:05.560 Demilade Agboola: With the increase on that particular day.
42 00:05:05.730 ⇒ 00:05:13.460 Demilade Agboola: So that allows us to catch things like if there’s a bad join being done, someone has pushed code that has, created
43 00:05:13.800 ⇒ 00:05:16.360 Demilade Agboola: Some new errors within the data.
44 00:05:16.790 ⇒ 00:05:21.860 Demilade Agboola: … And as a result, that is part of what we do to, like, monitor.
45 00:05:22.120 ⇒ 00:05:25.980 Demilade Agboola: So there are different types of monitors we could do. We can do freshness.
46 00:05:26.380 ⇒ 00:05:29.129 Demilade Agboola: We can do the count.
47 00:05:29.420 ⇒ 00:05:32.350 Demilade Agboola: Give me one second. So this is for this table.
48 00:05:32.820 ⇒ 00:05:44.119 Demilade Agboola: But we could do different types of monitors. We can basically do, like, the freshness, we can do counts, the row counts is just basically us trying to ensure that
49 00:05:44.700 ⇒ 00:05:49.300 Demilade Agboola: For a certain, table, the number of rows remain consistent.
50 00:05:49.990 ⇒ 00:05:54.649 Demilade Agboola: And you might not recognize these tables, but these are a lot of the tables that power
51 00:05:54.820 ⇒ 00:05:58.520 Demilade Agboola: by the number of our dashboards. So this is Tableau.comXF.
52 00:05:58.750 ⇒ 00:06:02.149 Demilade Agboola: And these are some of the models that we have been building out.
53 00:06:02.330 ⇒ 00:06:13.700 Demilade Agboola: within, our dbt new infrastructure. Again, the idea here is we just want to be sure that whatever data is being used downstream in your reports, if there’s any error.
54 00:06:13.960 ⇒ 00:06:17.389 Demilade Agboola: We will be the first to know, we’ll be the first to say, hey, like.
55 00:06:17.620 ⇒ 00:06:21.730 Demilade Agboola: Your data has exploded, or your data has dropped significantly.
56 00:06:22.040 ⇒ 00:06:28.260 Demilade Agboola: We are on it, we know that there’s a problem, before you reach out to us and let us know that there is a problem.
57 00:06:28.460 ⇒ 00:06:35.320 Demilade Agboola: Another example would be freshness, which has come up quite a bit over the last couple of weeks, but just being able to tell
58 00:06:35.650 ⇒ 00:06:37.549 Demilade Agboola: How fresh the data is.
59 00:06:37.720 ⇒ 00:06:43.340 Demilade Agboola: Before you reach out and say, hey, revenue has not refreshed over the past Kino.
60 00:06:43.470 ⇒ 00:06:50.460 Demilade Agboola: 6 hours or something. Us being able to go, hey, there is a freshness test running on the different models.
61 00:06:50.740 ⇒ 00:06:53.659 Demilade Agboola: That we have pushed to production.
62 00:06:53.990 ⇒ 00:06:57.380 Demilade Agboola: And as a result, it’s still…
63 00:06:57.850 ⇒ 00:07:05.889 Demilade Agboola: And obviously, that means the business stakeholders are looking at still data. So this is part of what work we’re implementing, so that we can be ahead of that.
64 00:07:06.220 ⇒ 00:07:10.279 Demilade Agboola: And so that we can always just be like, yes, the numbers…
65 00:07:10.680 ⇒ 00:07:16.340 Demilade Agboola: are bad, but we know that, and we’re already working on it before the stakeholders reach out to us.
66 00:07:16.630 ⇒ 00:07:18.500 Demilade Agboola: Any questions so far?
67 00:07:29.190 ⇒ 00:07:30.220 Demilade Agboola: Okay.
68 00:07:30.740 ⇒ 00:07:35.590 Demilade Agboola: … So the next theme is we…
69 00:07:36.950 ⇒ 00:07:41.479 Demilade Agboola: Within dbt, we have different jobs that run to produce the tables that we
70 00:07:41.800 ⇒ 00:07:44.870 Demilade Agboola: I’ve just looked at now, that we have tests on.
71 00:07:45.010 ⇒ 00:07:49.989 Demilade Agboola: And that in itself can also bring errors that we need to be on top of as well.
72 00:07:50.250 ⇒ 00:07:52.810 Demilade Agboola: So sometimes part of why
73 00:07:52.910 ⇒ 00:08:01.069 Demilade Agboola: if freshness test might fail. So, for instance, revenue might get out of sync for hours, is if a DBT job isn’t running.
74 00:08:01.290 ⇒ 00:08:16.530 Demilade Agboola: And if a job isn’t running, the table isn’t getting refreshed. So, these things are kind of linked, so we want to be able to know when things are failing within DBT, and also know the effect it’s having on our redshift tables before they get exposed to Looker.
75 00:08:17.610 ⇒ 00:08:23.960 Demilade Agboola: And so right now, we have monitors for the different jobs that we’re, … That we have in production.
76 00:08:24.230 ⇒ 00:08:35.619 Demilade Agboola: The cool thing about this is that we can kind of see things like the duration jump. So, for instance, there’s a jump in how long this took, relative to how long it normally takes.
77 00:08:35.789 ⇒ 00:08:41.620 Demilade Agboola: So we can see things like that. We can also see… The easy failure points.
78 00:08:41.850 ⇒ 00:08:49.589 Demilade Agboola: And then another thing we have set up is we have a DPT… well, we have a… an alerts channel.
79 00:08:50.320 ⇒ 00:08:53.180 Demilade Agboola: So, when these metaplane alerts fail.
80 00:08:53.430 ⇒ 00:09:01.779 Demilade Agboola: we get, like, a detailed message that just comes into the channel, and we can see, hey, it failed on this model. It lets us know
81 00:09:02.030 ⇒ 00:09:13.479 Demilade Agboola: the point of failure, and that allows us to decide how we want to be able to fix things. Do we want to hop in right there, restart the process? Do we need to check what the error is?
82 00:09:13.740 ⇒ 00:09:17.630 Demilade Agboola: And so, yeah, we have different monitors.
83 00:09:17.910 ⇒ 00:09:21.550 Demilade Agboola: going on here. Most of them are largely duration monitors.
84 00:09:21.780 ⇒ 00:09:26.470 Demilade Agboola: … And the idea is we can also…
85 00:09:27.250 ⇒ 00:09:30.010 Demilade Agboola: Set up, like, different rules as well.
86 00:09:31.500 ⇒ 00:09:35.899 Demilade Agboola: But it’s generally just very helpful for, like, just tracking what’s going on.
87 00:09:36.180 ⇒ 00:09:40.719 Demilade Agboola: And helping us have an idea of the…
88 00:09:40.990 ⇒ 00:09:43.060 Demilade Agboola: Like, when things are higher than normal.
89 00:09:43.310 ⇒ 00:09:49.079 Demilade Agboola: when it’s higher than normal, it creates an incident. So, like, for instance, this ran last at… it ran…
90 00:09:49.530 ⇒ 00:09:50.810 Demilade Agboola: Today.
91 00:09:51.350 ⇒ 00:09:54.490 Demilade Agboola: to bear in mind, I’m… This is, like, an hour ago.
92 00:09:54.950 ⇒ 00:10:01.060 Demilade Agboola: Or, like, 30 minutes, actually. So this ran for 41 minutes, compared to how long it tends to run, which is, like.
93 00:10:01.530 ⇒ 00:10:09.809 Demilade Agboola: about 20 minutes. They run for, like, doubly as long as it normally does. So, like, these are things we can always just, like, look into.
94 00:10:10.200 ⇒ 00:10:18.350 Demilade Agboola: And the beauty of it is we also get to… Be able to see.
95 00:10:19.270 ⇒ 00:10:20.250 Demilade Agboola: Second.
96 00:10:21.560 ⇒ 00:10:28.280 Demilade Agboola: We also get to be able to see, like, the… the lineage, so we can kind of see what, like, things it affects.
97 00:10:28.750 ⇒ 00:10:30.650 Demilade Agboola: As well as…
98 00:10:30.970 ⇒ 00:10:37.259 Demilade Agboola: Sorry, it’s a bit of a hard thing to… to visualize, but, like, it shows us, like, what
99 00:10:37.360 ⇒ 00:10:39.380 Demilade Agboola: Is in this job, basically.
100 00:10:39.610 ⇒ 00:10:45.169 Demilade Agboola: As you can tell, there are a lot of models going on, like, in this job right now.
101 00:10:45.630 ⇒ 00:10:53.229 Demilade Agboola: But, like, ideally, the idea is we can see if things break, what RW has broken.
102 00:10:53.460 ⇒ 00:11:07.470 Demilade Agboola: What it’s affecting, and you can easily tell, like, hey, if this… if this job isn’t running, therefore revenue will be down, or inventory data will be down, or most things will be down, because this, for instance, this core…
103 00:11:07.870 ⇒ 00:11:08.730 Demilade Agboola: tag.
104 00:11:09.170 ⇒ 00:11:11.830 Demilade Agboola: Affects so much within the infrastructure.
105 00:11:12.140 ⇒ 00:11:16.590 Demilade Agboola: So, yeah, these are things that we can, always just, like, do.
106 00:11:17.340 ⇒ 00:11:21.130 Demilade Agboola: I’m trying to… Second.
107 00:11:22.560 ⇒ 00:11:25.020 Demilade Agboola: I’m trying to….
108 00:11:33.850 ⇒ 00:11:38.290 Emily Giant: Does everyone understand, like, what a job is, when he says, job?
109 00:11:40.630 ⇒ 00:11:53.809 Emily Giant: Because I know that I didn’t when I was an analyst, and I’ll just say it just in case. A job is, like, the set of models that are refreshed at a cadence. So, like, the daily full refresh for all of the things that you interact with in Looker
110 00:11:53.810 ⇒ 00:12:03.969 Emily Giant: It rebuilds the entire table from scratch every night, and that was that big line that you saw going from, like, 2 hours to 4, so that we know something is,
111 00:12:03.970 ⇒ 00:12:12.749 Emily Giant: amiss with that particular job, but the job is just, like, whatever models, that will flow to Looker with your data.
112 00:12:12.750 ⇒ 00:12:18.119 Emily Giant: are selected to refresh, in…
113 00:12:18.240 ⇒ 00:12:26.969 Emily Giant: like, combination with each other. So, for example, like, all of the inventory refreshes in a job called, like, polyatomic 30-minute inventory.
114 00:12:27.090 ⇒ 00:12:30.590 Emily Giant: test. Sorry, that’s just my, …
115 00:12:31.290 ⇒ 00:12:36.879 Emily Giant: what is it, the TLDR about jobs, just in case, because I know we’re using that word a lot.
116 00:12:36.880 ⇒ 00:12:45.259 Stephanie Plaza: Thank you. And so, is it supposed to be, like, like, inventory will refresh, like, 4 times a day, or everything is once at night?
117 00:12:45.960 ⇒ 00:12:46.610 Emily Giant: I’m….
118 00:12:46.610 ⇒ 00:12:47.150 Demilade Agboola: Unknown.
119 00:12:47.150 ⇒ 00:12:49.170 Emily Giant: Everybody refreshes every half of an hour.
120 00:12:49.170 ⇒ 00:12:49.860 Stephanie Plaza: Okay.
121 00:12:50.990 ⇒ 00:12:55.340 Stephanie Plaza: Same with revenue, but they can’t run into each other, because there’s certain things that overlap.
122 00:12:55.340 ⇒ 00:13:09.919 Emily Giant: So, like, if they collide, the refresh will fail. And that, the other day, what we were experiencing, I think it was Monday or Tuesday when revenue wasn’t refreshing, or inventory wasn’t refreshing, this would have told us
123 00:13:10.120 ⇒ 00:13:17.399 Emily Giant: Like, as soon as it happened, with more insight, Into why and where.
124 00:13:17.400 ⇒ 00:13:27.299 Stephanie Plaza: And the ones that are currently taking, like, 4 hours, those are not public ones? Like, we’re trying to get the time down, or that’s just how that one operates and does take that much time?
125 00:13:28.030 ⇒ 00:13:44.029 Emily Giant: It ebbs and flows, like, I’ve seen it for 4 hours, and that’s not altogether strange, but that does start at 2 AM. So usually that full refresh is not when we are interacting with the data. However, …
126 00:13:44.120 ⇒ 00:13:48.369 Emily Giant: I have a massive deployment that is going to happen
127 00:13:48.370 ⇒ 00:14:04.700 Emily Giant: this week, and if we have any time at the end of this call after the demo, I wanted to chat through with everyone, like, what that might look like, and what to look out for. But, I would have done it 3 days ago if we had this, because I would have known, like, in my development instance, if
128 00:14:04.830 ⇒ 00:14:19.399 Emily Giant: like, the columns in ComponentsXF, for example, went from 5 million to 2,000. Sometimes, like, I always run my own internal tests, and I build them into dbt, but it’s really, really hard to…
129 00:14:19.520 ⇒ 00:14:29.370 Emily Giant: to identify every single thing and every single column you should be testing by the time you get to, like, components XF, where there’s 200 columns to be testing.
130 00:14:30.540 ⇒ 00:14:43.549 Emily Giant: I should say rows, not columns. There’s 200 columns, but there’s, like, 5 million rows. So, if rows are suddenly disappearing, that means, oops, somewhere along the line, I filtered out
131 00:14:43.550 ⇒ 00:14:52.839 Emily Giant: essential data, and unless I’m testing specifically for that thing at every step of the way that I’m developing, it’s really easy to lose
132 00:14:53.720 ⇒ 00:14:59.159 Emily Giant: where the issue is. And essentially, this tool…
133 00:14:59.350 ⇒ 00:15:02.599 Emily Giant: We’ll help say, like, this is…
134 00:15:03.380 ⇒ 00:15:20.509 Emily Giant: where the change happened that made it all go away, or this is where it’s colliding at this time on this model that takes way longer than the other models. And dbt has features for this, but they don’t, like, it’s…
135 00:15:21.370 ⇒ 00:15:28.880 Emily Giant: not good. It’s not good. Like, you really can’t set up alerting in this way. …
136 00:15:29.110 ⇒ 00:15:31.150 Emily Giant: You have to read through, like.
137 00:15:31.740 ⇒ 00:15:42.229 Emily Giant: a phone book of a log to debug the daily full refresh when it fails. So this is just, like, a time saver, and it’s much more proactive.
138 00:15:43.410 ⇒ 00:15:44.000 Stephanie Plaza: Awesome.
139 00:15:44.000 ⇒ 00:15:50.099 Demilade Agboola: Yeah I’m… So, a couple of things to add there.
140 00:15:50.280 ⇒ 00:15:55.509 Demilade Agboola: It also gives us, like, just a lot of info for us to be able to…
141 00:15:56.090 ⇒ 00:16:06.079 Demilade Agboola: Like, I just opened up, like, Insights right now, and you can kind of see the slowest queries, for instance, so we can then find it within dbt.
142 00:16:06.280 ⇒ 00:16:17.160 Demilade Agboola: And then optimize it as well, and say, hey, like, Veracel, we don’t want things running for an average of an hour and a half, you know? So how do we optimize that process?
143 00:16:17.960 ⇒ 00:16:29.300 Demilade Agboola: It also allows you to know what’s happening across different things. I’m trying to show you… give me one second… trying to show you my Slack, where these messages come in.
144 00:16:31.490 ⇒ 00:16:32.690 Demilade Agboola: …
145 00:16:36.660 ⇒ 00:16:37.580 Demilade Agboola: Okay.
146 00:16:39.030 ⇒ 00:16:43.520 Demilade Agboola: So, we do have a… block.
147 00:16:45.490 ⇒ 00:16:51.730 Demilade Agboola: Words, messages come in… And you can just kind of see here that
148 00:16:52.770 ⇒ 00:16:55.929 Demilade Agboola: When things fail, it just goes, hey, this model has failed.
149 00:16:56.290 ⇒ 00:16:59.089 Demilade Agboola: And it lets you know straight up what the failure is.
150 00:16:59.350 ⇒ 00:17:05.390 Demilade Agboola: And so you don’t necessarily have to even open, like, dbt, it just kind of tells you, hey.
151 00:17:05.710 ⇒ 00:17:09.589 Demilade Agboola: The duration of this is quite high.
152 00:17:09.890 ⇒ 00:17:18.650 Demilade Agboola: We expected it to take, what, 13 minutes? It’s taking 40 minutes instead, which is kind of what we saw in the graph I just showed right now.
153 00:17:18.930 ⇒ 00:17:28.189 Demilade Agboola: But yes, it kind of allows us to be proactive with these things. It allows us to be able to say, hey, what’s happening across our data?
154 00:17:28.359 ⇒ 00:17:36.560 Demilade Agboola: Because Urban Stems has a lot of, like, tables, some redundant, to be honest, but they’re just generating a lot of tables, and being able to manage them.
155 00:17:37.000 ⇒ 00:17:40.489 Demilade Agboola: And also being able to hand some of the… handle some of the…
156 00:17:40.740 ⇒ 00:17:49.210 Demilade Agboola: visibility to certain things will allow us to be proactive with things. Like, if we know these tables take so much time to run.
157 00:17:49.930 ⇒ 00:17:57.119 Demilade Agboola: It’s very easy to say, okay, so these… for the next couple of weeks, let’s focus on getting the runtime of these certain models down.
158 00:17:57.350 ⇒ 00:18:03.289 Demilade Agboola: And that will cascade to our overall runtime, because we’re tackling the most problematic tables quite easily.
159 00:18:03.600 ⇒ 00:18:08.789 Demilade Agboola: … Yeah, I think just being able to have this is very helpful.
160 00:18:09.130 ⇒ 00:18:13.820 Demilade Agboola: And then the incidents are, you know, all of that.
161 00:18:15.080 ⇒ 00:18:18.519 Demilade Agboola: So yeah, I think this is it. I don’t know if anyone has any other questions.
162 00:18:23.840 ⇒ 00:18:27.499 Demilade Agboola: Okay, if not, I’ll hand it over to Emily and Shawne to share something as well.
163 00:18:28.070 ⇒ 00:18:30.330 Emily Giant: Yeah, so,
164 00:18:30.330 ⇒ 00:18:49.540 Emily Giant: As you know, we’re moving on to revenue. Demolade and UTAM are largely focused on the revenue mart now. I’m still doing some cleanup and optimization with the inventory mart. Most of that has to do with, historical data and streamlining it with the way that our tables are set up now in Polytomic.
165 00:18:49.850 ⇒ 00:18:58.010 Emily Giant: this also… I mean, everything at Urban Stems is connected, so if you change historical data in inventory, it’s eventually gonna hit.
166 00:18:58.290 ⇒ 00:19:02.430 Emily Giant: current revenue, because we’re always doing comps. So, …
167 00:19:02.720 ⇒ 00:19:10.289 Emily Giant: the last two and a half weeks, I’ve been really overhauling that historical data, and …
168 00:19:11.380 ⇒ 00:19:27.440 Emily Giant: I want to, number one, ask everyone to, like, over the next 48 hours, if you’re doing any comp reporting, if you could send me the link in Slack, I’d like to see if the numbers change, I just want to see, like.
169 00:19:27.480 ⇒ 00:19:39.079 Emily Giant: given some of the tweaks that I’ve made, and I’m about to go over the biggest one, to get stakeholder sign-off or not, I just want to make sure that
170 00:19:39.370 ⇒ 00:19:47.010 Emily Giant: historical revenue, the boat is not rocked. It shouldn’t move the much. …
171 00:19:47.190 ⇒ 00:19:53.139 Emily Giant: that change, if there’s a change, will come during the revenue mart. This shouldn’t affect it, but
172 00:19:53.570 ⇒ 00:20:04.859 Emily Giant: one of the major changes here that I wanted to review with everyone was that, currently the way that non-florals are set up in dbt and are flowing to Looker is that it is
173 00:20:05.060 ⇒ 00:20:09.900 Emily Giant: Putting the, … to start out with regular kits, like, not…
174 00:20:09.900 ⇒ 00:20:29.739 Emily Giant: like, flowers, bouquet, bath bomb. Each item is represented on its own row. That’s not the case with our plants. So, with plants, it takes the plant, it takes the pot, and because they’re never sent separately, it sums the cost together, it sums the inventory together, it says it’s one unit instead of two.
175 00:20:29.990 ⇒ 00:20:47.579 Emily Giant: When you look back, historically, how we’ve sold kits, what that’s doing is taking legitimate plant kits with a candle, with a bath bomb, and saying it’s the same thing as the pot and the plant, and jamming it all together, and …
176 00:20:47.750 ⇒ 00:20:51.960 Emily Giant: misrepresenting inventory, as I’m sure you can imagine with, like.
177 00:20:51.970 ⇒ 00:21:10.509 Emily Giant: the thought was that, okay, we don’t send plants without pots, but this just isn’t a fact, like, across urban stems and how we sell non-florals. So, a lot of the revenue issues and a lot of the, Stephanie calling out that, like, this plant is type other. Why?
178 00:21:10.660 ⇒ 00:21:18.420 Emily Giant: it’s that. It’s that we have decided to put together things that are separate. So, … one of…
179 00:21:18.570 ⇒ 00:21:34.910 Emily Giant: the largest changes with this deployment would be that the plant and the pot are going to be separated out again. I’ve done extensive testing, I haven’t come across instances where you can’t pull both at once as, like, a parent SKU.
180 00:21:35.270 ⇒ 00:21:54.169 Emily Giant: But I wanted to float that to the team, and just ask if they see any major implications with their reporting, especially, Stephanie. I know that you do a lot of component reporting, and those two things in component reporting are gonna be broken out. It won’t be the same thing anymore. And if you need me to, like.
181 00:21:54.480 ⇒ 00:22:06.460 Emily Giant: create an only from, like, X point forward model, so that those are always together. We can talk through that, but just wanted to, like, float that to the team, so that I can, like, gather some downstream implications.
182 00:22:06.460 ⇒ 00:22:14.719 Stephanie Plaza: Yeah. So it would just… it would represent almost like a floral bundle then, where it’s, like, the full together, and then the plant separate, and the pot separate.
183 00:22:14.890 ⇒ 00:22:16.070 Emily Giant: Okay.
184 00:22:16.770 ⇒ 00:22:30.779 Stephanie Plaza: I’m gonna message Felipe, too, because I know that he uses that. I don’t see that being a problem. It would just be what, like, parent product plan, and then piece product plan to get only the plant sales, and not, like, doubled up.
185 00:22:31.510 ⇒ 00:22:33.190 Emily Giant: Exactly. Yep.
186 00:22:33.190 ⇒ 00:22:35.870 Stephanie Plaza: Yeah, I mean, I don’t think that that would be a problem on, like.
187 00:22:36.230 ⇒ 00:22:42.189 Stephanie Plaza: our reporting, and I’m just gonna message Felipe and see, if he has any thoughts on that, because I don’t think that should be….
188 00:22:43.250 ⇒ 00:22:47.860 Emily Giant: Yeah, … To me, it looks like the reporting will be…
189 00:22:47.860 ⇒ 00:23:05.539 Emily Giant: even historically, a lot easier and more consistent with it that way. Whenever you try to, like, put those layers over things, there’s always going to be outliers. It’s easier to, like, have them represented separately, and then give you the option of seeing them together, as opposed to, like, never giving the user the
190 00:23:05.540 ⇒ 00:23:22.530 Emily Giant: the option of seeing them separately. So, like, with Brainforge, we’re opting a lot towards that. Like, even with forced upgrades, we want to give you the option of saying, like, this is what was ordered, this is what was sent, as opposed to, like, putting a data layer over it that obscures, like, the journey of fulfillment.
191 00:23:22.530 ⇒ 00:23:30.719 Emily Giant: So that’s what I’m trying to do here. Also, just to normalize revenue in some of the years that we’ve seen it, like, jump. Some of that was…
192 00:23:30.880 ⇒ 00:23:37.380 Emily Giant: Very much due to us changing how we reported on plants without normalizing historical data.
193 00:23:39.190 ⇒ 00:23:40.530 Stephanie Plaza: Gotcha, okay.
194 00:23:41.280 ⇒ 00:23:47.759 Emily Giant: And the only issue I’m still seeing, like, so, for example, if you are in our production instance, and you count, like.
195 00:23:48.220 ⇒ 00:23:58.429 Emily Giant: duplicate components that shouldn’t be duplicates, you’re gonna get about 28,000. In my production instance, or excuse me, in my staging, like.
196 00:23:58.650 ⇒ 00:24:14.460 Emily Giant: I get 500, and every single one of them is a forced upgrade on an add-on on a plant. So, like, we’re getting down to, like, the nitty-gritty of, like, what actually happened in the fulfillment cycle, so it should be a lot clearer.
197 00:24:14.800 ⇒ 00:24:19.619 Emily Giant: But it’s not gonna be without some kinks until…
198 00:24:19.770 ⇒ 00:24:22.589 Emily Giant: what Demolade is doing with revenue is
199 00:24:22.880 ⇒ 00:24:27.039 Emily Giant: completed. Yeah. Yeah, and the reason being that, like.
200 00:24:27.820 ⇒ 00:24:39.050 Emily Giant: Historically, like, since the beginning of how we report on line item and component data, there have been issues, with forced upgrades.
201 00:24:39.300 ⇒ 00:24:46.280 Emily Giant: Forced upgrades counting towards revenue in reporting. … And, essentially, we’re…
202 00:24:46.340 ⇒ 00:25:05.229 Emily Giant: trying to replace all of that line item data where the force-upgraded product is not being represented correctly by using native Shopify tables. Like, streamlining the historical data, since we don’t have Shopify data for, like, pre-11724,
203 00:25:05.250 ⇒ 00:25:14.480 Emily Giant: So that’s gonna be a challenge, is, like, getting revenue right in the past, but, like, huge disclaimer that it’s actually never been correct because of forced upgrades.
204 00:25:15.490 ⇒ 00:25:23.120 Stephanie Plaza: So, currently, there are, like, non-revenue recognized orders of plants going out without a pot, as well.
205 00:25:23.450 ⇒ 00:25:25.349 Stephanie Plaza: Or no? Everything….
206 00:25:26.350 ⇒ 00:25:27.180 Emily Giant: ….
207 00:25:27.180 ⇒ 00:25:30.960 Stephanie Plaza: So everything is, like, every order does go out with the pot.
208 00:25:30.960 ⇒ 00:25:31.700 Emily Giant: Yes.
209 00:25:33.310 ⇒ 00:25:34.210 Stephanie Plaza: Okay.
210 00:25:34.210 ⇒ 00:25:34.820 Emily Giant: Yeah.
211 00:25:35.350 ⇒ 00:25:50.229 Emily Giant: The forced upgrades with plants are just add-ons. It has nothing to do with the plant. Those can’t go one without the other. But what I’m seeing in my staging is that, like, here, for example, you can see a good example here, if I’m able to share…
212 00:25:51.130 ⇒ 00:25:51.910 Emily Giant: Yeah.
213 00:25:54.730 ⇒ 00:25:57.419 Emily Giant: Okay, so, can everyone see my screen?
214 00:25:57.980 ⇒ 00:25:58.630 Stephanie Plaza: Yeah.
215 00:25:59.120 ⇒ 00:26:00.200 Emily Giant: So, what this…
216 00:26:00.260 ⇒ 00:26:19.519 Emily Giant: What this query is here is me running, in my staging, any component IDs that there are more than one of. So, I pulled this example here and looked at it in dash, so this is just, like, a test that we build in to make sure that things don’t blow up. But if I run…
217 00:26:19.880 ⇒ 00:26:35.209 Emily Giant: the table components XF, I’m like, okay, well, I can see from… the snapshot is, like, combining different fields for that line and saying, are these different? So right off the bat, I can tell these numbers are different, so I know that these aren’t actually duplicates. Like, something is…
218 00:26:35.770 ⇒ 00:26:43.549 Emily Giant: wrong with how the component ID was built, because these should be two different numbers. And when I go down, I’m like, yeah, okay, these are different.
219 00:26:44.530 ⇒ 00:26:52.020 Emily Giant: But, there should be an indication in here that one of them was not sent. That’s what’s going on, because when I look in Dash.
220 00:26:55.090 ⇒ 00:26:58.780 Emily Giant: It’s overriding… …
221 00:26:59.480 ⇒ 00:27:10.240 Emily Giant: the SKU for… essentially, it’s saying that these two were both sent, and not one and the other. So that’s the issue I’m still running into, since I am…
222 00:27:10.850 ⇒ 00:27:16.859 Emily Giant: not able to pull… like, this whole order was canceled. I’m not able to pull forward
223 00:27:17.650 ⇒ 00:27:25.820 Emily Giant: the tag of the SKU that was crossed out with how it’s set up right now. That’s not saying that I can’t, but it’s, …
224 00:27:26.310 ⇒ 00:27:33.130 Emily Giant: not, like, native to the table as it was built now or in the past, but they’re, like.
225 00:27:33.320 ⇒ 00:27:38.740 Emily Giant: Ideally would be a column that says, is canceled, and it would say, true.
226 00:27:39.170 ⇒ 00:27:51.780 Stephanie Plaza: Felipe, his response back was, it likely won’t work. We upload the finished item to Dash, although pots show up in Dash, the plants do not. But it would be from, like, a…
227 00:27:52.400 ⇒ 00:27:55.820 Stephanie Plaza: the parent, like, the dinero, for example.
228 00:27:56.840 ⇒ 00:28:01.939 Stephanie Plaza: component would then be the dinero, and then would be, like, the elephant pot.
229 00:28:02.120 ⇒ 00:28:05.279 Stephanie Plaza: Right? Or it would be, like, 3-inch succulent.
230 00:28:06.400 ⇒ 00:28:07.800 Emily Giant: Let me see if I can pull an order.
231 00:28:07.800 ⇒ 00:28:13.670 Alex K: Just to make sure we’re talking the same thing here, … we’re… the…
232 00:28:13.790 ⇒ 00:28:20.309 Alex K: Plants are sold and counted for in inventory as a completed unit.
233 00:28:20.310 ⇒ 00:28:21.250 Emily Giant: Yes.
234 00:28:21.650 ⇒ 00:28:25.549 Alex K: Yeah, so they’re… they’re not split up in any of the Shopify data.
235 00:28:25.830 ⇒ 00:28:33.910 Alex K: that’s coming back. And so this, just to make sure I’m on the same page, too, what we’re talking about now is how do we pair that with the NetSuite inventory?
236 00:28:35.040 ⇒ 00:28:35.660 Emily Giant: Yes.
237 00:28:36.240 ⇒ 00:28:37.569 Alex K: I’m on the same page now.
238 00:28:37.750 ⇒ 00:28:51.970 Emily Giant: Yes. So, essentially, if we were looking at a plant SKU, which, if someone can quickly, in the next one minute, shoot me an order for a plant, or Steph, I can even screen record for you, parent SKU would be, like, nk-
239 00:28:52.380 ⇒ 00:29:05.490 Emily Giant: whatever, like, NF-K for both of these. And then in the PSKU, it’s gonna be the plant here, and the pot here. And the parent quantity is not gonna be 2. It’s gonna be 1.
240 00:29:06.140 ⇒ 00:29:14.970 Emily Giant: You’re gonna count one. However, the piece quantity is gonna be two, because you’re gonna have sent one of the plants, and one of the, …
241 00:29:15.340 ⇒ 00:29:28.049 Emily Giant: one of the pots. So, it’s going to be some, like, working together to rebuild these reports so that you’re pulling exactly what it is that you want, but you’re gonna have the option to do both.
242 00:29:29.240 ⇒ 00:29:37.030 Stephanie Plaza: Okay. Yeah, because, like, from our purposes, we don’t need to see, like, 3-inch pink orchid, but having…
243 00:29:37.200 ⇒ 00:29:44.330 Stephanie Plaza: the name of the full thing as, like, a piece product, and then the pot as a piece product. That makes sense to me.
244 00:29:44.550 ⇒ 00:29:49.619 Emily Giant: Yeah, so you’ll just want to pull the parent quantity and the parent product.
245 00:29:49.700 ⇒ 00:29:50.610 Stephanie Plaza: And….
246 00:29:50.610 ⇒ 00:29:57.260 Emily Giant: Those will still have the entire revenue and product cost of both.
247 00:29:57.440 ⇒ 00:30:07.170 Emily Giant: But, for example, if Santi wanted to know, like, how many of this pot sold from, … Baltimore.
248 00:30:07.610 ⇒ 00:30:15.410 Emily Giant: You could just look at the inventory sales against the pot, since we don’t necessarily have the same amount of pots and plants.
249 00:30:15.960 ⇒ 00:30:16.670 Stephanie Plaza: Gotcha, okay.
250 00:30:16.670 ⇒ 00:30:25.150 Emily Giant: It could be, like, the dual purpose. So the parent product exists for, like, more revenue reporting, and the peace product exists more for, like, SCM reporting.
251 00:30:25.290 ⇒ 00:30:26.980 Stephanie Plaza: Okay. Cool.
252 00:30:27.550 ⇒ 00:30:28.460 Emily Giant: Cool.
253 00:30:28.660 ⇒ 00:30:35.380 Emily Giant: Yeah, and then also for, like, the product cost standpoint, prior, like, we couldn’t really parse out, like.
254 00:30:37.570 ⇒ 00:30:52.409 Emily Giant: like, sums of cost, because they were always jammed together for plants, which is fine. It’s a fine way to do it, because we know that that’s the entire cost of the product, to send it. One does not go without the other. But if we were doing, like, a more…
255 00:30:52.990 ⇒ 00:30:58.840 Emily Giant: hard goods-specific type of analysis. You might just want to see the pot.
256 00:31:01.150 ⇒ 00:31:13.710 Emily Giant: But we’ll work together. I just wanted to give a heads up that, like, those changes are coming down the pipeline in short order. So especially just reminding everyone, keep an eye on your historical
257 00:31:13.710 ⇒ 00:31:26.539 Emily Giant: like, comp data, please send me any reports that you, like, use on a daily basis that would look further back than the point of migration, because those are where you’re going to see the changes, not in anything past 11-7-2024.
258 00:31:30.020 ⇒ 00:31:31.030 Stephanie Plaza: Awesome.
259 00:31:31.270 ⇒ 00:31:36.300 Emily Giant: Cool. Alright, everyone, thank you so much, and …
260 00:31:36.530 ⇒ 00:31:47.839 Emily Giant: yeah, if you want any, like, when you see new fields pop up, if you want to throw time on my calendar to just, like, review, what does this mean? I’m trying to put definitions in as I go, but, …
261 00:31:48.340 ⇒ 00:31:57.320 Emily Giant: we’re trying to make it easier, not more difficult, and if I’m making it more difficult, please throw time on my calendar. Sorry, we’ve gone over. Goodbye, everybody, thank you so much!
262 00:31:58.590 ⇒ 00:31:59.130 Santiago Posso: Bye-bye.
263 00:31:59.130 ⇒ 00:31:59.630 Demilade Agboola: here.
264 00:31:59.960 ⇒ 00:32:00.550 Demilade Agboola: Goodbye.
265 00:32:00.550 ⇒ 00:32:01.260 Ian Biles: I guess.
266 00:32:02.020 ⇒ 00:32:02.800 Stephanie Plaza: Pew.