Meeting Title: US x BF | Standup Date: 2025-11-06 Meeting participants: Awaish Kumar, Uttam Kumaran, Emily Giant
WEBVTT
1 00:06:21.010 ⇒ 00:06:22.710 Uttam Kumaran: Hello, hello.
2 00:06:23.070 ⇒ 00:06:23.960 Emily Giant: Hi!
3 00:06:24.460 ⇒ 00:06:25.590 Uttam Kumaran: Good morning.
4 00:06:26.500 ⇒ 00:06:27.760 Emily Giant: How’s it going?
5 00:06:28.170 ⇒ 00:06:33.620 Uttam Kumaran: Good Got some… getting some more stuff out today. How was yesterday?
6 00:06:34.160 ⇒ 00:06:41.039 Emily Giant: Good, I had to work on some inventory stuff for,
7 00:06:41.260 ⇒ 00:06:48.260 Emily Giant: SNOP, but it’s all but done. I’m just running the tests, and Demolati already reviewed the bulk of it, so…
8 00:06:48.390 ⇒ 00:06:49.240 Emily Giant: It’s looking good.
9 00:06:49.240 ⇒ 00:06:49.880 Uttam Kumaran: Cool.
10 00:06:50.320 ⇒ 00:06:55.369 Emily Giant: Yep, just working on the Looker stuff for Awash right now.
11 00:06:55.580 ⇒ 00:07:01.140 Emily Giant: It’s a good, like, in-between meetings type of thing to do. So, working through that.
12 00:07:02.050 ⇒ 00:07:07.520 Emily Giant: Yeah, and then, Demolati and I are chipping away at the edge cases with revenue, but it’s getting really close.
13 00:07:08.110 ⇒ 00:07:08.730 Uttam Kumaran: Right.
14 00:07:12.380 ⇒ 00:07:13.270 Uttam Kumaran: Bye.
15 00:07:14.730 ⇒ 00:07:20.120 Uttam Kumaran: Awish, anything from your side? I just saw some of your… comments, so…
16 00:07:21.010 ⇒ 00:07:29.249 Awaish Kumar: So… I, like… just fix that schema for PR runs.
17 00:07:31.080 ⇒ 00:07:35.119 Awaish Kumar: And then I analyzed the logs also, like, the…
18 00:07:35.240 ⇒ 00:07:40.729 Awaish Kumar: A few models which are taking than more… taking more than 10, 12 minutes to execute.
19 00:07:41.240 ⇒ 00:07:47.080 Awaish Kumar: And then I have been working on North Beam injection part. Ingestation is done.
20 00:07:49.710 ⇒ 00:07:53.710 Awaish Kumar: I will be next working on this Northwametribution ticket.
21 00:07:53.820 ⇒ 00:07:57.620 Awaish Kumar: Which will let you… powers to the…
22 00:07:58.050 ⇒ 00:08:01.130 Awaish Kumar: channels from… which are… which are coming from Northwind.
23 00:08:01.450 ⇒ 00:08:11.589 Awaish Kumar: And, apart from that, yeah, I had a meeting with Emily yesterday. We discussed about local migration plan.
24 00:08:11.950 ⇒ 00:08:17.999 Awaish Kumar: And, what I… we’ve… Like, basically decided that we…
25 00:08:18.110 ⇒ 00:08:21.710 Awaish Kumar: are going to filter, like, we have a long list of dashboards.
26 00:08:24.020 ⇒ 00:08:27.599 Awaish Kumar: Some of them might just need to be deprecated.
27 00:08:27.800 ⇒ 00:08:32.950 Awaish Kumar: So we’ll just depicate those, like, mark them for deprecation.
28 00:08:33.159 ⇒ 00:08:35.059 Awaish Kumar: And then,
29 00:08:35.620 ⇒ 00:08:44.820 Awaish Kumar: assign priorities on the different dashboards, like, which ones are actually high priority, and which ones are P0, P1, P2.
30 00:08:45.060 ⇒ 00:08:47.740 Awaish Kumar: And then, start from P0.
31 00:08:48.040 ⇒ 00:09:01.549 Awaish Kumar: Gate Explorers created, like, the duplicate Explorer, as the existing one, but which basically uses the new data models, which we are creating.
32 00:09:01.770 ⇒ 00:09:08.480 Awaish Kumar: And, create the duplicated dashboard. Finally, Compare them…
33 00:09:09.730 ⇒ 00:09:15.310 Awaish Kumar: Deploy it, and then, like, incrementally iterate through all our
34 00:09:15.550 ⇒ 00:09:19.089 Awaish Kumar: Through all over, like, P0, P1, P2 dashboards.
35 00:09:20.140 ⇒ 00:09:25.810 Awaish Kumar: So right now, there is step one, where we are just… we’ll be assigning deed.
36 00:09:26.690 ⇒ 00:09:34.970 Awaish Kumar: like, figuring out which one needs to be duplicated, and which are the ones which are P0, and then we start from there.
37 00:09:36.030 ⇒ 00:09:36.600 Uttam Kumaran: Okay.
38 00:09:39.960 ⇒ 00:09:41.079 Uttam Kumaran: Okay, great.
39 00:09:43.010 ⇒ 00:09:46.790 Uttam Kumaran: And these are not coming… these are not coming from Metaplane, or…
40 00:09:49.590 ⇒ 00:09:50.520 Awaish Kumar: Sorry, what?
41 00:09:51.610 ⇒ 00:09:55.320 Uttam Kumaran: Like, are these errors that are coming from Metaplane, or these are just other errors?
42 00:09:56.810 ⇒ 00:10:02.070 Awaish Kumar: I’m not talking about Arabs, I’m talking about Looker dashboards, which we want to migrate.
43 00:10:02.070 ⇒ 00:10:07.180 Uttam Kumaran: Oh, okay, sorry, I heard P0, I thought you were talking more about, like, the issues. Okay, okay.
44 00:10:07.360 ⇒ 00:10:10.819 Awaish Kumar: No, no, we want to assign priority to dashboards, like, for example.
45 00:10:10.820 ⇒ 00:10:12.120 Uttam Kumaran: Yes, correct, correct, correct.
46 00:10:12.120 ⇒ 00:10:16.530 Awaish Kumar: I use priority, let’s migrate them first, and then others.
47 00:10:17.660 ⇒ 00:10:18.920 Uttam Kumaran: Okay, perfect.
48 00:10:20.790 ⇒ 00:10:21.150 Emily Giant: Yeah.
49 00:10:21.150 ⇒ 00:10:27.879 Uttam Kumaran: Yeah, I’m a fan of that, and I think there’s a lot of concentration about which dashboards are, like, P0, so…
50 00:10:28.140 ⇒ 00:10:29.419 Emily Giant: It’ll be pretty clear.
51 00:10:30.130 ⇒ 00:10:37.799 Emily Giant: Yep. There’s so many that are just, like, low, delete. Like, should I even put low, or should I just write delete, as far as priority?
52 00:10:37.800 ⇒ 00:10:39.950 Uttam Kumaran: Billy, I mean, we don’t need to have more…
53 00:10:40.510 ⇒ 00:10:44.179 Uttam Kumaran: Ideally, there are no P3, P4, you know, but… Yeah.
54 00:10:44.180 ⇒ 00:10:44.880 Emily Giant: Okay.
55 00:10:45.340 ⇒ 00:10:56.529 Emily Giant: because we were doing one column for priority, and one column for action, but, like, I don’t even want to write low next to the ones that are gonna get deleted, because…
56 00:10:56.810 ⇒ 00:11:01.329 Emily Giant: That implies that there’s any priority at all. Yeah. And there isn’t.
57 00:11:03.670 ⇒ 00:11:07.130 Emily Giant: Okay, cool. Well, that will be very easy to update.
58 00:11:07.250 ⇒ 00:11:08.210 Emily Giant: But…
59 00:11:08.740 ⇒ 00:11:21.020 Emily Giant: Yeah, as far as the high priority ones, I should be done soon. I know there’s 900, but because I did this exercise somewhat with, Kayo back in the day, I’ve got a lot of info on
60 00:11:21.450 ⇒ 00:11:23.279 Emily Giant: Like, what just needs to go?
61 00:11:23.600 ⇒ 00:11:26.270 Uttam Kumaran: Yeah, and we actually have that,
62 00:11:27.430 ⇒ 00:11:31.640 Uttam Kumaran: demo, Awash, did you happen to see that? We have it in our…
63 00:11:32.280 ⇒ 00:11:36.300 Awaish Kumar: We are, like, using the… our documentation for databases.
64 00:11:36.300 ⇒ 00:11:37.050 Uttam Kumaran: Okay, okay, cool.
65 00:11:37.050 ⇒ 00:11:38.230 Awaish Kumar: Not building on top.
66 00:11:39.040 ⇒ 00:11:40.090 Uttam Kumaran: Great, perfect, perfect.
67 00:11:40.950 ⇒ 00:11:41.720 Uttam Kumaran: Amazing.
68 00:11:42.420 ⇒ 00:11:48.249 Emily Giant: Yeah. There was a… I still can’t find the tab that…
69 00:11:48.390 ⇒ 00:11:55.189 Emily Giant: we ranked them. Like, all of it was ranked at one point, and, like, color-coded, and I don’t know where that tab went, but…
70 00:11:56.920 ⇒ 00:11:57.480 Emily Giant: But…
71 00:11:57.480 ⇒ 00:12:09.270 Uttam Kumaran: Is it, well, is it the dashboards audit? Like, it looks like there’s some hidden, .
72 00:12:11.910 ⇒ 00:12:13.800 Emily Giant: Oh, maybe if there’s hidden stuff…
73 00:12:15.060 ⇒ 00:12:18.640 Uttam Kumaran: Let’s see… No, these aren’t…
74 00:12:18.750 ⇒ 00:12:22.330 Uttam Kumaran: These don’t have ranks in them, necessarily.
75 00:12:22.330 ⇒ 00:12:26.409 Emily Giant: Yeah, cause there was a whole, like… Number system that was…
76 00:12:27.480 ⇒ 00:12:28.070 Uttam Kumaran: Yeah.
77 00:12:28.070 ⇒ 00:12:29.880 Emily Giant: You remember that?
78 00:12:30.170 ⇒ 00:12:36.790 Uttam Kumaran: I do. But I feel like maybe we did that for…
79 00:12:39.820 ⇒ 00:12:45.470 Uttam Kumaran: I’m gonna think, maybe we didn’t do that for Dash, we did it for, like, another object.
80 00:12:48.660 ⇒ 00:12:51.620 Emily Giant: Oh, we did, but either way.
81 00:12:51.620 ⇒ 00:12:53.819 Uttam Kumaran: I have a lot of reference, so it shouldn’t…
82 00:12:53.920 ⇒ 00:12:55.749 Emily Giant: It shouldn’t take very long.
83 00:12:55.750 ⇒ 00:13:03.720 Uttam Kumaran: I feel like we actually… this is actually a great place, we did a lot of… I mean, I’m actually… oh, oh, we did… oh, we did a ranking for the ingestions.
84 00:13:03.950 ⇒ 00:13:06.120 Emily Giant: I don’t think we did one.
85 00:13:06.180 ⇒ 00:13:09.040 Uttam Kumaran: And we did it for DBT, I don’t think we did one for…
86 00:13:09.040 ⇒ 00:13:09.454 Emily Giant: Hmm…
87 00:13:10.220 ⇒ 00:13:16.809 Uttam Kumaran: the dash, but, yeah, Wage, if you look at, like, the DBT audit and, like, the ingestion audit.
88 00:13:17.640 ⇒ 00:13:20.689 Uttam Kumaran: Some of these… I think some of these statuses are really, really helpful.
89 00:13:23.430 ⇒ 00:13:29.480 Uttam Kumaran: But yeah, I mean, this is just coming in handy. It was worth, like, do… spending that time back then.
90 00:13:29.850 ⇒ 00:13:30.610 Emily Giant: Yes.
91 00:13:30.610 ⇒ 00:13:31.440 Uttam Kumaran: For sure.
92 00:13:31.950 ⇒ 00:13:37.290 Emily Giant: Yeah, it’ll make this really fast, so I’m, I’m just, like, burning through it.
93 00:13:37.800 ⇒ 00:13:38.310 Uttam Kumaran: Okay.
94 00:13:41.410 ⇒ 00:13:45.120 Awaish Kumar: So that’s good, then today for me.
95 00:13:45.980 ⇒ 00:13:57.290 Emily Giant: I’m working on this, of course, the… the dashboard updates, because there are new dashboards that… we did that a long time ago, before, before some of the stuff was up and running.
96 00:13:57.600 ⇒ 00:13:59.570 Emily Giant: But,
97 00:13:59.950 ⇒ 00:14:06.789 Emily Giant: I have a lot of, like, ad hoc tickets in Linear that I still need to do, but I was gonna work on the refund thing, the historical refund table.
98 00:14:07.280 ⇒ 00:14:12.369 Emily Giant: So that I can pass that off to Awash once the foundation is built.
99 00:14:12.550 ⇒ 00:14:13.140 Uttam Kumaran: Okay.
100 00:14:13.740 ⇒ 00:14:14.300 Emily Giant: Yeah.
101 00:14:16.830 ⇒ 00:14:22.240 Emily Giant: Yeah, but no, all good, and I know Joe Milotti’s working on, revenue, so…
102 00:14:22.240 ⇒ 00:14:24.450 Uttam Kumaran: Yeah, and I guess, how’s the North Beam stuff?
103 00:14:24.570 ⇒ 00:14:25.230 Uttam Kumaran: Away.
104 00:14:26.820 ⇒ 00:14:33.109 Awaish Kumar: Yeah, I… like, congested the North Beam data?
105 00:14:33.650 ⇒ 00:14:36.019 Awaish Kumar: As I said before, it is,
106 00:14:36.310 ⇒ 00:14:40.480 Awaish Kumar: From Northstream, it is coming in to the…
107 00:14:41.000 ⇒ 00:14:45.170 Awaish Kumar: S3, and from S3, it is moving to that shift.
108 00:14:48.030 ⇒ 00:14:51.839 Awaish Kumar: And then, like, today, I will be working on modeling it, and…
109 00:14:51.840 ⇒ 00:14:52.380 Uttam Kumaran: Okay.
110 00:14:52.710 ⇒ 00:14:55.749 Awaish Kumar: Adding it to the fact order, basically.
111 00:14:56.430 ⇒ 00:14:58.460 Uttam Kumaran: Okay, you mind sending a quick…
112 00:14:58.960 ⇒ 00:15:02.279 Uttam Kumaran: Yeah, note to Kristen, like, just on progress.
113 00:15:02.670 ⇒ 00:15:06.209 Uttam Kumaran: Just want to make sure she hears from us that we’re getting it done.
114 00:15:07.710 ⇒ 00:15:09.210 Awaish Kumar: Okay, sure, I understand.
115 00:15:09.210 ⇒ 00:15:10.480 Uttam Kumaran: Whenever, yeah.
116 00:15:11.420 ⇒ 00:15:17.029 Emily Giant: Do we want to look at, your PR together, Utam, so that we can deploy that? It’s been sitting.
117 00:15:17.030 ⇒ 00:15:17.829 Uttam Kumaran: for a while.
118 00:15:17.830 ⇒ 00:15:19.250 Emily Giant: I wanna make sure we get that out.
119 00:15:19.250 ⇒ 00:15:21.149 Uttam Kumaran: Let’s just… let’s just do that right now.
120 00:15:21.520 ⇒ 00:15:22.220 Emily Giant: Okay.
121 00:15:22.670 ⇒ 00:15:24.820 Uttam Kumaran: I’m going to pull it up on my side.
122 00:15:24.820 ⇒ 00:15:25.970 Emily Giant: Oh, go for it.
123 00:15:27.490 ⇒ 00:15:31.129 Emily Giant: I just shared, but it would make more sense if you did, so that you can…
124 00:15:32.370 ⇒ 00:15:36.820 Uttam Kumaran: Yeah, and I’ll just… I just also want to just share, I think… I don’t know, Wish, how…
125 00:15:37.470 ⇒ 00:15:46.829 Uttam Kumaran: I don’t know if… how much you worked with retchiv. I think you’ve done some retchiv work a while, you know, whenever, but I wanted to share, Emily, for you, sort of, like, the importance of.
126 00:15:46.830 ⇒ 00:15:47.350 Emily Giant: Yeah.
127 00:15:47.350 ⇒ 00:15:50.459 Uttam Kumaran: It’s like this sort key, so let me just find that PR and get that.
128 00:15:50.460 ⇒ 00:15:53.380 Emily Giant: Yeah, I read that and was like, this is great.
129 00:15:53.570 ⇒ 00:15:57.590 Emily Giant: But I know we were holding on deploying it because there were some errors, but I think that, like.
130 00:15:57.590 ⇒ 00:15:59.349 Uttam Kumaran: Yeah, let’s just rip through it.
131 00:15:59.350 ⇒ 00:16:00.240 Emily Giant: Yeah.
132 00:16:01.610 ⇒ 00:16:02.720 Awaish Kumar: Yeah. Oh, shit.
133 00:16:03.170 ⇒ 00:16:05.629 Awaish Kumar: Just keys are, like, partitioning.
134 00:16:07.340 ⇒ 00:16:15.549 Uttam Kumaran: Yeah, it’s interesting, and it kind of sucks that, like, they… you have to do this, but it’s actually a huge issue that these aren’t set up.
135 00:16:15.940 ⇒ 00:16:19.710 Uttam Kumaran: Because, these would run just, like, way faster.
136 00:16:20.290 ⇒ 00:16:30.220 Uttam Kumaran: So… Kinda let me walk you guys through how I did this, so…
137 00:16:31.280 ⇒ 00:16:37.999 Uttam Kumaran: I use a mix of cursor and, Redshift query logs to show
138 00:16:38.100 ⇒ 00:16:47.419 Uttam Kumaran: like, basically find the tables with the largest number of rows, and then identify, through Redshift, which does or doesn’t have dist or short keys.
139 00:16:47.550 ⇒ 00:16:56.359 Uttam Kumaran: And, I also put in a little thing here about, like, they’re actually put in somewhere about in… in cursor.
140 00:16:56.920 ⇒ 00:17:01.489 Uttam Kumaran: Why we use disks or short keys? Yeah, like.
141 00:17:01.800 ⇒ 00:17:06.300 Uttam Kumaran: So let me see if I can even just pull this, so…
142 00:17:07.280 ⇒ 00:17:22.230 Uttam Kumaran: so that way, I like actually… I like… I’m liking writing this documentation, just shoving it into here, but it… this should be helpful for, like, why use… why sort keys are helpful, and why use IDs, like, versus dates. The sort keys are basically, like.
143 00:17:24.099 ⇒ 00:17:36.510 Uttam Kumaran: you want your table to be sorted in the way that it’s going to be queried. Ideally, those match. But for example, sorting on ID doesn’t really matter, because IDs are not, like, alphabetical, but sorting on dates.
144 00:17:36.730 ⇒ 00:17:44.680 Uttam Kumaran: Sorting on, you know, geos, or, like, whatever is commonly queried, or commonly put into a WHERE clause.
145 00:17:44.900 ⇒ 00:17:47.060 Uttam Kumaran: Is very helpful.
146 00:17:47.530 ⇒ 00:17:56.519 Uttam Kumaran: And so that’s… that’s sort of, like, what we try to do is, like, for example, do you commonly want to sort transactions by transaction date and status?
147 00:17:57.150 ⇒ 00:18:00.870 Uttam Kumaran: And so these are, like, helpful. These won’t, like, make or break anything.
148 00:18:01.010 ⇒ 00:18:09.160 Uttam Kumaran: But they can help queries run, like, way faster on Redshift. These are not great keys, because
149 00:18:09.390 ⇒ 00:18:17.070 Uttam Kumaran: they just sort them in, like, by the IDs, and the IDs don’t mean it, like, they’re not, like, in a… they’re just a random set of characters, right? So…
150 00:18:17.460 ⇒ 00:18:35.919 Uttam Kumaran: I’ve just put in some examples of, like, here’s, like, what we had before, and we sorted by adjustment date and inventory ID, but we don’t need to sort by inventory number ID, it’s probably… unless this is, like, 1234, it’s probably best to sort by, like, this, or something that’s, like.
151 00:18:36.230 ⇒ 00:18:38.530 Uttam Kumaran: Commonly filtered on.
152 00:18:39.570 ⇒ 00:18:40.370 Emily Giant: I’ll start.
153 00:18:40.370 ⇒ 00:18:46.449 Uttam Kumaran: Yeah. Yeah, exactly. So, so that’s what I would suggest, is, like, we start here, and then,
154 00:18:46.650 ⇒ 00:19:00.910 Uttam Kumaran: we’re gonna… some of these tables we’ll remove, of course, but as we think about, like, large tables, this is, like, a must-have, and this will really get us a lot of incremental benefit. And so all I’ve done is just gone into
155 00:19:01.100 ⇒ 00:19:10.939 Uttam Kumaran: I basically looked through the hour-long… hour of the full refresh, the 30-minute of the full refresh, and I basically just looked at, like, what are the… what are the rough tables that are…
156 00:19:10.940 ⇒ 00:19:11.460 Emily Giant: Yeah.
157 00:19:11.460 ⇒ 00:19:13.770 Uttam Kumaran: Taking the longest, and a lot of these, one.
158 00:19:13.930 ⇒ 00:19:21.669 Uttam Kumaran: if they’re not already incremental, I’ve, like, made that so. So, like, this one was incremental, but,
159 00:19:22.370 ⇒ 00:19:25.560 Uttam Kumaran: I… this is, actually BigQuery language.
160 00:19:27.870 ⇒ 00:19:39.370 Uttam Kumaran: Which was weird. I feel like I may have saw this, like, months ago, and I was like, oh, what the heck? But yeah, this is… this doesn’t do anything, actually, right now. So, I’ve added this, so this will be how we, sort of.
161 00:19:39.890 ⇒ 00:19:51.189 Uttam Kumaran: sort, and then similarly across the board of basically… some of these are just cleanup things, like making sure there is a… the dbt runs and things like that, and then, for example, inventory adjustments.
162 00:19:51.190 ⇒ 00:19:55.639 Emily Giant: Yeah, these are great. So, they already run fast, but they need to run faster.
163 00:19:55.870 ⇒ 00:19:59.190 Uttam Kumaran: Yeah, so these are all just put dist and sort keys on here.
164 00:20:01.030 ⇒ 00:20:08.189 Uttam Kumaran: And then… this one I just, I actually just cleaned up and made sure that there was… I just cleaned up the formatting a bit, actually.
165 00:20:08.190 ⇒ 00:20:08.620 Emily Giant: Yeah.
166 00:20:08.620 ⇒ 00:20:16.100 Uttam Kumaran: and turn this incremental, because we don’t need to be creating another table that is just a clone of Shopify orders.
167 00:20:16.100 ⇒ 00:20:16.680 Emily Giant: Yeah.
168 00:20:16.680 ⇒ 00:20:23.679 Uttam Kumaran: Right? Like, that’s… so that’s what’s happening. And all it’s doing is cleaning up and, like, renaming stuff, so… just be incremental, so… so that’ll…
169 00:20:23.840 ⇒ 00:20:25.050 Uttam Kumaran: Picks a lot.
170 00:20:25.160 ⇒ 00:20:29.369 Uttam Kumaran: similar, like, staging, legacy order deliveries. Basically, I looked at, like.
171 00:20:29.780 ⇒ 00:20:34.930 Uttam Kumaran: is there a reason not to make it incremental? And for super large tables.
172 00:20:35.480 ⇒ 00:20:40.970 Uttam Kumaran: where they… it’s just, like, a running log, they should, for the most part, all be incremental. For anything that’s, like, a…
173 00:20:41.140 ⇒ 00:20:47.459 Uttam Kumaran: Dimension table, and there’s no series, then it’s not really as relevant, right? So…
174 00:20:47.460 ⇒ 00:20:51.559 Emily Giant: Yeah, and that one, like, is never gonna change either, so it shouldn’t even run.
175 00:20:51.560 ⇒ 00:20:52.930 Uttam Kumaran: Exactly.
176 00:20:53.070 ⇒ 00:20:56.569 Uttam Kumaran: the mode for this, dim Google Analytics orders, right? It’s like…
177 00:20:56.570 ⇒ 00:20:57.020 Emily Giant: No.
178 00:20:57.040 ⇒ 00:21:00.120 Uttam Kumaran: This should totally just be an incremental table, sorted.
179 00:21:00.120 ⇒ 00:21:00.660 Emily Giant: Yep.
180 00:21:01.160 ⇒ 00:21:01.740 Uttam Kumaran: Yeah.
181 00:21:02.920 ⇒ 00:21:04.120 Emily Giant: Yeah.
182 00:21:04.300 ⇒ 00:21:05.649 Uttam Kumaran: But yeah, that’s just roughly…
183 00:21:05.650 ⇒ 00:21:06.020 Emily Giant: Yes.
184 00:21:06.020 ⇒ 00:21:06.670 Uttam Kumaran: Most of it.
185 00:21:07.740 ⇒ 00:21:11.819 Uttam Kumaran: For FedEx, I was like, I am allergic to select star, so I just…
186 00:21:11.820 ⇒ 00:21:12.550 Emily Giant: You know?
187 00:21:12.860 ⇒ 00:21:15.309 Emily Giant: I just cleaned that up, basically.
188 00:21:15.310 ⇒ 00:21:16.459 Uttam Kumaran: A little bit,
189 00:21:17.310 ⇒ 00:21:23.629 Uttam Kumaran: and remove… and make sure all the columns are aliased, and then I’m gonna be using this table for dipping stuff, so…
190 00:21:23.630 ⇒ 00:21:28.669 Emily Giant: Does it do anything? Oh, IPD, okay, yeah, great. No, this all looks good. What did it do?
191 00:21:28.670 ⇒ 00:21:34.159 Uttam Kumaran: Like, I don’t… I don’t… I don’t know if… I don’t know if any of this old data is, like, refreshing.
192 00:21:35.360 ⇒ 00:21:36.060 Uttam Kumaran: But, like.
193 00:21:36.060 ⇒ 00:21:37.280 Emily Giant: I don’t think it is, but…
194 00:21:37.280 ⇒ 00:21:40.650 Uttam Kumaran: I will figure it out when I go through the shipping sort of exploration.
195 00:21:40.650 ⇒ 00:21:43.080 Emily Giant: Yeah, I should do, like, a max whatever.
196 00:21:43.080 ⇒ 00:21:43.750 Uttam Kumaran: like, the last…
197 00:21:43.750 ⇒ 00:21:44.930 Emily Giant: Nothing.
198 00:21:45.770 ⇒ 00:21:46.690 Emily Giant: Okay, fabulous.
199 00:21:46.690 ⇒ 00:21:51.679 Uttam Kumaran: Because ideally, what we should do is a way for stuff like this, where it’s, like, no longer running.
200 00:21:51.810 ⇒ 00:21:56.800 Uttam Kumaran: We can move these tables to, like, a…
201 00:21:58.090 ⇒ 00:22:00.469 Uttam Kumaran: I mean, maybe they go into Legacy.
202 00:22:00.950 ⇒ 00:22:04.549 Uttam Kumaran: And then we create a job that just can run legacy stuff.
203 00:22:04.650 ⇒ 00:22:06.769 Emily Giant: When you need it, if you need it.
204 00:22:06.770 ⇒ 00:22:08.550 Uttam Kumaran: But, like, these should not be running.
205 00:22:08.890 ⇒ 00:22:10.150 Uttam Kumaran: These aren’t… there’s no… there’s no…
206 00:22:10.150 ⇒ 00:22:16.870 Emily Giant: Legacy tags, I think. In the project.yaml, a lot of the newer legacy ones have legacy tags, but then, like.
207 00:22:17.320 ⇒ 00:22:22.019 Emily Giant: like, at the end of our engagement, Pandera Paradigm should not exist.
208 00:22:22.020 ⇒ 00:22:22.870 Uttam Kumaran: Yeah.
209 00:22:22.870 ⇒ 00:22:25.040 Emily Giant: They should be moved into the new model.
210 00:22:25.360 ⇒ 00:22:28.200 Emily Giant: Section, and then, like, tagged in the…
211 00:22:28.700 ⇒ 00:22:33.799 Emily Giant: in the YAML, because I think tags in the config is a deprecated function, which is annoying.
212 00:22:34.370 ⇒ 00:22:35.840 Uttam Kumaran: Yes, yes.
213 00:22:36.540 ⇒ 00:22:40.040 Uttam Kumaran: But yeah, you’re gonna wanna tag… we’re gonna tag everything at the folder structure.
214 00:22:40.450 ⇒ 00:22:41.440 Emily Giant: Yeah. Level.
215 00:22:42.080 ⇒ 00:22:43.889 Uttam Kumaran: Okay, cool, so let me… I’m just gonna…
216 00:22:43.890 ⇒ 00:22:44.280 Emily Giant: What’s…
217 00:22:44.280 ⇒ 00:22:49.990 Uttam Kumaran: Yeah, just merge it. See what happens. Alright, I’ll fix this one thing, I don’t know what this is.
218 00:22:50.350 ⇒ 00:22:53.720 Emily Giant: It’s… snapshot shit that…
219 00:22:53.720 ⇒ 00:22:58.720 Uttam Kumaran: I do not use anymore. Ever. Okay, I’ll fix this locally, and then… yeah.
220 00:22:58.720 ⇒ 00:22:59.300 Emily Giant: Okay.
221 00:23:01.610 ⇒ 00:23:20.879 Emily Giant: Yeah, I had a… that’s why the runs were failing the other day, is because I had a new snapshot, and, like, it… the system does not like new snapshots, for some reason. I actually had a question about that, if y’all can take a look real quick. Yeah. I know we’re, like, way over time, but, so PK…
222 00:23:21.110 ⇒ 00:23:26.759 Emily Giant: Wanted snapshot data of, let me find this branch, one sec.
223 00:23:28.070 ⇒ 00:23:31.930 Emily Giant: He wanted snapshot data of, like.
224 00:23:32.260 ⇒ 00:23:41.109 Emily Giant: customers looking at certain SKUs, I think it’s in the aptly named GA4 Snapshot.
225 00:23:42.720 ⇒ 00:23:55.150 Emily Giant: I haven’t been running it because I’m fundamentally not understanding something about snapshots. Even though they’re, like, easy to make, they don’t like me. When I run it.
226 00:23:55.610 ⇒ 00:23:57.480 Emily Giant: It’s…
227 00:24:03.380 ⇒ 00:24:07.760 Emily Giant: Okay, there’s your PR coming through into my… yeah, yay.
228 00:24:07.840 ⇒ 00:24:26.339 Emily Giant: all it is is just, like, snapshotting this table here, but it throws an error in production, so I’ve just excluded it from all the jobs. And I don’t really have many questions outside of, does somebody have knowledge of snapshot data so that I can get this to work?
229 00:24:28.360 ⇒ 00:24:30.479 Uttam Kumaran: Yeah, Wish, you wanna… do you wanna take that?
230 00:24:34.760 ⇒ 00:24:35.790 Awaish Kumar: Okay, yup.
231 00:24:39.140 ⇒ 00:24:41.520 Uttam Kumaran: Yeah, maybe I can leave you guys to debug, and I’m just gonna push this.
232 00:24:41.520 ⇒ 00:24:41.990 Emily Giant: Yeah. So check.
233 00:24:42.290 ⇒ 00:24:42.900 Uttam Kumaran: Okay.
234 00:24:42.900 ⇒ 00:24:47.409 Emily Giant: Perfect. Alright, and I can already see the… Nicely updated.
235 00:24:48.540 ⇒ 00:24:54.910 Emily Giant: stuff here. Okay, so it… maybe it… go back and resolve errors. Now we got errors. What is it?
236 00:24:56.670 ⇒ 00:24:58.220 Awaish Kumar: Sorry, can I see?
237 00:25:04.770 ⇒ 00:25:07.130 Emily Giant: So, something is airing with FBI.
238 00:25:07.130 ⇒ 00:25:13.790 Uttam Kumaran: Oh, yeah, there… well, there… this… this warning is, I don’t think, is super… I don’t think the warning matters, but…
239 00:25:14.160 ⇒ 00:25:16.089 Uttam Kumaran: Yeah, the other thing is the issue.
240 00:25:17.710 ⇒ 00:25:26.150 Emily Giant: Oh, it’s the stupid… sorry, I shouldn’t say the word stupid. It’s the surrogate key, generate surrogate key. It just needs to be… it’s a deprecated thing.
241 00:25:28.290 ⇒ 00:25:29.750 Uttam Kumaran: We’ll try and see what happens.
242 00:25:29.950 ⇒ 00:25:34.990 Emily Giant: Alright, so this is in… FB Ad Insights XFDMA.
243 00:25:46.130 ⇒ 00:25:48.120 Awaish Kumar: Can we go to the, like.
244 00:25:52.140 ⇒ 00:25:53.139 Emily Giant: It’s this one.
245 00:25:54.550 ⇒ 00:25:55.330 Awaish Kumar: Okay.
246 00:25:56.080 ⇒ 00:25:57.649 Awaish Kumar: This one is…
247 00:26:05.190 ⇒ 00:26:07.940 Emily Giant: Okay, is it gonna give me, like, a million of these?
248 00:26:08.780 ⇒ 00:26:09.110 Emily Giant: Yeah.
249 00:26:09.500 ⇒ 00:26:11.329 Emily Giant: It just wants me to do this.
250 00:26:11.830 ⇒ 00:26:14.459 Awaish Kumar: No, no, let’s see… That’s a good job for cursor.
251 00:26:15.550 ⇒ 00:26:22.110 Awaish Kumar: Yeah, we can, like, ignore this right now, and look at the compiled error, which is…
252 00:26:22.240 ⇒ 00:26:25.709 Awaish Kumar: Because it’s just a warning, and it does not occur.
253 00:26:26.350 ⇒ 00:26:27.020 Emily Giant: Okay.
254 00:26:27.220 ⇒ 00:26:29.050 Awaish Kumar: We can go into the details.
255 00:26:29.410 ⇒ 00:26:30.030 Awaish Kumar: Absolutely.
256 00:26:30.030 ⇒ 00:26:33.700 Emily Giant: The details on which… on logs?
257 00:26:34.610 ⇒ 00:26:40.260 Awaish Kumar: Yeah, like, logs. In the bottom, you can see summary and details.
258 00:26:40.890 ⇒ 00:26:43.509 Awaish Kumar: If we click on, the detail.
259 00:26:44.260 ⇒ 00:26:45.010 Emily Giant: It is.
260 00:27:05.540 ⇒ 00:27:06.199 Awaish Kumar: So where’s…
261 00:27:06.200 ⇒ 00:27:11.040 Emily Giant: All the updated files with the new config, something is going weird.
262 00:27:11.200 ⇒ 00:27:14.740 Awaish Kumar: Oh, here it is, and… encounter.
263 00:27:14.740 ⇒ 00:27:16.660 Emily Giant: Yeah, it’s just a warning.
264 00:27:17.470 ⇒ 00:27:19.420 Awaish Kumar: Oh, it says compilation error.
265 00:27:19.640 ⇒ 00:27:20.480 Emily Giant: Yeah.
266 00:27:20.800 ⇒ 00:27:24.860 Awaish Kumar: That means there is a, can we stay there?
267 00:27:27.630 ⇒ 00:27:36.020 Awaish Kumar: Like, at the Arab line, like, I want to read the… Scroll down at the arrow.
268 00:27:43.360 ⇒ 00:27:44.320 Awaish Kumar: Modern.
269 00:27:46.460 ⇒ 00:27:48.439 Awaish Kumar: Facebook ads more than possible.
270 00:27:49.250 ⇒ 00:27:55.139 Awaish Kumar: FP add inside. Okay, can we run this in, in the… at the bottom?
271 00:27:55.650 ⇒ 00:27:59.700 Awaish Kumar: Where you can run the command, right here.
272 00:28:00.200 ⇒ 00:28:04.640 Emily Giant: It’s not gonna let me, because of the errors. It doesn’t let you run if there’s an error.
273 00:28:09.010 ⇒ 00:28:13.070 Awaish Kumar: Yeah, like, you can’t click there?
274 00:28:13.420 ⇒ 00:28:14.619 Awaish Kumar: You can, right?
275 00:28:15.050 ⇒ 00:28:15.610 Emily Giant: Yeah.
276 00:28:15.610 ⇒ 00:28:17.130 Awaish Kumar: You can write, maybe.
277 00:28:18.150 ⇒ 00:28:18.980 Emily Giant: Write what?
278 00:28:19.260 ⇒ 00:28:20.590 Awaish Kumar: dbt run, like.
279 00:28:20.850 ⇒ 00:28:22.949 Emily Giant: Or just run this model right here.
280 00:28:26.890 ⇒ 00:28:32.180 Awaish Kumar: DBT run… minus S and the name of the model.
281 00:28:32.710 ⇒ 00:28:34.800 Emily Giant: You wanna run this model that we’re on?
282 00:28:34.800 ⇒ 00:28:35.760 Awaish Kumar: Yes, yes.
283 00:28:35.760 ⇒ 00:28:36.450 Emily Giant: Okay.
284 00:28:43.090 ⇒ 00:28:44.749 Awaish Kumar: Okay, let’s see…
285 00:28:47.210 ⇒ 00:28:55.900 Emily Giant: Oh, not system logs. You want… alright, I guess it’s not gonna give me… Same, same mayors, yeah.
286 00:29:07.130 ⇒ 00:29:08.760 Awaish Kumar: What was that?
287 00:29:12.200 ⇒ 00:29:19.330 Awaish Kumar: Okay, that means file is not being compiled, but it’s not returning that, is it…
288 00:29:22.470 ⇒ 00:29:26.520 Awaish Kumar: Can… can you, like, minimize this one?
289 00:29:26.630 ⇒ 00:29:27.190 Awaish Kumar: Hold on.
290 00:29:28.510 ⇒ 00:29:36.890 Awaish Kumar: And, in the File Explorer, Do we have a target?
291 00:29:38.110 ⇒ 00:29:39.380 Awaish Kumar: older, like…
292 00:29:46.020 ⇒ 00:29:48.099 Awaish Kumar: Okay, so then… hold on.
293 00:29:48.600 ⇒ 00:29:50.150 Emily Giant: What did you say? I’m sorry.
294 00:29:50.150 ⇒ 00:29:51.080 Awaish Kumar: target.
295 00:29:52.040 ⇒ 00:29:55.830 Awaish Kumar: T-A-R-G-E-T… it’s a folder, right, so it’s not.
296 00:29:55.830 ⇒ 00:29:57.689 Emily Giant: Yeah, yeah, yeah, okay, gotcha.
297 00:29:58.240 ⇒ 00:30:01.340 Emily Giant: Can we, like, tests?
298 00:30:02.570 ⇒ 00:30:07.690 Awaish Kumar: Outside of the… yeah, this target, click on this, And compiled.
299 00:30:10.020 ⇒ 00:30:16.760 Awaish Kumar: Albon’s terms… models… Facebook ads.
300 00:30:18.640 ⇒ 00:30:20.160 Awaish Kumar: Yeah, Facebook ads.
301 00:30:20.970 ⇒ 00:30:21.680 Awaish Kumar: And the line.
302 00:30:21.680 ⇒ 00:30:25.729 Emily Giant: Where’s Facebook ads? Oh, derp. Okay, gotcha. Models.
303 00:30:32.030 ⇒ 00:30:34.449 Emily Giant: So, like, any of these, they all seem to be…
304 00:30:35.590 ⇒ 00:30:38.979 Emily Giant: Let’s find this one, Insights XF, that’s gonna be in Transform.
305 00:30:39.940 ⇒ 00:30:42.230 Awaish Kumar: Okay, can we open that?
306 00:30:43.280 ⇒ 00:30:45.390 Awaish Kumar: It’s in the… it’s at the bottom one.
307 00:30:46.500 ⇒ 00:30:49.610 Awaish Kumar: Is it in the bottom, like, the XF region?
308 00:30:49.860 ⇒ 00:30:50.900 Awaish Kumar: Oh, gosh.
309 00:30:51.870 ⇒ 00:30:52.720 Awaish Kumar: Okay.
310 00:30:53.850 ⇒ 00:30:55.230 Awaish Kumar: Oh…
311 00:31:00.970 ⇒ 00:31:02.999 Awaish Kumar: Can we copy-paste this?
312 00:31:03.220 ⇒ 00:31:04.580 Emily Giant: Yeah. Yeah.
313 00:31:04.580 ⇒ 00:31:09.479 Awaish Kumar: and put it in Redshift, like, Do you have redshift?
314 00:31:10.160 ⇒ 00:31:14.910 Emily Giant: Ugh, it’s not gonna let me because of… I have mode.
315 00:31:17.820 ⇒ 00:31:20.360 Emily Giant: Same deal. I still have to replace the…
316 00:31:22.000 ⇒ 00:31:24.729 Awaish Kumar: I can… I have opened it.
317 00:31:26.610 ⇒ 00:31:28.530 Emily Giant: Did it really not? Okay, one sec.
318 00:31:30.860 ⇒ 00:31:33.799 Emily Giant: I do think we need to revert that PR.
319 00:31:34.240 ⇒ 00:31:36.610 Emily Giant: So that we can not have, like.
320 00:31:36.860 ⇒ 00:31:39.710 Emily Giant: because I think it was Utam’s PR that…
321 00:31:40.420 ⇒ 00:31:43.450 Emily Giant: caused this. Should we go ahead and revert that while we’re here?
322 00:31:46.310 ⇒ 00:31:50.390 Emily Giant: Because the… all the updates are gonna fail, or the refreshes are gonna fail.
323 00:31:51.490 ⇒ 00:31:55.499 Awaish Kumar: Okay, so is it the… is it the recent one?
324 00:31:56.310 ⇒ 00:31:59.070 Emily Giant: Yeah, so if I… if I revert this…
325 00:31:59.290 ⇒ 00:31:59.900 Awaish Kumar: Okay.
326 00:32:01.070 ⇒ 00:32:04.129 Emily Giant: then I believe that it will…
327 00:32:09.110 ⇒ 00:32:12.870 Emily Giant: So it’s just some kind of troubleshooting within…
328 00:32:17.500 ⇒ 00:32:19.550 Emily Giant: Oh, I might still need to click a button.
329 00:32:19.890 ⇒ 00:32:20.800 Emily Giant: Merge.
330 00:32:40.650 ⇒ 00:32:43.920 Awaish Kumar: We are on branch GF4 in the snapshot.
331 00:32:47.880 ⇒ 00:32:50.969 Awaish Kumar: you are on a branch like GF4 Inventory Snapshot.
332 00:32:52.800 ⇒ 00:32:54.579 Emily Giant: a GA4 snapshot.
333 00:32:55.250 ⇒ 00:33:00.709 Emily Giant: Oh, let me get… here we go. I’m just… this is the PR, so let’s see if it resolves the error.
334 00:33:01.760 ⇒ 00:33:02.430 Awaish Kumar: Okay.
335 00:33:04.960 ⇒ 00:33:05.680 Emily Giant: Yep.
336 00:33:06.230 ⇒ 00:33:07.139 Emily Giant: It does.
337 00:33:08.100 ⇒ 00:33:14.440 Emily Giant: So it’s something with his… Something with this, PR.
338 00:33:14.590 ⇒ 00:33:18.580 Emily Giant: That’s not… working here.
339 00:33:23.120 ⇒ 00:33:23.850 Awaish Kumar: Okay.
340 00:33:25.110 ⇒ 00:33:25.960 Emily Giant: Oh.
341 00:33:28.070 ⇒ 00:33:31.140 Emily Giant: Well, that was just me, right?
342 00:33:32.100 ⇒ 00:33:34.489 Emily Giant: No, because I didn’t commit anything.
343 00:33:36.500 ⇒ 00:33:39.260 Emily Giant: So he updated this to this.
344 00:33:41.660 ⇒ 00:33:43.699 Emily Giant: Which seems like it should.
345 00:33:44.680 ⇒ 00:33:50.630 Emily Giant: have worked, but… Because this was the warning we were getting, but that’s not what bonked it out.
346 00:33:50.950 ⇒ 00:33:52.109 Awaish Kumar: Is it dead.
347 00:33:52.150 ⇒ 00:33:55.840 Emily Giant: It’s like the macro… the macro made an error.
348 00:33:56.750 ⇒ 00:33:57.470 Awaish Kumar: But…
349 00:34:07.920 ⇒ 00:34:13.320 Awaish Kumar: I think it’s a revert PR, that means… The changes are, like…
350 00:34:14.480 ⇒ 00:34:18.240 Awaish Kumar: The changes in the red are the ones which are made by Huto.
351 00:34:18.889 ⇒ 00:34:22.589 Emily Giant: Oh, oh, okay, you’re right, you’re totally right. Okay.
352 00:34:24.429 ⇒ 00:34:27.639 Emily Giant: So, if we’re looking at it from that perspective.
353 00:34:30.939 ⇒ 00:34:41.219 Emily Giant: I’m trying to see, like, what… these all look very normal. It does look like this was a deprecated macro that got switched back to this.
354 00:34:42.159 ⇒ 00:34:42.959 Awaish Kumar: Yeah.
355 00:34:43.260 ⇒ 00:34:45.150 Emily Giant: And that’s why it’s happening.
356 00:34:45.600 ⇒ 00:34:50.500 Emily Giant: I bet you anything. I’ll go through… I can go through in his branch and just…
357 00:34:51.750 ⇒ 00:34:56.140 Emily Giant: Or even go through this PR and just make sure that those are updated and try it again.
358 00:34:56.679 ⇒ 00:35:01.809 Awaish Kumar: Yeah, like, no, like, even, like, macro is okay, like.
359 00:35:02.030 ⇒ 00:35:06.740 Awaish Kumar: Thing is, the changes which he has made, like, regarding sort and disk keys.
360 00:35:10.970 ⇒ 00:35:12.810 Emily Giant: Say it again, please? Sorry, you cut out.
361 00:35:13.210 ⇒ 00:35:18.929 Awaish Kumar: So, only the changes which he has made, on, like, sort keys, test keys.
362 00:35:19.750 ⇒ 00:35:23.660 Awaish Kumar: Tunic keys, or incremental, like, these are the important changes.
363 00:35:23.850 ⇒ 00:35:24.550 Emily Giant: Yes.
364 00:35:24.550 ⇒ 00:35:26.670 Awaish Kumar: Her name is… isn’t that.
365 00:35:26.840 ⇒ 00:35:27.980 Awaish Kumar: Important.
366 00:35:28.490 ⇒ 00:35:29.130 Emily Giant: Yeah.
367 00:35:31.740 ⇒ 00:35:35.010 Emily Giant: But… Okay.
368 00:35:46.070 ⇒ 00:35:52.140 Emily Giant: Well, I can go through one at a time, and deploy these, and see which one is making it crap out.
369 00:35:52.470 ⇒ 00:35:59.499 Emily Giant: Not the most efficient way to do it, but at least it will keep things from breaking.
370 00:36:00.280 ⇒ 00:36:00.880 Awaish Kumar: Okay.
371 00:36:01.700 ⇒ 00:36:02.700 Awaish Kumar: Okay.
372 00:36:02.700 ⇒ 00:36:10.089 Emily Giant: Do you want to work on it, or would you like me to do this and, like, fix this PR, or I can keep working on our dashboard stuff?
373 00:36:10.820 ⇒ 00:36:12.830 Emily Giant: If you want to take a look at this.
374 00:36:13.410 ⇒ 00:36:17.490 Emily Giant: Because it seems like it shouldn’t be complicated at all. There’s not that many files changed.
375 00:36:20.820 ⇒ 00:36:24.230 Awaish Kumar: Yeah, you can look at dashboard, and this one, like, I can,
376 00:36:25.160 ⇒ 00:36:29.079 Awaish Kumar: see if I can resolve this. Otherwise, maybe…
377 00:36:29.080 ⇒ 00:36:29.670 Emily Giant: Okay.
378 00:36:29.670 ⇒ 00:36:32.089 Awaish Kumar: Yeah, I can fix it, maybe.
379 00:36:32.280 ⇒ 00:36:41.280 Emily Giant: Okay, take a look, don’t spend too much time, because this is something I can do, like, in between meetings, probably. Actually, don’t even worry about it, I’ll do this in between meetings.
380 00:36:41.540 ⇒ 00:36:46.669 Emily Giant: And then, if you want to focus on North Beam and stuff, that’s way more important.
381 00:36:47.020 ⇒ 00:36:47.740 Awaish Kumar: Okay.
382 00:36:48.700 ⇒ 00:36:53.190 Emily Giant: Alright, okay, I will ping you when I’m done, with the dashboard stuff.
383 00:36:53.520 ⇒ 00:36:55.159 Awaish Kumar: Okay, sure, thank you.
384 00:36:55.160 ⇒ 00:36:56.340 Emily Giant: Talk to you soon.
385 00:36:56.340 ⇒ 00:36:57.090 Awaish Kumar: I think.
386 00:36:57.340 ⇒ 00:36:57.920 Emily Giant: Bye.