Meeting Title: Analytics Engineering Daily Sync Date: 2025-02-24 Meeting participants: Luke Daque, Uttam Kumaran, Awaish Kumar, Caio Velasco
WEBVTT
1 00:00:26.090 ⇒ 00:00:27.939 Uttam Kumaran: Hey? Guys, good morning.
2 00:00:28.700 ⇒ 00:00:29.690 Caio Velasco: Good morning!
3 00:00:30.000 ⇒ 00:00:32.960 Uttam Kumaran: And how are you? How’s the weekend.
4 00:00:34.590 ⇒ 00:00:40.162 Caio Velasco: That’s good. It’s good. Like play some football. Oh, some soccer. Sorry.
5 00:00:40.540 ⇒ 00:00:41.440 Uttam Kumaran: Football, football.
6 00:00:43.720 ⇒ 00:00:45.820 Caio Velasco: Yeah, it was cool. It was a good weather here.
7 00:00:45.820 ⇒ 00:00:47.409 Uttam Kumaran: What team do you support?
8 00:00:48.270 ⇒ 00:00:51.109 Caio Velasco: In Brazil we support Flamenco.
9 00:00:51.500 ⇒ 00:00:52.050 Caio Velasco: Okay.
10 00:00:52.050 ⇒ 00:00:52.420 Uttam Kumaran: Okay.
11 00:00:52.420 ⇒ 00:01:06.290 Caio Velasco: It’s 1 of the biggest in Brazil. It’s 1 of. I think it’s the biggest right. And in Europe I don’t really have like a team that I support. But I usually like real Madrid.
12 00:01:06.540 ⇒ 00:01:07.930 Caio Velasco: That’s why I usually like.
13 00:01:07.930 ⇒ 00:01:11.598 Uttam Kumaran: Okay, that’s easy. I went to. You know, I went to go see?
14 00:01:12.570 ⇒ 00:01:21.300 Uttam Kumaran: I went to Burnabout in 2020 to go see real versus versus Barcelo play.
15 00:01:21.300 ⇒ 00:01:23.189 Caio Velasco: Oh, my God! That’s that was amazing!
16 00:01:23.190 ⇒ 00:01:23.570 Uttam Kumaran: Say.
17 00:01:23.860 ⇒ 00:01:31.980 Caio Velasco: Yeah, yeah, no. I still have to do that. I haven’t had the chance, because I was in Barcelona, and the stadium was in under construction.
18 00:01:32.460 ⇒ 00:01:33.050 Uttam Kumaran: Oh!
19 00:01:33.050 ⇒ 00:01:35.260 Caio Velasco: So yeah, I’ll still have to do that.
20 00:01:36.130 ⇒ 00:01:40.701 Uttam Kumaran: It was incredible. I mean, like, I don’t really feel like going to new another soccer game in my life.
21 00:01:41.740 ⇒ 00:01:45.466 Uttam Kumaran: like I feel like I saw the best game.
22 00:01:46.080 ⇒ 00:01:53.250 Uttam Kumaran: Yeah. And but what a great sport! Yeah, here in us. It’s not picking up like that fast.
23 00:01:53.820 ⇒ 00:01:57.680 Uttam Kumaran: I don’t know. May take some more time. Mls. Is coming slowly, but
24 00:01:58.250 ⇒ 00:02:02.079 Uttam Kumaran: all the money in the Us. Is football and basketball, you know, so.
25 00:02:02.230 ⇒ 00:02:02.750 Caio Velasco: That’s true.
26 00:02:02.750 ⇒ 00:02:07.230 Uttam Kumaran: They have to raise a lot of money to support it, because they have to basically get the fans there and
27 00:02:07.500 ⇒ 00:02:14.740 Uttam Kumaran: also soccer us. People like. I don’t think us fans have that sort of patience.
28 00:02:15.000 ⇒ 00:02:17.419 Uttam Kumaran: I don’t know. Like I, that’s what I think.
29 00:02:17.640 ⇒ 00:02:23.928 Uttam Kumaran: As an Nba. It’s fast, and football is sort of on and off, and I just think that there’s
30 00:02:25.140 ⇒ 00:02:28.799 Uttam Kumaran: It’s sometimes not as as easy.
31 00:02:29.630 ⇒ 00:02:36.689 Caio Velasco: Yeah, no, I totally understand. I remember when I was there, it was all about football and basketball. That’s really true. I even went to
32 00:02:37.213 ⇒ 00:02:45.049 Caio Velasco: a baseball game, but you can see that. Still, it’s it’s football first, st maybe basketball after something like that, and then baseball. I don’t know.
33 00:02:52.360 ⇒ 00:03:02.309 Uttam Kumaran: Cool. Yeah, I I this is my favorite meeting of the day. So I’m always happy to be here and talk to more Dbt people. I can’t believe that we have
34 00:03:03.030 ⇒ 00:03:18.469 Uttam Kumaran: like a bunch of us. So I’m very, very excited. We’ve also extended an offer to Demo a day. He was on the call yesterday. I would have had him connect with you, Kyle, but you just joined so I didn’t want to put pressure on you. To be like. I don’t know how it is.
35 00:03:18.470 ⇒ 00:03:19.030 Caio Velasco: Yes.
36 00:03:20.450 ⇒ 00:03:37.770 Uttam Kumaran: But he connected with Luke he connected with pious! Who’s on the analyst team? I talked to him very amazing, like, I I’m kind of hopeful that he moves forward, and ideally I would. He’s gonna start, probably in another week from now, but I will include him in meetings and stuff.
37 00:03:38.350 ⇒ 00:03:42.639 Uttam Kumaran: The overall theme of this week is gonna be called like
38 00:03:43.762 ⇒ 00:03:52.170 Uttam Kumaran: I don’t know like reformation, or like restructuring, or something like that. I’m sort of taking a much bigger role in
39 00:03:52.300 ⇒ 00:04:03.859 Uttam Kumaran: all of our data client, you know, sort of project management. It’s what I used to do. So I’m very excited. I sort of freed up a lot of my time
40 00:04:04.000 ⇒ 00:04:08.420 Uttam Kumaran: from other stuff kind of very strategically over the last like month.
41 00:04:08.930 ⇒ 00:04:29.889 Uttam Kumaran: and of course, you know we’ve we’ve hired great people, so I think we’re sort of at the finishing line of a lot of this big hiring push. I’m sure as Luke knows if I get involved, things go a little bit smoother. So I’m like, I’m I’m very excited, although I I really will be leaning on you guys on the team to sort of assist with the actual execution. Of course.
42 00:04:30.380 ⇒ 00:04:54.010 Uttam Kumaran: the project management. I see us as all horizontal meaning there. It’s not like a go do this, and sort of like me telling people what to do. It’s a discussion, and my job is to take on all of the stuff off your plate. That is related to getting more information, telling you what the prioritization is, and then, of course, just checking to validate the work. Is delivered
43 00:04:54.040 ⇒ 00:05:08.009 Uttam Kumaran: everything in between that I’m sort of expecting you guys to handle. And I I think that’s a fair, you know, bargain. The other thing that I think what we’re gonna do today is we’re gonna run 2 roadmapping sessions for Joby and Eden.
44 00:05:08.070 ⇒ 00:05:20.940 Uttam Kumaran: depending on how those go, I will most likely run an additional 2 for the or 3 for the for the rest of the clients. It’s a little bit of smaller crew on the rest of the clients. These are our 2 like highest priority
45 00:05:21.030 ⇒ 00:05:22.480 Uttam Kumaran: clients right now.
46 00:05:22.720 ⇒ 00:05:27.560 Uttam Kumaran: So I’m very excited to go through that I’m honestly going to sort of skip
47 00:05:27.940 ⇒ 00:05:33.249 Uttam Kumaran: notion for the short term. I think you know we have a lot in there.
48 00:05:33.420 ⇒ 00:05:47.199 Uttam Kumaran: There. There could be a lot much better, but I do think that for all of these clients we only have anywhere from 5 to 7 in flight priorities. So I’m not too worried about having every single thing ticketed.
49 00:05:47.552 ⇒ 00:06:05.499 Uttam Kumaran: I think, as you guys know, like I, I want us to just look at the 5 things that we’re working on. We don’t need to have a hundred tickets, and we don’t need to look in the past or the future. So I think what we’re gonna do is basically try to make sure, all that is outlined on figma today. And then we’re just gonna go through. So we will. What?
50 00:06:05.730 ⇒ 00:06:24.829 Uttam Kumaran: So on our clients, we sort of have 2 different major roles we have like engagement leads, and then we have, like project managers. So Robert is really the engagement. Lead on both Javi and Eden, meaning. He’s meeting with a client every day. He’s demoing our work. He’s getting requirements. My job is to take his big picture like
51 00:06:24.890 ⇒ 00:06:36.860 Uttam Kumaran: they need help solving this marketing problem and then break it down into, is it a dashboard problem? Is it analysis problems or data modeling issue? Or is it a new data source. So I’m asking him to sort of give me the priorities.
52 00:06:37.040 ⇒ 00:06:52.970 Uttam Kumaran: He gives it in the way of, like the client just sees priority one priority, 2 priority 3, we are gonna break it down based on area of expertise. Right? So we will take those priorities and break it down like, this is a dashboarding request. This is an analysis request. This is a data model request.
53 00:06:53.100 ⇒ 00:07:02.169 Uttam Kumaran: partly that will help to see. Like, okay, we have 5 data model requests around customer service. We can start to build up that mark and sort of help help. You know, this
54 00:07:02.650 ⇒ 00:07:03.860 Uttam Kumaran: sort of area?
55 00:07:04.295 ⇒ 00:07:24.559 Uttam Kumaran: So that’s kind of like the overall goal. I think this should be pretty clarifying for everybody. On what actually needs to happen. There is still probably some like tech debt and things like that. That I’m kind of going through every client and making sure that, like the warehouses, are clean and stuff like that. But ultimately, this week we want to sort of make sure that
56 00:07:24.800 ⇒ 00:07:33.199 Uttam Kumaran: by Thursday Robert is set up to do the client Demos with all of the resources he needs to prepare the slides and then demo our dashboards and our work.
57 00:07:33.705 ⇒ 00:07:53.370 Uttam Kumaran: So that’s what we’ll be working on. I think today, you know, I we can kind of take this meeting anywhere we want. But maybe we could even have a discussion. You know, for folks are at our opinionated about how we do that planning session and and what we talk about. I will share
58 00:07:53.510 ⇒ 00:07:59.677 Uttam Kumaran: this. I was gonna share this link anyways in slack. But I’ll share this here, which is
59 00:08:00.460 ⇒ 00:08:04.610 Uttam Kumaran: basically the fig jam board that we’ll be running for Javi coffee?
60 00:08:17.930 ⇒ 00:08:30.860 Uttam Kumaran: So if you guys, I mean, I would love to have a discussion about like how you if you guys have done planning in the past what’s been helpful. I know, Kyle, I think we have now pretty set structure for doing dashboard requirements would love to kind of
61 00:08:31.290 ⇒ 00:08:36.459 Uttam Kumaran: get a sense for how we should do data modeling requirements, especially for new marts.
62 00:08:36.850 ⇒ 00:08:45.489 Uttam Kumaran: But I don’t know if you guys have any thoughts on that. We can discuss that here, and then maybe take the last 10 min, and we could do reviews of anything.
63 00:08:47.660 ⇒ 00:08:48.680 Caio Velasco: Cool. Okay.
64 00:08:56.450 ⇒ 00:08:57.810 Luke Daque: Probably share.
65 00:08:57.810 ⇒ 00:08:59.639 Awaish Kumar: What do you mean by.
66 00:09:00.840 ⇒ 00:09:02.170 Luke Daque: Yeah, go. Go ahead.
67 00:09:02.380 ⇒ 00:09:03.290 Luke Daque: I wish.
68 00:09:05.380 ⇒ 00:09:13.710 Awaish Kumar: Yeah, like, are, we are ex like expecting to get the data modeling request from the analyst or something.
69 00:09:14.500 ⇒ 00:09:22.989 Uttam Kumaran: Yeah. So my job is gonna be to break down. Who gets what request? Basically, Robert is? Gonna say, they have no insight into
70 00:09:23.210 ⇒ 00:09:30.109 Uttam Kumaran: their customer service team what they’re he’s gonna give me those types of questions that are in the tickets. Now, which is like.
71 00:09:30.210 ⇒ 00:09:56.809 Uttam Kumaran: I want to know the measurement of our most effective customer service agents. How long does it take tickets to close? Then, within this group during planning, we will say, is the data anywhere. Okay, if it’s there. Okay, let’s work on the requirements. If it’s not there. I’m just gonna we don’t even have to talk to the analysts at that point. Because I’m like, just make sure that there’s some data available. And then we will work with the analysts to basically create that those dashboard requirements. That’s the that’s the process.
72 00:09:58.030 ⇒ 00:09:58.820 Awaish Kumar: Okay.
73 00:10:16.030 ⇒ 00:10:17.290 Uttam Kumaran: Any thoughts
74 00:10:18.130 ⇒ 00:10:27.560 Uttam Kumaran: Luke or Kyle on this process. I mean, we have an hour. So I think, basically, I’m gonna try to break everything up into new data source, new data model
75 00:10:28.554 ⇒ 00:10:32.506 Uttam Kumaran: changing a data model or or dashboard, basically
76 00:10:33.190 ⇒ 00:10:36.489 Uttam Kumaran: And then at the end of the session, I’m basically gonna get a sense of like.
77 00:10:36.970 ⇒ 00:10:38.949 Uttam Kumaran: do we think we can get this done this week?
78 00:10:39.320 ⇒ 00:10:41.200 Uttam Kumaran: And that’ll be my.
79 00:10:41.310 ⇒ 00:10:57.139 Uttam Kumaran: that question. I I can. I only feel comfortable asking when everybody is bought in on the requirements. Typically the way product management goes, which I’ve I’m very much against is product manager gets all the things, writes everything down, and then is like.
80 00:10:57.310 ⇒ 00:11:00.230 Uttam Kumaran: Okay, you read it for 10 seconds. How many points?
81 00:11:00.410 ⇒ 00:11:28.599 Uttam Kumaran: That’s like totally useless right? And that’s because product managers go to school for product management. They don’t go to school for engineering. So my job is to build the requirements with you, and then we all can say, how much is, how much is this gonna take? And it’s really obvious. If you have 10 things, then it’s it’s gonna be hard for me to to say like, Okay, we can’t go. We can’t do 2 days, but if you have one thing, then even you will have confidence that you could do it in a few days. Right? And
82 00:11:29.072 ⇒ 00:11:33.199 Uttam Kumaran: that’s the sort of framework I want to build eventually.
83 00:11:33.280 ⇒ 00:11:47.169 Uttam Kumaran: this is even a part of the area that actually believe everybody on this call, and even the analysts can start to do on their own right. The I will. One thing that you’ll notice over the next few weeks is I’ll be starting to erode the concept of
84 00:11:47.390 ⇒ 00:12:14.299 Uttam Kumaran: analytics, engineer data engineer data model like, there’s I consider everybody as just data folks with expert with specialties right? And so I want, I don’t want there to be barriers, because barriers cause like throwing things over the fence. So I’m gonna start to push more of this requirements gathering onto people as you guys get a really good sense for the client, because I’m telling you for Javi coffee. We have 2 dashboards and 3 data models.
85 00:12:14.930 ⇒ 00:12:37.360 Uttam Kumaran: 1 1 of us, if they, if we sit and learn it, can basically understand the whole project. The difficulty is, can you understand the whole project? And then figure out which piece you need to work on? So over time, I’m gonna start to really hug this. But over the weeks. I’m gonna start to let some pieces go and expect that each of us sort of has brings a little bit of that project management into their own scope
86 00:12:37.360 ⇒ 00:12:48.849 Uttam Kumaran: because one, it gives you a lot of ownership and agency over the problems. Right? You’re not just wake up and like what exactly is going on. But second, I want you guys, we are an extension of their business
87 00:12:48.910 ⇒ 00:12:52.780 Uttam Kumaran: right? And their business. They don’t. Someone doesn’t wake up in their business and is like
88 00:12:53.010 ⇒ 00:13:07.890 Uttam Kumaran: feels detached. Right? So our job is to be in there. And that way when you guys are getting requirements. If you’re like, Hey, that doesn’t seem like very important, because I talked to Cmo, and you know you’ll have that that 6th sense of what actually is important for the client.
89 00:13:08.110 ⇒ 00:13:27.210 Uttam Kumaran: That’s the thing I want our engineering team to be extremely good at. It’s very rare for client engineers to have that sense. And on the data team like, that’s our job. Otherwise, we are going to build dashboard that nobody looks at. We’re gonna build data bottles that nobody queries and that’s like almost the situation right now, so
90 00:13:27.650 ⇒ 00:13:35.859 Uttam Kumaran: I guess I’ll I’ll pause there if there’s no more questions on this. Maybe I I know there’s a couple of Prs and a couple of things that we wanted to review. Maybe we can go through that
91 00:13:36.060 ⇒ 00:13:39.619 Uttam Kumaran: as approved. Is there anything else on this topic?
92 00:13:40.660 ⇒ 00:13:47.053 Luke Daque: Yeah, I can probably share what we did before, like for a previous client that I had
93 00:13:48.420 ⇒ 00:13:53.289 Luke Daque: in relation to like data modeling. And like, how
94 00:13:53.500 ⇒ 00:13:56.439 Luke Daque: we get requirements, basically something like that.
95 00:13:56.810 ⇒ 00:14:02.423 Luke Daque: Yeah, let me, it’s also Google Sheet. So maybe we can
96 00:14:03.930 ⇒ 00:14:09.780 Luke Daque: it, we can expand like what Kai already made something like that.
97 00:14:10.030 ⇒ 00:14:27.150 Caio Velasco: Yeah, I do. I do need like more help on the data modeling side. Because when I was working in something close to that was more a very high level as a Va guy that goes to the client and gets requirements. But I wasn’t really building the data modeling part. So that’s definitely gonna be helpful.
98 00:14:30.190 ⇒ 00:14:38.639 Luke Daque: Yeah, we can. We can probably have a separate call on this or something, maybe Kai, or like, yeah. But but this would be a great team effort as well, or something.
99 00:14:39.080 ⇒ 00:14:39.520 Caio Velasco: Perfect.
100 00:14:39.520 ⇒ 00:14:45.660 Luke Daque: This is like the Google sheet that they provide right? So
101 00:14:46.240 ⇒ 00:14:49.350 Luke Daque: it has like raw data, which is basically the sources
102 00:14:49.450 ⇒ 00:14:52.510 Luke Daque: it has, like all the different sources, and like
103 00:14:52.620 ⇒ 00:14:57.129 Luke Daque: what the method of ingestion is, whether it’s from a 3rd party
104 00:14:57.300 ⇒ 00:15:12.099 Luke Daque: like flatran, for example, whether it’s direct or like, if there’s a specific Api like Python code to load it into. This was they were using bigquery. So like in Gcp, Google cloud platform and like, for
105 00:15:12.220 ⇒ 00:15:17.320 Luke Daque: if it’s like Api, or if it’s full load, or like upsurd stuff like that.
106 00:15:19.400 ⇒ 00:15:25.379 Luke Daque: Yeah. So it will have the source and all the tables related to that source.
107 00:15:25.990 ⇒ 00:15:33.556 Luke Daque: and then these would be the staging models because they were using staging models for Dbt models.
108 00:15:34.970 ⇒ 00:15:45.340 Luke Daque: and like base models as well, and you should. They’re in you. There be. They’ll be in views materialized as views. But since we don’t do this, we can. We cannot.
109 00:15:45.740 ⇒ 00:15:52.559 Luke Daque: Oh, we don’t have to do this. In bridges we’re not using. We’re not creating staging models, but maybe we can create
110 00:15:52.690 ⇒ 00:15:56.950 Luke Daque: base models or raw models for the
111 00:15:57.160 ⇒ 00:16:03.760 Luke Daque: tables that needed to be extracted from Json fields. I guess then maybe we can add it here.
112 00:16:03.760 ⇒ 00:16:10.219 Uttam Kumaran: So my question on this is like, let’s say, we have a summary table which has the same dimensions as like
113 00:16:10.460 ⇒ 00:16:12.670 Uttam Kumaran: some of the dimmer marts, tables like.
114 00:16:13.430 ⇒ 00:16:13.830 Luke Daque: Yeah.
115 00:16:13.830 ⇒ 00:16:16.149 Uttam Kumaran: We just we just replicate it, or
116 00:16:16.860 ⇒ 00:16:21.219 Uttam Kumaran: I feel like maybe we should just do it. This for the dim and Mart’s tables. Maybe.
117 00:16:21.510 ⇒ 00:16:23.950 Luke Daque: Yeah, that’s a good Demos. Yes.
118 00:16:23.950 ⇒ 00:16:25.140 Uttam Kumaran: Like thinking as well.
119 00:16:25.140 ⇒ 00:16:26.150 Uttam Kumaran: Tables.
120 00:16:26.150 ⇒ 00:16:27.919 Luke Daque: Like we can have a a March
121 00:16:28.270 ⇒ 00:16:30.779 Luke Daque: sheet here that has, like the dim
122 00:16:30.980 ⇒ 00:16:36.419 Luke Daque: dimensions in the mark, the the metrics, or like fact tables.
123 00:16:36.540 ⇒ 00:16:41.769 Luke Daque: So it’s particularly something like this, for example, where it subscribed or daily snapshot. So
124 00:16:42.310 ⇒ 00:16:48.049 Luke Daque: this would be the fact table, right? This is coming from a dimension. And this would be coming from this would be like logic.
125 00:16:48.230 ⇒ 00:16:53.730 Luke Daque: like, if there’s calculated fields. If if it’s a logic, then we’ll have like the logic here
126 00:16:54.196 ⇒ 00:16:58.439 Luke Daque: like customers, with at least one subscription, for example, like something like that.
127 00:16:58.580 ⇒ 00:17:08.560 Luke Daque: And then where the source would be, so it would be coming from whatever source it was like, whether it’s a dimension table or direct from the source itself, a raw data.
128 00:17:09.010 ⇒ 00:17:15.299 Luke Daque: And yeah, we even have, like the data formats here. So we don’t mess up like dates are that are in string, or like.
129 00:17:15.430 ⇒ 00:17:16.999 Luke Daque: yeah, stuff like that.
130 00:17:17.380 ⇒ 00:17:17.970 Uttam Kumaran: Okay.
131 00:17:18.319 ⇒ 00:17:20.759 Luke Daque: And then also like tests, if ever
132 00:17:21.019 ⇒ 00:17:32.949 Luke Daque: we are going to test like what test will be using with, is it like Dbt details, or a simple UA uniqueness test or some package we’re using like Dbt expectations. For example.
133 00:17:33.969 ⇒ 00:17:35.599 Luke Daque: it could be like a
134 00:17:35.789 ⇒ 00:17:42.929 Luke Daque: test for each field, or it could be a test for the whole model itself, like this one, like aggregates, should be equal to the
135 00:17:43.299 ⇒ 00:17:46.489 Luke Daque: source, overall tables and stuff like that.
136 00:17:47.289 ⇒ 00:17:53.619 Luke Daque: And then, yeah, if there’s and all like what they did, basically, if it’s us
137 00:17:53.879 ⇒ 00:17:57.439 Luke Daque: specifically complicated logic, then there’s also like a logic
138 00:17:58.579 ⇒ 00:18:00.559 Luke Daque: tab over here. That’s just like.
139 00:18:01.339 ⇒ 00:18:08.459 Luke Daque: like, if it’s an if, then else statement, for example, it doesn’t fit in one cell, you can just like, look it up here.
140 00:18:10.559 ⇒ 00:18:12.509 Luke Daque: But yeah, that’s essentially like
141 00:18:13.099 ⇒ 00:18:19.659 Luke Daque: how, how we did it in my, with my like other client, some of these are drop downs. Just
142 00:18:20.619 ⇒ 00:18:21.369 Luke Daque: yeah.
143 00:18:22.300 ⇒ 00:18:29.160 Uttam Kumaran: Okay, can you? Can you copy like a 1 version of this?
144 00:18:29.320 ⇒ 00:18:31.430 Uttam Kumaran: And I know we have the template
145 00:18:32.060 ⇒ 00:18:41.119 Uttam Kumaran: be right? Maybe we could just create a new tab there, and I can try to take a look at all of these and sort of build like one that brings all all these elements together.
146 00:18:41.720 ⇒ 00:18:44.330 Luke Daque: You’re sure, yeah, so sounds good.
147 00:18:46.360 ⇒ 00:18:49.819 Uttam Kumaran: Yeah, what is the what is that? Is there? What is that template sheet called.
148 00:18:51.000 ⇒ 00:18:54.390 Luke Daque: The the one that okay, you created.
149 00:18:54.390 ⇒ 00:18:55.170 Uttam Kumaran: Yeah.
150 00:19:14.670 ⇒ 00:19:16.520 Uttam Kumaran: Oh, I got it! I got it here.
151 00:19:31.740 ⇒ 00:19:35.159 Luke Daque: Can you send the Linkedin in chat? Oh, yeah, I see.
152 00:19:36.680 ⇒ 00:19:41.549 Luke Daque: Yeah, sure. I’ll I’ll add the I’ll create a sheet here and maybe create some.
153 00:19:42.440 ⇒ 00:19:44.490 Luke Daque: Maybe I name it Martz, or
154 00:19:45.320 ⇒ 00:19:47.990 Luke Daque: March tables like for the dimensions and
155 00:19:49.370 ⇒ 00:19:52.969 Luke Daque: and fact tables, and then maybe we can go from there.
156 00:19:54.610 ⇒ 00:20:02.040 Caio Velasco: This one is the previous one, I believe, because the one I’m I’m using. It’s already updated in some other parts.
157 00:20:03.130 ⇒ 00:20:06.470 Caio Velasco: just just to let you know I’m great.
158 00:20:12.310 ⇒ 00:20:13.000 Caio Velasco: There’s.
159 00:20:18.240 ⇒ 00:20:19.000 Luke Daque: Okay.
160 00:20:24.370 ⇒ 00:20:31.940 Caio Velasco: So there, I think this Beta modeling thing should be something either in between those like the business and the engineering tabs.
161 00:20:32.492 ⇒ 00:20:42.110 Caio Velasco: be able to be like the data model part. Because, yeah, we might get to this problem that you mentioned of replicating things. If you just try to do like a normal like table.
162 00:20:42.731 ⇒ 00:20:47.170 Caio Velasco: So yeah, somehow would be interesting to see how to connect both sides.
163 00:20:47.170 ⇒ 00:20:48.780 Uttam Kumaran: I also may
164 00:20:48.940 ⇒ 00:20:58.850 Uttam Kumaran: just do cause. It’s nice in in Luke, in your sheet. It has the name, the Dims, and and the measures. I’m actually just do that in figma, because.
165 00:20:58.850 ⇒ 00:20:59.520 Luke Daque: Hmm.
166 00:20:59.520 ⇒ 00:21:11.600 Uttam Kumaran: We? We’re only gonna we’re not gonna be looking at. Enter any of the intermediate models right? And so what are we left with? We’re left with maybe 10 to 30 core fact, Tim Tim. Tables.
167 00:21:11.840 ⇒ 00:21:22.379 Uttam Kumaran: I’m building for this larger architecture diagram. I think it’ll be easy for us to just use that whole thing anytime right? And I know Google sheets is probably like
168 00:21:22.570 ⇒ 00:21:25.260 Uttam Kumaran: Google sheets is is useful for, like everything.
169 00:21:25.450 ⇒ 00:21:35.579 Uttam Kumaran: But I think that it’s hard for any clients to digest those table structures, and maybe even for the analysts. So I’m going to try to build that out in figma
170 00:21:36.001 ⇒ 00:21:46.139 Uttam Kumaran: meaning we’re gonna have our. We’re gonna have our data sources. We’re gonna have what? How it gets replicated into the warehouse where it lives in the warehouse. And then all and those will just be
171 00:21:46.250 ⇒ 00:21:53.450 Uttam Kumaran: basically the databases. And then ultimately they will pull into dimmer fact tables.
172 00:21:53.610 ⇒ 00:22:02.859 Uttam Kumaran: And it’ll be like sort of the erd diagrams that you’ll get out of Dbt cloud, except like simpler right? Like the 80 20 of of what that of those are.
173 00:22:03.340 ⇒ 00:22:07.960 Uttam Kumaran: So that’s what I’ll work on. But this is that’s really helpful. Yeah, Luke, if you can just paste that
174 00:22:08.200 ⇒ 00:22:12.829 Uttam Kumaran: somewhere in here I would love to take as I like sort of some of the columns in there.
175 00:22:17.010 ⇒ 00:22:17.770 Uttam Kumaran: Cool.
176 00:22:17.990 ⇒ 00:22:21.720 Luke Daque: Yeah. And we can like you can change it like how we
177 00:22:21.850 ⇒ 00:22:24.440 Luke Daque: would be like best for us. Yes.
178 00:22:24.440 ⇒ 00:22:25.030 Uttam Kumaran: Okay.
179 00:22:26.780 ⇒ 00:22:44.220 Uttam Kumaran: I think one thing that I would love to do. You know, while we have a little bit more time, and then I’m gonna go, you know. Prep a little bit for the roadmap. Is just do like a Pr review, maybe with everybody on the call here. I 1 thing that I’m starting to do is making sure that we’re all we have a code owners process.
180 00:22:44.450 ⇒ 00:22:49.219 Uttam Kumaran: There’s gonna be times where we just need to push stuff, and I think, probably for the next few weeks. It may be
181 00:22:49.740 ⇒ 00:22:56.299 Uttam Kumaran: that, but I want to make sure that every Pr gets reviewed and that we do have a little bit of a process for
182 00:22:56.450 ⇒ 00:23:00.700 Uttam Kumaran: handling those reviews. I think a wish. If you’re okay with
183 00:23:01.250 ⇒ 00:23:04.960 Uttam Kumaran: being the guinea pig today, then I can review
184 00:23:05.280 ⇒ 00:23:09.759 Uttam Kumaran: your we can review your march. Pr, that I just saw. Come in.
185 00:23:10.408 ⇒ 00:23:11.440 Uttam Kumaran: What do you think.
186 00:23:11.440 ⇒ 00:23:12.259 Awaish Kumar: Yeah, sure.
187 00:23:13.330 ⇒ 00:23:17.430 Uttam Kumaran: Okay, cool. But I I think as part of this exercise, I’m
188 00:23:17.960 ⇒ 00:23:25.919 Uttam Kumaran: I’m very interested in everybody’s feedback. I don’t know. Is everybody in this call done like more formal Pr reviews.
189 00:23:27.120 ⇒ 00:23:32.949 Uttam Kumaran: okay, cool. I think this will probably I don’t know. Wait, did you guys do more formal Pr reviews at your last place?
190 00:23:35.310 ⇒ 00:23:37.109 Awaish Kumar: Like, yes, we did like
191 00:23:37.470 ⇒ 00:23:38.110 Uttam Kumaran: Okay.
192 00:23:39.120 ⇒ 00:23:39.969 Awaish Kumar: Okay.
193 00:23:41.740 ⇒ 00:23:52.309 Awaish Kumar: like everybody has done the Pr Pr reviews, gets comments, maybe improvements in the code or the or the maybe explanations for the logic. And yeah.
194 00:23:53.220 ⇒ 00:23:59.979 Uttam Kumaran: So I’m just gonna I’ll walk through this exercise. But I think it’s actually really good to know. I think, especially for Kyle and for Luke.
195 00:24:00.360 ⇒ 00:24:01.520 Uttam Kumaran: There’s a
196 00:24:01.930 ⇒ 00:24:20.020 Uttam Kumaran: you know, to date. Our Pr review process has been like, Okay, just go give me a review and the typical engineering teams. It’ll be like, just give me the positive review and move forward. We’re gonna start to move, to be more strict. Really, the responsibility for code working and being accurate falls on the engineer and the reviewer.
197 00:24:20.441 ⇒ 00:24:37.239 Uttam Kumaran: Typically, those are 2 separate people, right? And so one of the things I’ll just walk you through. How I look at Pr reviews and ideally, anybody on the team should be able to review Prs regardless of if you worked on this client or not. Of course at the moment. That’s probably not
198 00:24:37.240 ⇒ 00:24:51.130 Uttam Kumaran: the case. However, that’s where we want to get it. Similarly, how anybody can take any ticket on anybody should on the team should be able to review any Pr across the team. So one thing that I look at is, first, st I just look to understand, what is this. Pr, about right?
199 00:24:51.531 ⇒ 00:25:09.860 Uttam Kumaran: Okay, it looks like it’s related to marketing. Mart. It’s it’s for Eden, and then I, we look to read the Pr description. This is another thing that I think we will work to make something that works for our team, which is what is what is a great Pr description, right? But I can just read this. It looks like it’s adding Key, Dbt, models for marketing analytics.
200 00:25:11.180 ⇒ 00:25:17.589 Uttam Kumaran: I see a couple of the core dim and fact. Tables that I’m familiar with.
201 00:25:18.320 ⇒ 00:25:27.709 Uttam Kumaran: I think the one thing here is it’s not associated with any ticket. So I can’t go. Reference. Okay, what actually did the project manager want, or what were the key goals?
202 00:25:27.840 ⇒ 00:25:39.929 Uttam Kumaran: Of course, like, I’m just picking on this one there, there really wasn’t 1 but that’s sort of one thing that I’m that I’d be looking at. The other thing immediately that I look at is, if it ran validation.
203 00:25:40.260 ⇒ 00:25:47.250 Uttam Kumaran: The P. Dbtpr validation step. This runs Dbt in the Cicd profile.
204 00:25:48.060 ⇒ 00:25:53.669 Uttam Kumaran: Meaning it’s gonna run the entire Dbt build process in staging
205 00:25:53.980 ⇒ 00:25:56.830 Uttam Kumaran: on Pr that way, we know that
206 00:25:56.920 ⇒ 00:26:01.559 Uttam Kumaran: this this pass is staging, and that when we get pushed to prod, it’s gonna succeed.
207 00:26:01.580 ⇒ 00:26:28.730 Uttam Kumaran: Typically what happens? Right? You you push a Pr, it looks really good. It then goes to the production. And then you’re missing a comma, or you’re there’s a table, alias that’s missing. So this is our process of catching this Dicd is is short for continuous integration, continuous development meaning the integration process or continuous deployment meaning the integration process and the process of deploying this to production is seamless. And it’s continuous.
208 00:26:29.044 ⇒ 00:26:38.220 Uttam Kumaran: And this is really common in software practice. And it’s something that we we adopt as well, and so one of the things that I go and look quickly is, I’m like, Okay, it failed.
209 00:26:38.699 ⇒ 00:26:42.439 Uttam Kumaran: I’m gonna go look at the Dbt build step that failed.
210 00:26:42.720 ⇒ 00:26:48.680 Uttam Kumaran: And here classic Dbt process, it looks like there’s an error 1st in them ads.
211 00:26:49.205 ⇒ 00:26:50.980 Uttam Kumaran: And then it looks like
212 00:26:51.850 ⇒ 00:26:56.439 Uttam Kumaran: great. There’s there’s probably a semicolon at the end of demands, Away
213 00:26:56.998 ⇒ 00:26:58.730 Uttam Kumaran: and so what I can do.
214 00:26:58.730 ⇒ 00:26:59.700 Awaish Kumar: Oh no!
215 00:27:01.450 ⇒ 00:27:07.420 Uttam Kumaran: Yeah, I could go back and then basically go to dim ads, and probably yeah, see that at the end.
216 00:27:07.560 ⇒ 00:27:14.699 Uttam Kumaran: But that’s a pretty common like. Dbt, error that happens. So when you push a Pr. It’s always helpful to
217 00:27:15.130 ⇒ 00:27:20.820 Uttam Kumaran: open the Pr. And then just check whether it’s run or not, because right now I would say.
218 00:27:21.400 ⇒ 00:27:30.699 Uttam Kumaran: this is the number one thing to fix right. There’s nothing for me to review until it runs. So this is something that the individual engineer can work on. Let’s say it did run.
219 00:27:30.900 ⇒ 00:27:42.499 Uttam Kumaran: And I’m gonna continue just looking at the Pr. I’m checking for a couple of things. One I’m looking at. If the tables match our naming conventions right like we had fact int
220 00:27:42.680 ⇒ 00:27:47.560 Uttam Kumaran: dim, you know, sort of tables. So I’m looking at the naming
221 00:27:48.065 ⇒ 00:27:50.260 Uttam Kumaran: I would say. These look like
222 00:27:50.420 ⇒ 00:27:58.959 Uttam Kumaran: old legacy tables. So I’ll probably leave a comment to say, can we make sure that like? Why isn’t this named as a?
223 00:27:59.210 ⇒ 00:28:02.129 Uttam Kumaran: Is this a intermediate table? Is this a raw table?
224 00:28:02.470 ⇒ 00:28:04.380 Uttam Kumaran: Right? So I’ll probably ask about that.
225 00:28:04.938 ⇒ 00:28:10.030 Uttam Kumaran: So what I’m just looking at to just see like, okay, is generally the structure of of things. Okay.
226 00:28:12.250 ⇒ 00:28:20.160 Uttam Kumaran: my, the, my, probably my last. My comment here will be, this is standardized product name from north beam data.
227 00:28:20.890 ⇒ 00:28:22.069 Uttam Kumaran: I would say.
228 00:28:22.260 ⇒ 00:28:28.499 Uttam Kumaran: this reject seems like something we could use across any sort of input data. So I wonder if we can
229 00:28:29.410 ⇒ 00:28:39.570 Uttam Kumaran: combine this. There’s an existing macro already here. If you go to Dbt project and Macros, there’s already a standardized product name.
230 00:28:40.100 ⇒ 00:28:42.459 Uttam Kumaran: My comment will be. Can we combine these 2?
231 00:28:43.420 ⇒ 00:28:49.240 Uttam Kumaran: Right? Because all this Macro does is take in a name and then categorize it. So that’ll probably be another comment of mine.
232 00:28:50.690 ⇒ 00:29:06.450 Uttam Kumaran: and then to. I’ll kind of give you a sense of how you can actually start this review, I guess, while we’re doing it. So I might as well, I might as well just leave these comments. So you click, you click, start review, and it’ll say, review changes. So one thing I’m gonna do here is leave this comment here. And I’m gonna say, can we combine
233 00:29:06.570 ⇒ 00:29:16.770 Uttam Kumaran: with the existing product standardized product? Name Macro?
234 00:29:18.540 ⇒ 00:29:23.130 Awaish Kumar: The existing one actually has one parameter.
235 00:29:23.320 ⇒ 00:29:24.519 Awaish Kumar: Okay, here we are.
236 00:29:24.520 ⇒ 00:29:25.650 Uttam Kumaran: Oh!
237 00:29:25.650 ⇒ 00:29:27.879 Awaish Kumar: We need 3 different parameters.
238 00:29:28.390 ⇒ 00:29:41.089 Uttam Kumaran: So then can I? So then, instead, my given that like, let’s say so. What if? Let’s let’s walk through actually like how that this conversation would happen in Github, I would leave this. I would leave it as a review, a way should respond
239 00:29:41.230 ⇒ 00:29:45.360 Uttam Kumaran: with like, Oh, actually, we need this 3 parameter model.
240 00:29:45.540 ⇒ 00:29:55.740 Uttam Kumaran: And then my next comment will be, can we rename this to standardized product?
241 00:29:55.890 ⇒ 00:29:58.670 Uttam Kumaran: Name from add
242 00:29:59.150 ⇒ 00:30:04.119 Uttam Kumaran: data that way, we can use it on any data source that has add name, asset campaign.
243 00:30:04.630 ⇒ 00:30:08.629 Uttam Kumaran: It’s a small thing. It’s not really like the end of the world. But
244 00:30:08.770 ⇒ 00:30:22.043 Uttam Kumaran: again, if we’re doing a Review. I want to sort of pick up the small things that that could make this a little bit better that way. Any ad source. We’re using this, we can use this, and that makes sense my, the other suggestion. A wish is
245 00:30:22.450 ⇒ 00:30:24.529 Uttam Kumaran: We probably could combine this
246 00:30:24.670 ⇒ 00:30:34.569 Uttam Kumaran: with the other one, and sort of do some sort of parameters where, if it’s null, do something, if it’s other. But but I think it’s probably over complicating. We could just rename this.
247 00:30:35.050 ⇒ 00:30:39.462 Uttam Kumaran: I’m pretty happy. My comment on this one is, gonna be
248 00:30:41.120 ⇒ 00:30:42.390 Uttam Kumaran: This doesn’t.
249 00:30:42.390 ⇒ 00:30:42.720 Awaish Kumar: But.
250 00:30:42.720 ⇒ 00:30:44.249 Uttam Kumaran: Follow. Yeah. Go ahead.
251 00:30:46.170 ⇒ 00:30:48.679 Awaish Kumar: No about the other one like add data is.
252 00:30:48.840 ⇒ 00:30:52.579 Awaish Kumar: I’m not sure like it has 3 add at certain campaign.
253 00:30:52.580 ⇒ 00:30:53.489 Awaish Kumar: So yeah, I guess.
254 00:30:54.920 ⇒ 00:30:59.259 Uttam Kumaran: That’s pretty common for any ad platform we’re gonna use. We’ll have these 3.
255 00:31:01.760 ⇒ 00:31:07.940 Uttam Kumaran: We’ll, we’ll we’ll we’ll leverage these 3 dimensions. Google, Facebook, bing, tick, tock.
256 00:31:08.477 ⇒ 00:31:12.050 Uttam Kumaran: snapchat. They all will have this concept of add ad set campaign.
257 00:31:12.530 ⇒ 00:31:16.479 Uttam Kumaran: So I feel pretty comfortable if we want to create a generic macro like this.
258 00:31:18.245 ⇒ 00:31:18.819 Awaish Kumar: Okay.
259 00:31:19.400 ⇒ 00:31:26.350 Caio Velasco: I’m not. Yeah. That was what I was. Gonna say, I’m not extremely experienced with Macros, but I believe you can somehow
260 00:31:26.620 ⇒ 00:31:29.880 Caio Velasco: create to input parameters or something. And then it.
261 00:31:29.880 ⇒ 00:31:31.610 Uttam Kumaran: Exactly. So, Macro Macro.
262 00:31:31.610 ⇒ 00:31:38.999 Uttam Kumaran: So yeah, exactly. The way you should think about Macros is, if you’re writing SQL code, that’s the same across 2 or more files.
263 00:31:39.230 ⇒ 00:31:40.480 Uttam Kumaran: It should be a macro.
264 00:31:40.880 ⇒ 00:32:01.020 Uttam Kumaran: What that allows is it allows us to go make a change in logic in one place, and then that gets sent to every single downstream. So the the key things you should be looking for if you’re like deciding on. If this is a macro, large case, when’s like this, where it’s hard to iterate. And then you’re like, Okay, I need to go make this change in 10 other places,
265 00:32:01.650 ⇒ 00:32:08.909 Uttam Kumaran: other sort of functions like any sort of udfs or other python functions that you want to put into a macro.
266 00:32:09.200 ⇒ 00:32:12.439 Uttam Kumaran: Usually it’s this sort of categorization which is like
267 00:32:13.990 ⇒ 00:32:19.013 Uttam Kumaran: customer to segment categorization. You may use that in 10 different places. Right?
268 00:32:19.810 ⇒ 00:32:22.889 Uttam Kumaran: it’s common to do that that there and that way.
269 00:32:23.030 ⇒ 00:32:30.349 Uttam Kumaran: Anybody in the Dbt can use the macro and not worry about like, also copying the specific logic things like that.
270 00:32:31.502 ⇒ 00:32:36.589 Uttam Kumaran: So that’s 1 thing. There, I think. Second, I’m gonna say, this doesn’t follow the in naming convention.
271 00:32:37.610 ⇒ 00:32:43.780 Uttam Kumaran: If this is a legacy table, should just.
272 00:32:44.770 ⇒ 00:32:50.349 Uttam Kumaran: I mean we should make a decision to leave in another folder.
273 00:32:50.670 ⇒ 00:33:02.409 Uttam Kumaran: My point here away is that I’m pretty sure this is like the we have some legacy tables that we moved into Dbt. However they still be, they still may be getting queried by a downstream dashboard.
274 00:33:02.580 ⇒ 00:33:03.280 Uttam Kumaran: So
275 00:33:03.730 ⇒ 00:33:17.029 Uttam Kumaran: one, I want to know, like, if we’re gonna if we’re gonna bring it into ours, we should follow our own naming convention, and we should migrate those downstream models. If we’re not, then we should leave this in place somewhere until we can go do that replacement. So
276 00:33:17.140 ⇒ 00:33:20.790 Uttam Kumaran: I’ll leave that comment. In 1st order. So.
277 00:33:20.790 ⇒ 00:33:21.220 Awaish Kumar: Right.
278 00:33:21.220 ⇒ 00:33:22.010 Uttam Kumaran: Yeah. Go ahead.
279 00:33:23.110 ⇒ 00:33:27.389 Awaish Kumar: Yeah, I have actually in this. This is a kind of new table. Because
280 00:33:27.620 ⇒ 00:33:35.100 Awaish Kumar: I, we have moved this mapping into a macro. And we have like changed little bit of structure here.
281 00:33:35.560 ⇒ 00:33:37.369 Awaish Kumar: and the rejects part.
282 00:33:37.370 ⇒ 00:33:37.740 Uttam Kumaran: Okay.
283 00:33:37.740 ⇒ 00:33:40.429 Awaish Kumar: Was in the table. We have moved it out as a macro.
284 00:33:40.430 ⇒ 00:33:45.320 Uttam Kumaran: Then I would just rename this to ours. Yeah, I would just rename it to something in our world.
285 00:33:46.400 ⇒ 00:33:46.950 Uttam Kumaran: and.
286 00:33:46.950 ⇒ 00:33:47.630 Awaish Kumar: Yes.
287 00:33:47.630 ⇒ 00:33:49.080 Uttam Kumaran: That’s that’s totally fine.
288 00:33:51.830 ⇒ 00:34:03.220 Uttam Kumaran: So I’ll let you. With that, I’m looking at this. I mean, this, basically looks basically looks fine.
289 00:34:05.910 ⇒ 00:34:07.949 Uttam Kumaran: Looking at in source. Spend?
290 00:34:09.280 ⇒ 00:34:12.009 Uttam Kumaran: Yeah, this looks fine. My, only
291 00:34:13.460 ⇒ 00:34:18.959 Uttam Kumaran: I’ll probably anytime there’s like a right join or like a not a basically, not a left or an inner join.
292 00:34:19.100 ⇒ 00:34:22.949 Uttam Kumaran: I’m sort of looking to see like, okay, was there a reason that we did that?
293 00:34:23.529 ⇒ 00:34:30.080 Uttam Kumaran: It looks like we probably want both spends on any dates dance. Okay?
294 00:34:31.212 ⇒ 00:34:33.500 Uttam Kumaran: Dim ads, okay, great.
295 00:34:33.739 ⇒ 00:34:50.879 Uttam Kumaran: This is gonna probably flag an issue. One of the things you can do in reviews is you can actually add a suggestion. So I could, instead of just saying, remove the semicolon, and always just to go remove the semicolon. I can actually just click this, add a suggestion. I can remove the semicolon and add a review comment.
296 00:34:51.030 ⇒ 00:34:56.830 Uttam Kumaran: It will provide this, do a waste where he can just say, commit that right? So
297 00:34:57.210 ⇒ 00:34:59.779 Uttam Kumaran: save everybody a couple seconds there.
298 00:35:00.485 ⇒ 00:35:04.060 Uttam Kumaran: Another one here. So I’m gonna probably I’ll just do the same thing.
299 00:35:04.380 ⇒ 00:35:13.239 Uttam Kumaran: And all of that perfect these look the same.
300 00:35:14.820 ⇒ 00:35:20.860 Uttam Kumaran: I guess my only thing here as platforms.
301 00:35:23.470 ⇒ 00:35:29.389 Uttam Kumaran: I may ask. I guess a wish is any difference like, should we use like a hash here
302 00:35:29.870 ⇒ 00:35:37.539 Uttam Kumaran: versus just like the rows, because maybe this will change if if new platforms come in.
303 00:35:39.850 ⇒ 00:35:43.500 Awaish Kumar: Yes, but the right now our modeling work is
304 00:35:43.830 ⇒ 00:35:54.860 Awaish Kumar: like we don’t have incremental models. So anyway, we will be running the downstream models as well if this model gets run again, so the Ids will update. Accordingly.
305 00:35:55.050 ⇒ 00:35:57.800 Uttam Kumaran: I guess I’m just worried that, like the Id shouldn’t update
306 00:35:59.620 ⇒ 00:36:03.350 Uttam Kumaran: like we see what I mean. I think we should just do
307 00:36:03.540 ⇒ 00:36:06.420 Uttam Kumaran: like a hash of the platform. Name
308 00:36:07.730 ⇒ 00:36:11.260 Uttam Kumaran: that way, regardless of if a new platform comes in
309 00:36:11.550 ⇒ 00:36:17.429 Uttam Kumaran: the Id. Stay the same, because, you know. So my suggestion would be use. Md. 5.
310 00:36:18.690 ⇒ 00:36:25.240 Luke Daque: And we can make it that standard process right? Anytime we create. Ies should be Mv. 5 or something.
311 00:36:25.460 ⇒ 00:36:31.680 Uttam Kumaran: Yeah. Md, 5 are similar to hash the platform, hey?
312 00:36:32.310 ⇒ 00:36:34.419 Uttam Kumaran: So I’ll leave. I’ll do that.
313 00:36:34.560 ⇒ 00:36:40.850 Uttam Kumaran: The nice thing I’m also always does a really good job at having ids and everything. This is not something that I do. Well.
314 00:36:41.468 ⇒ 00:36:52.410 Uttam Kumaran: so I’m I’m really happy to see that I I will be starting to add that in my Prs as well. And then, yeah, now I’m gonna move on to the fact that performance fact you should all be on a time time series.
315 00:36:52.560 ⇒ 00:36:54.139 Uttam Kumaran: So I’m I’m seeing that
316 00:36:54.585 ⇒ 00:37:01.529 Uttam Kumaran: one of the things that this is small. But I think I’ll leave a comment. You can fix it if you’d like.
317 00:37:02.915 ⇒ 00:37:05.000 Uttam Kumaran: Is aliasing.
318 00:37:05.000 ⇒ 00:37:05.969 Awaish Kumar: Resource, ma’am.
319 00:37:05.970 ⇒ 00:37:10.220 Uttam Kumaran: It’s just aliasing meaning, like, yeah, basically.
320 00:37:10.481 ⇒ 00:37:13.800 Awaish Kumar: Think that source name in the beginning, like the table name.
321 00:37:14.140 ⇒ 00:37:15.649 Uttam Kumaran: Yeah. Or like, if, for example.
322 00:37:15.850 ⇒ 00:37:31.559 Uttam Kumaran: some of these are P. Dot, some of these are north beam. The one thing that’s gonna happen is. Let’s say, north beam. Someone adds one of these columns in the north beam, and it’s also in dim platforms. It’s gonna break right? It’s gonna say, it’s ambiguous. So alias
323 00:37:32.830 ⇒ 00:37:34.250 Uttam Kumaran: all columns.
324 00:37:35.830 ⇒ 00:37:37.090 Uttam Kumaran: That’s an easy one.
325 00:37:39.140 ⇒ 00:37:43.509 Uttam Kumaran: And then I’m gonna look at source. Spend summary.
326 00:37:43.660 ⇒ 00:37:45.469 Uttam Kumaran: This looks pretty fine.
327 00:37:51.220 ⇒ 00:37:53.210 Uttam Kumaran: Okay, okay, I see what you mean.
328 00:37:53.680 ⇒ 00:37:54.690 Uttam Kumaran: Great.
329 00:37:57.180 ⇒ 00:38:00.309 Uttam Kumaran: Let me go. I just want to look at one thing here.
330 00:38:00.920 ⇒ 00:38:05.310 Uttam Kumaran: So this is, spend summary.
331 00:38:07.890 ⇒ 00:38:10.090 Uttam Kumaran: Where is spend summary?
332 00:38:14.275 ⇒ 00:38:14.790 Uttam Kumaran: Okay.
333 00:38:20.140 ⇒ 00:38:28.900 Uttam Kumaran: I’m almost wondering. I guess it’s okay. I my, my thought here was, maybe we should just create.
334 00:38:29.330 ⇒ 00:38:33.370 Uttam Kumaran: We should just create one Cac field for both.
335 00:38:39.130 ⇒ 00:38:44.609 Awaish Kumar: Other like we have this in 2 formats, like the source span summary
336 00:38:45.670 ⇒ 00:38:47.649 Awaish Kumar: is, has has the kind of.
337 00:38:48.050 ⇒ 00:38:49.639 Awaish Kumar: But you are suggesting, like.
338 00:38:50.180 ⇒ 00:38:51.889 Uttam Kumaran: Yeah, yeah, yeah, yeah.
339 00:38:54.970 ⇒ 00:38:59.860 Uttam Kumaran: yeah. Medicac, okay. Okay. Okay. I see. I see, I see, okay, this is fine.
340 00:39:00.120 ⇒ 00:39:03.819 Uttam Kumaran: I don’t think this is a big deal, just being annoying.
341 00:39:03.820 ⇒ 00:39:04.580 Awaish Kumar: Oh, okay.
342 00:39:05.153 ⇒ 00:39:06.299 Uttam Kumaran: And then.
343 00:39:06.900 ⇒ 00:39:16.019 Uttam Kumaran: yeah, my, my thing here is like, are we okay with doing select stars? I guess this is probably a question for everybody. I sort of get nervous
344 00:39:16.550 ⇒ 00:39:21.680 Uttam Kumaran: because I want us to have control over what comes into tables.
345 00:39:21.950 ⇒ 00:39:26.120 Uttam Kumaran: But also, I’m okay with this for now.
346 00:39:27.034 ⇒ 00:39:30.459 Uttam Kumaran: It’s sort of a question about like how we do aliasing as well like
347 00:39:30.790 ⇒ 00:39:34.790 Uttam Kumaran: anytime that something can go wrong. It’s gonna go wrong.
348 00:39:34.980 ⇒ 00:39:38.530 Uttam Kumaran: So I just wanna make sure that if we are doing select stars.
349 00:39:38.670 ⇒ 00:39:41.750 Uttam Kumaran: we’re comfortable with the fact that anytime we update this.
350 00:39:42.060 ⇒ 00:39:44.179 Uttam Kumaran: This table is going to get all those updates.
351 00:39:44.580 ⇒ 00:39:51.420 Uttam Kumaran: We’re okay with that. Then I’m fine. With that, I think testing may help to solve some of those problems. But roughly, I feel, okay with that
352 00:39:52.493 ⇒ 00:39:58.319 Uttam Kumaran: and then, yeah, the last thing is, I know, across the board we probably all have opinions on.
353 00:39:58.440 ⇒ 00:40:03.541 Uttam Kumaran: If we should tab things out, how things should be ordered. We’re I’m gonna I’m gonna be adding this,
354 00:40:04.020 ⇒ 00:40:10.990 Uttam Kumaran: I’m basically gonna be adding this thing called SQL fluff, SQL, fluff is a SQL formatter.
355 00:40:11.743 ⇒ 00:40:14.649 Uttam Kumaran: And I wish you use SQL. Fluff before.
356 00:40:16.582 ⇒ 00:40:17.460 Awaish Kumar: Yes, I have.
357 00:40:17.460 ⇒ 00:40:21.920 Uttam Kumaran: Okay, okay? So basically, I’m gonna add it to our free commit or
358 00:40:22.940 ⇒ 00:40:38.934 Uttam Kumaran: yeah, or post commit basically whenever you push code into the repo like, whenever you push code into a Pr. It will actually format everything automatically up to our standards. So you don’t really. You can push, for example, like something that looks like
359 00:40:39.900 ⇒ 00:40:47.639 Uttam Kumaran: Let’s like a good example is like, let’s see like this. Right?
360 00:40:48.330 ⇒ 00:40:51.729 Uttam Kumaran: My, my point would be like, Okay, put these on each new row.
361 00:40:51.890 ⇒ 00:41:06.730 Uttam Kumaran: The SQL. Fluff will actually go do that for us. So it’ll make sure that all of our files in the repo are consistent with our formatting. This doesn’t mean things will get aliases. This doesn’t mean logic gets formatted, but it’s just like the look and feel of everything. So
362 00:41:06.920 ⇒ 00:41:12.610 Uttam Kumaran: I’ll be adding that so, for now I’m not really too critical about these sorts of things.
363 00:41:14.400 ⇒ 00:41:17.840 Uttam Kumaran: I used to like my 1st job. Yeah, literally, they’d be like.
364 00:41:18.490 ⇒ 00:41:23.930 Uttam Kumaran: send it back, send it back, send it back. So I’m a little bit like more.
365 00:41:24.160 ⇒ 00:41:31.200 Uttam Kumaran: We will just get more critical over time right? And I expect all of us right to have a really critical eye when we look at Prs.
366 00:41:31.310 ⇒ 00:41:42.409 Uttam Kumaran: For the most part, these days. I’m looking at every Pr. But this is sort of the process. I think everybody should go through again. I wouldn’t have looked at this Pr. If if the Dbt thing didn’t pass
367 00:41:42.948 ⇒ 00:41:57.180 Uttam Kumaran: right? And it’s like, so I basically like, okay, that needs to pass before I get, I get alerted to go review it. The other thing is the reviewing, alert? We’re gonna start to have come in slack. So that way. You’re not like sitting here looking at your github notifications.
368 00:41:57.280 ⇒ 00:42:06.889 Uttam Kumaran: I would also probably turn off the emails. If you’re getting them, it’s a little bit annoying. I’m gonna try to streamline everything to come into slack. Basically. So if you’re tagged in a pr. It’ll it’ll come into there.
369 00:42:09.720 ⇒ 00:42:13.919 Luke Daque: Are we adding a SQL. Fluff in the Ci. CD. As well.
370 00:42:14.690 ⇒ 00:42:17.471 Uttam Kumaran: Yeah, I’m deciding I may add it as like a
371 00:42:17.750 ⇒ 00:42:18.360 Luke Daque: Quest.
372 00:42:19.610 ⇒ 00:42:23.009 Uttam Kumaran: Yeah, I may add it as a Po, as like a post commit.
373 00:42:23.560 ⇒ 00:42:25.180 Uttam Kumaran: I don’t know. I have to decide
374 00:42:25.707 ⇒ 00:42:30.399 Uttam Kumaran: whether I want it to run on everybody’s machines, or whether I want it to run after the fact
375 00:42:30.870 ⇒ 00:42:37.859 Uttam Kumaran: and like you can do 2 things. I could have it, everybody install it, and that way, anytime you commit it’ll change yours.
376 00:42:38.540 ⇒ 00:42:47.200 Uttam Kumaran: I think that may be better. The other thing I could do is I can also have it. Look at every new Pr go through, and then it will, it will commit
377 00:42:48.090 ⇒ 00:42:52.590 Uttam Kumaran: however, my worry there is that like, let’s say it commits, and then you’re like.
378 00:42:52.750 ⇒ 00:42:55.169 Uttam Kumaran: you haven’t pulled, put, pulled that down.
379 00:42:55.290 ⇒ 00:42:58.100 Uttam Kumaran: and you make a change. It may like cause some issues.
380 00:42:58.510 ⇒ 00:43:06.030 Uttam Kumaran: So I’m gonna I’m gonna probably try the 1st version where we each install it on our machine. Or it’s basically part of the requirements file.
381 00:43:06.300 ⇒ 00:43:09.198 Uttam Kumaran: And it’s part of the it’ll it’ll run on
382 00:43:10.450 ⇒ 00:43:15.370 Uttam Kumaran: it’ll basically run as a pre commit. So as soon as you hit commit on Github, it’ll it’ll run
383 00:43:15.780 ⇒ 00:43:19.779 Uttam Kumaran: on your entire on the Dbt Project folder, basically.
384 00:43:22.920 ⇒ 00:43:25.059 Luke Daque: Cool. Yeah, that makes sense.
385 00:43:26.070 ⇒ 00:43:27.800 Uttam Kumaran: Okay, cool.
386 00:43:28.270 ⇒ 00:43:36.300 Uttam Kumaran: Thank you guys for bearing with me there. But yeah, I mean, I think for Kyle and for Luke. I think that’s probably a little bit more context into the best way to do reviews.
387 00:43:37.830 ⇒ 00:43:46.719 Uttam Kumaran: again, we sort of as issues come up. The best way that we’re going to start to triage is 1st go to look at who made the Pr. Who reviewed the Pr.
388 00:43:46.860 ⇒ 00:43:55.740 Uttam Kumaran: And then we sort of learn. And we’ll that’s sort of our retrospective process. But I’m I’m really happy to have shared this today. If there’s any questions about this or this process.
389 00:43:56.450 ⇒ 00:44:00.610 Uttam Kumaran: Let me know. But but yeah, any other questions before we jump.
390 00:44:03.650 ⇒ 00:44:07.239 Caio Velasco: No, no, I think we should just tag this video recording just to.
391 00:44:07.240 ⇒ 00:44:07.700 Uttam Kumaran: Yes.
392 00:44:07.700 ⇒ 00:44:09.809 Caio Velasco: Pr. Stuff. That would be very good.
393 00:44:09.810 ⇒ 00:44:15.909 Uttam Kumaran: Yes, I’ll I’ll take this and put it into loom, and then I’ll I’ll have it get cut down and put into documentation.
394 00:44:18.470 ⇒ 00:44:23.500 Uttam Kumaran: Okay, thanks. Guys. Talk to you in in a little bit. And then, yeah, have a good day. Otherwise.
395 00:44:23.690 ⇒ 00:44:24.810 Caio Velasco: Thank you. Thank you. Appreciate.
396 00:44:24.810 ⇒ 00:44:25.959 Uttam Kumaran: Cool thanks guys.
397 00:44:25.960 ⇒ 00:44:27.370 Luke Daque: Sounds good thanks. Guys.
398 00:44:27.560 ⇒ 00:44:28.220 Uttam Kumaran: Bye.