Meeting Title: Javy-Project-Internal-Review Date: 2024-11-14 Meeting participants: Luke Daque, Payas Parab
WEBVTT
1 00:06:16.110 ⇒ 00:06:17.480 Payas Parab: Hey, Ryan, how are you?
2 00:06:18.960 ⇒ 00:06:20.660 Luke Daque: Hi! Piazz! How’s it going.
3 00:06:21.250 ⇒ 00:06:21.880 Payas Parab: Hold on!
4 00:06:23.040 ⇒ 00:06:24.142 Payas Parab: How’s it going.
5 00:06:24.510 ⇒ 00:06:25.960 Luke Daque: Yeah. Doing well. Can you hear me?
6 00:06:26.250 ⇒ 00:06:27.350 Payas Parab: Yeah, I can hear you.
7 00:06:27.350 ⇒ 00:06:28.719 Luke Daque: Oh, cool! Nice!
8 00:06:29.803 ⇒ 00:06:49.319 Payas Parab: Yeah, awesome. I just sent some items in the chat. We can quickly discuss. Yeah. And then we can try and figure out a time to sync with the rest of the team. I just like also wanted to let you know, like I I sent through Tom and discuss with Robert. Some just like, I think, like for our collaboration, we just might need to be a little more organized, myself included. But it’s not like it’s not.
9 00:06:49.320 ⇒ 00:06:50.020 Luke Daque: Interesting.
10 00:06:50.260 ⇒ 00:06:54.699 Payas Parab: I wanna be clear that it’s not just a you thing. It’s just like, how can we collaborate better.
11 00:06:55.040 ⇒ 00:06:55.550 Luke Daque: Wow!
12 00:06:55.550 ⇒ 00:06:56.967 Payas Parab: Because I just
13 00:06:57.770 ⇒ 00:07:25.930 Payas Parab: yeah, it. It’s just like to make it a little more organized in my end, too. Like, I just like randomly drop stuff in the slack, which I know for you, for all the things you have to do isn’t like super duper, helpful, right when it’s just like a bunch of loose items in the slack. So if there’s a better process, you guys wanna do. And we can implement with Nico and everything, it might just make everyone’s life easier, you know. So it’s not me sending random, slack messages and things like that. I that was like I just wanted to make sure you knew I was gonna ping Utam as well like the feedback was meant to be like, how can we
14 00:07:25.930 ⇒ 00:07:30.609 Payas Parab: make this all better? Because I’m just asking you for a bunch of random stuff, making some random pull requests.
15 00:07:30.610 ⇒ 00:07:31.000 Luke Daque: Right.
16 00:07:31.000 ⇒ 00:07:37.419 Payas Parab: It can also, the way I do. It isn’t helpful for you guys, either. So if we have a ticketing system set up.
17 00:07:37.550 ⇒ 00:07:57.610 Payas Parab: and then we can like merge things in like groupings. Right? That might just be easier than like loose changes. And then I think the other thing is, I’m just unfamiliar with Dbt in general, frankly. And that whole flow, like, I just believe, from my basic take, is like, it’s a little bit too complicated like. It’s just a little bit too complicated in the way the format it’s currently in
18 00:07:58.052 ⇒ 00:08:18.649 Payas Parab: or, if I can like, easily see right? Like I was told that like with Dbt, we have like easy visibility into like how pipelines are built and like where things are breaking down, how data flows. If you can like, familiarize me with that interface that might help a lot as well, because I’m kind of just like getting a little lost where it’s like, okay, orders and order line. And then, like, where does that originally come from? And what raw data.
19 00:08:18.650 ⇒ 00:08:19.190 Luke Daque: Fine.
20 00:08:19.190 ⇒ 00:08:30.160 Payas Parab: And then there’s like some intermediate tables. And then I’m finding, like, I want to run some like, I just want to play around with analysis. And I have to like randomly use an intermediate table. And I’m like, Well, I shouldn’t be right if we’re gonna like.
21 00:08:30.160 ⇒ 00:08:30.830 Luke Daque: Yeah.
22 00:08:30.830 ⇒ 00:08:31.840 Payas Parab: Ron. So
23 00:08:31.900 ⇒ 00:08:52.869 Payas Parab: maybe it’s just like I need to be a little more educated on it. So if you could like, just kind of show me the Dbt. Workflow a little bit before we get into the actual tactical items that might be a little bit helpful just for me to understand like, how do you trace the data right? And like how things are built and stuff like, how you guys look at it. That would be helpful, because I’m just not familiar with it.
24 00:08:53.110 ⇒ 00:08:56.448 Luke Daque: Yeah, sure. Yeah, to to your point, like,
25 00:08:57.030 ⇒ 00:09:10.279 Luke Daque: yeah, having a better collaboration like you mentioned, like, maybe having a ticketing system or like like for us to like, improve our process, that I definitely agree with that. I did feel that as well like, it’s like we’re all over the place. Basically.
26 00:09:10.280 ⇒ 00:09:14.030 Payas Parab: Yeah, yeah, we’re all. And and I’m part of that, too. By the way, so it’s not just.
27 00:09:14.030 ⇒ 00:09:14.440 Luke Daque: Yeah, it’s.
28 00:09:14.440 ⇒ 00:09:15.690 Payas Parab: Like trying to be like, yeah, you guys.
29 00:09:15.690 ⇒ 00:09:16.260 Luke Daque: Yeah.
30 00:09:16.260 ⇒ 00:09:19.499 Payas Parab: I’m just like randomly sending you stuff. And you’re like, okay, like.
31 00:09:19.500 ⇒ 00:09:21.680 Luke Daque: It’s it’s both both both both ways.
32 00:09:21.680 ⇒ 00:09:22.060 Payas Parab: Yeah.
33 00:09:22.060 ⇒ 00:09:26.020 Luke Daque: Right? So yeah, sometimes, sometimes us as well like, we don’t.
34 00:09:26.610 ⇒ 00:09:31.620 Luke Daque: You’re doing stuff that we should be doing, maybe or like, maybe I’m also doing stuff that you should be doing like we don’t.
35 00:09:31.620 ⇒ 00:09:41.149 Payas Parab: Exactly. And and I’m doing. I’m probably doing it poorly, too. Right like that’s the other issue is like, I’m probably doing it, not as well as you guys can. So I wanna make sure it’s just like clear that I give.
36 00:09:41.150 ⇒ 00:09:41.770 Luke Daque: Yeah.
37 00:09:41.770 ⇒ 00:09:43.200 Payas Parab: And anyway.
38 00:09:43.200 ⇒ 00:09:46.320 Luke Daque: Yeah, yeah, I agree, like, we, we need to like standardize things or something.
39 00:09:46.320 ⇒ 00:09:46.700 Payas Parab: Yeah.
40 00:09:46.700 ⇒ 00:09:49.979 Luke Daque: Those. But yeah, I can show you a bit of DVD, and maybe
41 00:09:50.120 ⇒ 00:09:51.519 Luke Daque: I don’t know if, like
42 00:09:51.810 ⇒ 00:09:55.659 Luke Daque: maybe you can install it in on your end as well, since you already have.
43 00:09:55.660 ⇒ 00:09:57.339 Payas Parab: It might make things easier. Yeah.
44 00:09:57.340 ⇒ 00:10:00.519 Luke Daque: Right, since you already have access to Snowflake or something.
45 00:10:00.938 ⇒ 00:10:07.199 Luke Daque: You can always like use. Dbt, because it’s an it’s open source, anyway. So, yeah, but I can share your little
46 00:10:07.470 ⇒ 00:10:10.029 Luke Daque: bit of what it looks like or like.
47 00:10:12.310 ⇒ 00:10:20.299 Luke Daque: basically, dbt, is a a data transformation. It’s it’s open source, right? And what
48 00:10:20.670 ⇒ 00:10:22.670 Luke Daque: happens? Basically what? Usually
49 00:10:22.800 ⇒ 00:10:25.879 Luke Daque: everything’s under this Dbt project folder.
50 00:10:26.010 ⇒ 00:10:29.942 Luke Daque: And like, there’s a sources. If you go to models.
51 00:10:30.680 ⇒ 00:10:36.746 Luke Daque: there’s a sources. Yaml file. This is basically where all the sources from
52 00:10:38.135 ⇒ 00:10:49.474 Luke Daque: what is snowflake come from like shopify tables, for instance, they’re all coming from Snowflake. Amazon tables, Okando. Gorgeous. Everything that we have in the raw
53 00:10:50.090 ⇒ 00:10:50.920 Payas Parab: Yep.
54 00:10:51.457 ⇒ 00:10:53.070 Luke Daque: Basically the raw.
55 00:10:54.130 ⇒ 00:10:59.680 Luke Daque: After all, database, right? Or yeah, database that’s there. And then
56 00:11:00.265 ⇒ 00:11:05.134 Luke Daque: how Brian did, because Brian started this right? So how Brian form?
57 00:11:05.720 ⇒ 00:11:11.900 Luke Daque: how he did the architecture was. He created intermediate models like for each of the data sources.
58 00:11:12.327 ⇒ 00:11:14.463 Luke Daque: Which are all these in models? Right? And
59 00:11:14.730 ⇒ 00:11:15.440 Payas Parab: Yeah.
60 00:11:15.810 ⇒ 00:11:21.060 Luke Daque: Like, for example, this one in customer. It’s just basically coming from the source.
61 00:11:21.677 ⇒ 00:11:27.039 Luke Daque: This, this specific table customer address. There’s Cts here, like there’s.
62 00:11:27.040 ⇒ 00:11:27.410 Payas Parab: Yes.
63 00:11:27.410 ⇒ 00:11:31.764 Luke Daque: Tag. And then, basically, he’s just like joining all these
64 00:11:32.520 ⇒ 00:11:34.759 Luke Daque: raw tables to get a
65 00:11:35.140 ⇒ 00:11:37.070 Luke Daque: a customer like from.
66 00:11:37.070 ⇒ 00:11:37.430 Payas Parab: Right.
67 00:11:37.430 ⇒ 00:11:38.270 Luke Daque: Customer
68 00:11:39.790 ⇒ 00:11:42.799 Luke Daque: model that has, like all the all the stuff that’s like.
69 00:11:42.800 ⇒ 00:11:43.310 Payas Parab: Yeah.
70 00:11:43.310 ⇒ 00:11:44.940 Luke Daque: Calculations would be here
71 00:11:44.970 ⇒ 00:11:48.329 Luke Daque: so like, I think, like how he envisioned it was like
72 00:11:48.340 ⇒ 00:11:51.779 Luke Daque: all the calculation would be in the int models, so that
73 00:11:53.232 ⇒ 00:12:02.140 Luke Daque: which is like intermediate models, right? And the March models would actually be the final models as much as possible. We don’t have any
74 00:12:02.957 ⇒ 00:12:04.900 Luke Daque: calculations or logic.
75 00:12:04.900 ⇒ 00:12:05.690 Payas Parab: Right, right.
76 00:12:05.690 ⇒ 00:12:07.789 Luke Daque: In here right? Because it’s just selecting
77 00:12:08.510 ⇒ 00:12:11.439 Payas Parab: Just what I find myself right like. And this is just like. And again.
78 00:12:11.440 ⇒ 00:12:11.950 Luke Daque: Right.
79 00:12:11.950 ⇒ 00:12:34.280 Payas Parab: Probably the right way to do this right? Okay, see, this lineage thing is really helpful, because I sit there sometimes and go like, oh, well, like, I don’t know how this got created. And I have to look through like, okay, this came from intermediate table. Then I open intermediate table. Then I’m like, Okay, this is the logic of this raw table. And then I’m like going back to check it. And I’m like small part of me. That’s like I could have just used the raw table right? Like I know how to write a sequel query as well. So I’m like
80 00:12:34.320 ⇒ 00:12:38.430 Payas Parab: I could just use it. So I want to just see. But I also don’t want to like build shitty
81 00:12:38.510 ⇒ 00:12:41.960 Payas Parab: SQL. Code. That’s like gonna break later for them. Because I know, that’s not
82 00:12:42.440 ⇒ 00:12:47.110 Payas Parab: so. I think this helps. So this is a this is a plugin and cursor that you have basically.
83 00:12:47.110 ⇒ 00:12:48.759 Luke Daque: Yes. Are you using cursor as well.
84 00:12:48.760 ⇒ 00:12:50.659 Payas Parab: I use cursor, too. Yeah, so this helps.
85 00:12:50.660 ⇒ 00:12:52.819 Luke Daque: Yeah, I just started using this like.
86 00:12:52.820 ⇒ 00:12:53.140 Payas Parab: Yeah.
87 00:12:53.140 ⇒ 00:12:59.579 Luke Daque: Ago. Actually, because, like Utah mentioned it because I was always using Vs code, and this is pretty much like.
88 00:12:59.880 ⇒ 00:13:01.859 Luke Daque: yeah, it has AI stuff. So yeah.
89 00:13:01.860 ⇒ 00:13:02.570 Payas Parab: Yeah.
90 00:13:02.570 ⇒ 00:13:06.781 Luke Daque: So, yeah, I I installed this specific extension.
91 00:13:07.250 ⇒ 00:13:07.960 Payas Parab: Got it.
92 00:13:08.200 ⇒ 00:13:11.940 Luke Daque: It’s a it’s like it’s called Dbt power user.
93 00:13:12.130 ⇒ 00:13:13.709 Payas Parab: Dbt power user. Okay, I got it.
94 00:13:13.710 ⇒ 00:13:15.909 Luke Daque: Yeah, this one power user for dbt.
95 00:13:15.910 ⇒ 00:13:16.699 Payas Parab: Got it. Okay.
96 00:13:16.700 ⇒ 00:13:21.820 Luke Daque: And as long as I as long as like Dbt is installed, I basically did
97 00:13:22.540 ⇒ 00:13:24.640 Luke Daque: install dbt, here.
98 00:13:24.770 ⇒ 00:13:25.389 Payas Parab: Yeah, yeah.
99 00:13:25.390 ⇒ 00:13:30.560 Luke Daque: And and as well as like all the profiles Yamo in.
100 00:13:31.060 ⇒ 00:13:37.520 Luke Daque: you need to have, like a profile to be able to connect to Snowflake, and, like it has all your credentials in there for.
101 00:13:37.520 ⇒ 00:13:38.119 Payas Parab: Sure. Yep.
102 00:13:38.120 ⇒ 00:13:39.420 Luke Daque: The connection and stuff. So
103 00:13:39.510 ⇒ 00:13:40.957 Luke Daque: once that’s done.
104 00:13:41.580 ⇒ 00:13:42.730 Luke Daque: yeah, it it
105 00:13:43.000 ⇒ 00:13:49.889 Luke Daque: any any problems with the connections you’ll be able to see problems. And the the. This is actually coming from
106 00:13:50.410 ⇒ 00:13:56.790 Luke Daque: the Dbt power user like this one problems. There’s query results like, I can query.
107 00:13:56.870 ⇒ 00:13:58.569 Luke Daque: whatever I have here
108 00:13:59.056 ⇒ 00:14:13.743 Luke Daque: like this specific model. I can just run it. And it shows me I don’t have to go to Snowflake and run this query there. Yeah, it has the lineage, and if we added any documentation it would show here as well. But
109 00:14:14.130 ⇒ 00:14:16.170 Luke Daque: We didn’t add any documentation, but.
110 00:14:16.170 ⇒ 00:14:17.090 Payas Parab: Sure. Sure. Yeah.
111 00:14:17.090 ⇒ 00:14:18.180 Luke Daque: Black at the moment.
112 00:14:18.290 ⇒ 00:14:20.121 Luke Daque: So yeah, basically that
113 00:14:20.930 ⇒ 00:14:21.270 Payas Parab: Go on!
114 00:14:21.270 ⇒ 00:14:21.920 Luke Daque: More look, more.
115 00:14:21.920 ⇒ 00:14:30.210 Payas Parab: So it’s like tracing lineage right? Like, if I’m like, Hey, this cogs flow. I want to understand how it’s like, how would you go about that like, how would you go
116 00:14:30.240 ⇒ 00:14:38.700 Payas Parab: like, Hey, this cogs? It’s coming through. There’s a bunch of calculations. I know it’s coming from like 5 Tran all the way up to here. There’s some intermediate transformations.
117 00:14:38.970 ⇒ 00:14:39.410 Luke Daque: Right.
118 00:14:39.410 ⇒ 00:14:42.359 Payas Parab: Would I go about like understanding that right, that flow.
119 00:14:42.810 ⇒ 00:14:46.309 Luke Daque: Yeah. Like, for instance, we’re looking at fact orders at the moment.
120 00:14:46.310 ⇒ 00:14:47.070 Payas Parab: Yep, correct.
121 00:14:47.420 ⇒ 00:14:50.220 Luke Daque: And then, like we know, it’s coming from
122 00:14:50.270 ⇒ 00:14:54.040 Luke Daque: these 3 intermediate models, Amazon shopify and.
123 00:14:54.040 ⇒ 00:14:54.659 Payas Parab: Sure. Yep.
124 00:14:54.660 ⇒ 00:14:59.089 Luke Daque: My order line, and like we’re not, we don’t care about Amazon for now, so we can
125 00:14:59.340 ⇒ 00:15:03.589 Luke Daque: look into like shopify order. These would be the upstream models
126 00:15:03.860 ⇒ 00:15:07.439 Luke Daque: we’ll be able to see like where the cogs information is. We can always like.
127 00:15:08.154 ⇒ 00:15:12.720 Luke Daque: expand this. This could be a lot. But yeah, we’ll be able to see
128 00:15:16.670 ⇒ 00:15:17.810 Luke Daque: see more.
129 00:15:18.750 ⇒ 00:15:22.699 Luke Daque: Yeah, we will be able to see that one of the sources here is the cogs. 3 p.
130 00:15:22.700 ⇒ 00:15:26.429 Payas Parab: Right. And so I intuitively know that’s the source right? But that exact.
131 00:15:26.430 ⇒ 00:15:27.020 Luke Daque: Right.
132 00:15:27.020 ⇒ 00:15:48.130 Payas Parab: I want to like really quickly, just tap into like, how that’s created like, is there a way to do that into Dbt, right where it’s like in the final table. Fact orders. I’ve got my cogs line, and I know it’s like a series of transformations from upstream tables like, and I want to go check where it came from raw, and how it gets transformed like, how does one go kind of about that?
133 00:15:49.430 ⇒ 00:15:53.668 Luke Daque: Well, this is like the the extent of the lineage that it has.
134 00:15:55.290 ⇒ 00:15:56.909 Luke Daque: But yeah, we can.
135 00:15:57.070 ⇒ 00:16:02.609 Luke Daque: I’m not sure like I I like like for me. I just like read the code, for example, right.
136 00:16:02.820 ⇒ 00:16:03.570 Payas Parab: Sure. Sure. Yeah.
137 00:16:03.570 ⇒ 00:16:04.100 Luke Daque: See that!
138 00:16:04.100 ⇒ 00:16:08.880 Payas Parab: Yeah, I’m just like, like when it comes to like building some of these like views and stuff, like, I think
139 00:16:09.460 ⇒ 00:16:14.039 Payas Parab: I just feel like. There’s like a giant like tracing of precedence that I have to go through.
140 00:16:14.120 ⇒ 00:16:16.780 Payas Parab: That is sort of making the.
141 00:16:17.780 ⇒ 00:16:18.859 Luke Daque: Process, more.
142 00:16:18.860 ⇒ 00:16:26.459 Payas Parab: More tedious like. I know we do a lot of cleaning on the raw tables for sure. But then there’s like sort of like intermediate that I’ve like trace back. And then
143 00:16:26.710 ⇒ 00:16:33.870 Payas Parab: some of the intermediate like, I I don’t know. Maybe I’m just like I I just wanna make sure that like this is the way I know you guys do a lot of problems.
144 00:16:33.870 ⇒ 00:16:34.430 Luke Daque: Projects.
145 00:16:34.430 ⇒ 00:16:36.179 Payas Parab: This is the way you guys do it. It’s just
146 00:16:36.540 ⇒ 00:16:41.360 Payas Parab: from like a like when I’m running some analysis. It does just create a little bit of like
147 00:16:42.170 ⇒ 00:16:48.440 Payas Parab: added confusion, because I can’t like if I see like. Oh, this looks like off to me, and I have to like trace how that happens.
148 00:16:48.520 ⇒ 00:16:51.959 Payas Parab: There’s sort of like this like series of steps that I have to go through
149 00:16:52.090 ⇒ 00:16:53.050 Payas Parab: right.
150 00:16:54.080 ⇒ 00:16:55.629 Luke Daque: Yeah, I don’t know how
151 00:16:57.930 ⇒ 00:17:02.860 Luke Daque: like, how, how we can make the tracing easier or anything.
152 00:17:03.060 ⇒ 00:17:15.929 Luke Daque: It’s just just basically the the high level overview is like all the in models the intermediate models are where the calculations are. So any logic that’s in place you’ll be able to see from the intermediate models.
153 00:17:16.230 ⇒ 00:17:20.040 Luke Daque: And ideally, we’re just using the fact tables. For
154 00:17:20.390 ⇒ 00:17:22.629 Luke Daque: you know, the final outputs that you need to
155 00:17:23.020 ⇒ 00:17:29.100 Luke Daque: like, whether you need to create a a report out of, or a dashboard, or or whatever like, do some.
156 00:17:29.600 ⇒ 00:17:29.960 Payas Parab: Yeah.
157 00:17:29.960 ⇒ 00:17:30.819 Luke Daque: So on.
158 00:17:31.060 ⇒ 00:17:32.389 Luke Daque: But yeah.
159 00:17:32.670 ⇒ 00:17:34.669 Luke Daque: all of the calculations should be
160 00:17:35.180 ⇒ 00:17:38.150 Luke Daque: in one of these intermediate models. Basically.
161 00:17:38.450 ⇒ 00:17:42.468 Luke Daque: yeah, I actually in the intermediate in the order line.
162 00:17:43.170 ⇒ 00:17:46.160 Luke Daque: intermediate model. I actually added, via.
163 00:17:46.160 ⇒ 00:17:47.740 Payas Parab: The shipping cost.
164 00:17:47.740 ⇒ 00:17:51.870 Luke Daque: Yeah, yeah, like the the this logic for all the cog stuff.
165 00:17:52.170 ⇒ 00:17:52.670 Payas Parab: That was coming.
166 00:17:52.670 ⇒ 00:17:53.683 Luke Daque: From that
167 00:17:54.190 ⇒ 00:17:54.730 Payas Parab: The script.
168 00:17:54.730 ⇒ 00:17:57.570 Luke Daque: Our architect artifact. Yeah, the script. Yeah.
169 00:17:59.490 ⇒ 00:18:06.950 Luke Daque: so so it’s also here. But currently, I’m just using whatever is in the the Google sheets right like for the cogs in the insert cost.
170 00:18:06.950 ⇒ 00:18:07.800 Payas Parab: Yeah, yeah.
171 00:18:07.800 ⇒ 00:18:11.259 Luke Daque: Quantity. Everything multiplied by the quantity would be
172 00:18:11.480 ⇒ 00:18:13.229 Luke Daque: what we want here. Right?
173 00:18:13.843 ⇒ 00:18:16.879 Luke Daque: Yeah, that’s what I’m working at at the moment, because I’m like.
174 00:18:17.710 ⇒ 00:18:20.451 Luke Daque: like the field names have changed, and there were like additional like.
175 00:18:20.680 ⇒ 00:18:25.029 Payas Parab: Yeah. And again, that’s that’s on me, right? Like, just like like changing the field names. Just fucks.
176 00:18:25.030 ⇒ 00:18:25.520 Luke Daque: Yeah. No.
177 00:18:25.520 ⇒ 00:18:26.999 Payas Parab: So sorry about that like I, I know.
178 00:18:27.000 ⇒ 00:18:27.890 Luke Daque: No worries.
179 00:18:27.890 ⇒ 00:18:33.070 Payas Parab: And that’s why I think, even like I want to create, like some type of like a dev prod staging process between us, so that I’m not.
180 00:18:33.070 ⇒ 00:18:33.920 Luke Daque: Right.
181 00:18:33.920 ⇒ 00:18:46.697 Payas Parab: Hey? I did this thing, and then I change it, and it breaks a bunch of stuff. So you just put in a lot of work to do these columns that I’m gonna change. It’s not, it’s not. It’s bad part on my part. So anyway, okay, this is clear. Let’s let me just
182 00:18:47.130 ⇒ 00:18:52.090 Payas Parab: This kind of makes sense to me like I I do see how you would kind of trace these order flows. I do still feel like
183 00:18:52.230 ⇒ 00:18:57.660 Payas Parab: I’m not full. This is like a side note of like, I’m not fully convinced that this is like more traceable than like
184 00:18:57.830 ⇒ 00:19:04.664 Payas Parab: creating the raw, or like. You know what I mean. Like, I’m not like fully convinced. But if this is the way, that’s the best practice to do it.
185 00:19:04.900 ⇒ 00:19:06.399 Luke Daque: I can share with you.
186 00:19:07.340 ⇒ 00:19:09.937 Luke Daque: I I think dbt has their
187 00:19:11.440 ⇒ 00:19:12.100 Payas Parab: Yeah, that might.
188 00:19:12.100 ⇒ 00:19:20.169 Luke Daque: Maybe that’s good documentation, like, even in the like, this is like how they structure their projects. They have like staging
189 00:19:20.180 ⇒ 00:19:21.440 Luke Daque: models and.
190 00:19:21.440 ⇒ 00:19:22.299 Payas Parab: Can be in my.
191 00:19:22.300 ⇒ 00:19:29.459 Luke Daque: Models. But looks like for current for Javi coffee, I think, like brian skipped. The staging models. Part
192 00:19:29.820 ⇒ 00:19:31.880 Luke Daque: are essentially just the raw
193 00:19:31.920 ⇒ 00:19:41.489 Luke Daque: files. Where there’s like very basic transformation, whether it’s like name changing the name or like changing the data type stuff like that. But anything like
194 00:19:41.820 ⇒ 00:19:46.910 Luke Daque: we’re joining tables is usually in the intermediate model. So I can. I can send share this to you.
195 00:19:46.910 ⇒ 00:19:49.689 Payas Parab: Yeah, that’d be nice. Let me just review that. Actually, just a.
196 00:19:49.910 ⇒ 00:19:50.930 Luke Daque: Yeah.
197 00:19:58.340 ⇒ 00:19:59.249 Luke Daque: just to you
198 00:20:01.380 ⇒ 00:20:02.630 Luke Daque: in slack.
199 00:20:03.040 ⇒ 00:20:05.596 Payas Parab: Perfect. Got it? Yeah, let me also review that.
200 00:20:08.550 ⇒ 00:20:11.779 Luke Daque: It’s pretty cool. Yeah, if you wanna
201 00:20:12.170 ⇒ 00:20:16.349 Luke Daque: try to play around with Dbt, I can also help like, give you an overview. How like we.
202 00:20:16.350 ⇒ 00:20:17.060 Payas Parab: Sure. Yeah.
203 00:20:17.060 ⇒ 00:20:19.930 Luke Daque: Your DVD project and and stuff. So you can. Yeah.
204 00:20:20.560 ⇒ 00:20:21.890 Payas Parab: Excellent. Okay, awesome.
205 00:20:22.000 ⇒ 00:20:24.755 Payas Parab: That’s super helpful dude. I really appreciate this.
206 00:20:25.230 ⇒ 00:20:28.369 Payas Parab: do you want to get into the some of the tactical stuff I shared in the, in, the.
207 00:20:28.370 ⇒ 00:20:28.880 Luke Daque: Yeah, sure.
208 00:20:28.880 ⇒ 00:20:30.431 Payas Parab: Just quickly review it.
209 00:20:31.230 ⇒ 00:20:31.845 Payas Parab: I
210 00:20:32.800 ⇒ 00:20:33.879 Payas Parab: see. Yeah.
211 00:20:34.420 ⇒ 00:20:35.449 Luke Daque: You. Wanna
212 00:20:35.930 ⇒ 00:20:37.750 Luke Daque: I don’t know if you want to share screen or something.
213 00:20:37.989 ⇒ 00:20:41.340 Payas Parab: Yeah, I just gotta find where which chat this thing is in. Okay? Yep.
214 00:20:42.034 ⇒ 00:20:56.950 Payas Parab: yeah. Okay. So the daily I can share my screen here, the daily aggregation table. I had this like query that was working, and I frankly don’t know what happened like I can go through and look at the pull request. But I just wanna confirm that I’m not making any errors. And and this is like sort of like
215 00:20:57.850 ⇒ 00:21:04.980 Payas Parab: one of my feedback like was like, Okay, like, I do get where we’re coming from with, like the daily aggregation or the monthly aggregation and stuff.
216 00:21:04.980 ⇒ 00:21:05.700 Luke Daque: Fine.
217 00:21:05.700 ⇒ 00:21:16.909 Payas Parab: I I do just think frankly, it adds a layer of too much complication to things. That’s that’s a personal opinion. But you know, I cause one of the things I was like. I had this thing working the other day, as like.
218 00:21:16.950 ⇒ 00:21:25.770 Payas Parab: you know, we have this like group by for date day, right should be like there should be like a number of orders for every single day, right.
219 00:21:25.770 ⇒ 00:21:26.600 Luke Daque: Right.
220 00:21:26.600 ⇒ 00:21:36.120 Payas Parab: And then I had this like date day as the filter, and then I had this thing, and it like looked correct before. And now it looks way too high, like they did not have a million orders in.
221 00:21:36.130 ⇒ 00:21:39.549 Payas Parab: So I wonder if there’s like a duplication error, or some
222 00:21:40.100 ⇒ 00:21:44.320 Payas Parab: type of like aggregation error in the date day. Talk
223 00:21:44.820 ⇒ 00:21:45.740 Payas Parab: here.
224 00:21:48.583 ⇒ 00:21:49.890 Luke Daque: Yeah, that’s
225 00:21:50.980 ⇒ 00:21:55.560 Luke Daque: like, would you know, like how many orders like they should have? And.
226 00:21:55.560 ⇒ 00:21:58.890 Payas Parab: It. It should be like a hundred 20 k. Roughly.
227 00:21:59.460 ⇒ 00:22:03.760 Luke Daque: Gotcha, and orders would be. These are new subs orders right?
228 00:22:04.640 ⇒ 00:22:09.180 Payas Parab: Yeah, so it’s like, this, is this 100. This 1 million is like a a sum of all.
229 00:22:09.180 ⇒ 00:22:10.420 Luke Daque: Not just the total order.
230 00:22:10.420 ⇒ 00:22:14.750 Payas Parab: Yeah, the total should be something like a hundred to 120.
231 00:22:19.730 ⇒ 00:22:21.729 Luke Daque: We’ll have to look into that.
232 00:22:22.720 ⇒ 00:22:25.169 Luke Daque: It doesn’t look like there’s any.
233 00:22:26.050 ⇒ 00:22:29.320 Payas Parab: Yeah. And and this is like another thing like, I wanna discuss right where it’s like.
234 00:22:29.490 ⇒ 00:22:39.898 Payas Parab: I don’t know like how valuable it is for you to like, investigate, and fix this if I can just use the orders table right and use the created app field. It’s like, maybe not perfect. But like I think it gets the job done.
235 00:22:40.130 ⇒ 00:22:41.399 Luke Daque: Like the fat order, stable.
236 00:22:41.400 ⇒ 00:22:45.379 Payas Parab: Yeah, the fact orders. And just rolling with that, even if I have to use.
237 00:22:45.800 ⇒ 00:22:51.870 Payas Parab: even if I don’t have to use sequel, whatever you know what I mean like, I think like, maybe that’s just a better way than some of these like aggregation tables.
238 00:22:52.280 ⇒ 00:22:52.940 Luke Daque: Right.
239 00:22:52.940 ⇒ 00:22:55.290 Payas Parab: Cause something got duplicated. I I just like.
240 00:22:55.710 ⇒ 00:22:56.960 Payas Parab: yeah, like this.
241 00:22:56.990 ⇒ 00:23:04.339 Payas Parab: It’s just like, and and it was working like a couple of days ago, so I don’t know like what happened, or whether there was an update or something like that. But.
242 00:23:04.680 ⇒ 00:23:05.794 Luke Daque: Yeah, I can look into this.
243 00:23:05.980 ⇒ 00:23:15.369 Payas Parab: Yeah, that’s the other thing is I I don’t know if it’s like super valuable. This is like part of the discussion I wanted to have was like, Okay, I’m like, these are final tables, like, once we have them in a good state. I think we should just like somewhat like.
244 00:23:15.420 ⇒ 00:23:21.990 Payas Parab: lock them almost right like it’s like this is the finalized table. And like, let’s just roll with that, you know, instead of.
245 00:23:25.020 ⇒ 00:23:26.210 Luke Daque: Makes sense. Yeah.
246 00:23:34.240 ⇒ 00:23:37.310 Luke Daque: yeah. But I can look into this. I’ll let you know once.
247 00:23:37.620 ⇒ 00:23:38.580 Luke Daque: So
248 00:23:38.830 ⇒ 00:23:42.069 Luke Daque: yeah, once I get something out of this.
249 00:23:57.060 ⇒ 00:23:57.665 Payas Parab: Excellent
250 00:24:06.880 ⇒ 00:24:13.589 Payas Parab: awesome. So that was item one I wanted to trace, the other was, Yeah, the cogs workflow. And properly understanding that, want to make sure we get
251 00:24:13.950 ⇒ 00:24:25.490 Payas Parab: on the same page about that. I think I can try and like view through. But yeah, so it goes from the 5 Tran. Then it kind of pull pulls into the intermediate table, and then there’s some logic applied in the intermediate table
252 00:24:27.020 ⇒ 00:24:29.590 Payas Parab: to like. Determine the cost, and then
253 00:24:29.890 ⇒ 00:24:32.590 Payas Parab: it adds it to the order essentially.
254 00:24:33.510 ⇒ 00:24:38.909 Luke Daque: Right. Yes, both the order lines and the order. So the order lines is like per
255 00:24:39.040 ⇒ 00:24:43.740 Luke Daque: per line. Item, right? And then for the order it’s should be the aggregated
256 00:24:44.980 ⇒ 00:24:45.920 Luke Daque: values.
257 00:24:45.920 ⇒ 00:24:47.479 Payas Parab: And pick and black, cross.
258 00:24:47.480 ⇒ 00:24:53.330 Luke Daque: Yeah, cause. Like, if a single order has multiple products like multiple lines
259 00:24:54.110 ⇒ 00:24:54.690 Luke Daque: like
260 00:24:55.380 ⇒ 00:24:58.069 Luke Daque: the product, a has quantity, one
261 00:24:58.730 ⇒ 00:25:02.957 Luke Daque: product B as quantity 2, then it’s aggregated so the quantity would be like
262 00:25:03.620 ⇒ 00:25:06.629 Luke Daque: 4 times, whatever the price was for each. Yeah.
263 00:25:08.500 ⇒ 00:25:10.750 Payas Parab: Okay. Excellent. Awesome.
264 00:25:13.140 ⇒ 00:25:15.570 Payas Parab: great. So that was the other one, the
265 00:25:16.310 ⇒ 00:25:29.980 Payas Parab: I’m gonna wanna double check that logic just to. I just wanna make sure. And then the last thing I wanted to check also, was there some additional categorical fields that I see are being pulled in from shopify? I have these screenshots in the slack that I just sent
266 00:25:30.280 ⇒ 00:25:34.390 Payas Parab: where they have, like a product name for, like a grouping of the product, they have, like
267 00:25:34.430 ⇒ 00:25:36.530 Payas Parab: page. Name the like
268 00:25:36.600 ⇒ 00:25:38.750 Payas Parab: offer name, or like the offer.
269 00:25:39.710 ⇒ 00:25:45.040 Payas Parab: I believe these are sitting somewhere in the shopify raw tables, and or
270 00:25:45.950 ⇒ 00:25:52.639 Payas Parab: I think they’re sitting somewhere in this, because I can see them in the Shopify Admin Portal. So I believe they’re like a metadata tag of some kind.
271 00:25:52.970 ⇒ 00:25:56.279 Luke Daque: Okay? And and you want this in the fact order line.
272 00:25:56.280 ⇒ 00:25:57.100 Payas Parab: Yeah.
273 00:25:57.310 ⇒ 00:25:58.780 Payas Parab: No. Worries. Yeah.
274 00:25:59.000 ⇒ 00:26:00.470 Luke Daque: Is this an order line.
275 00:26:01.270 ⇒ 00:26:02.750 Payas Parab: That hype.
276 00:26:03.260 ⇒ 00:26:04.419 Luke Daque: Or we’ll have to.
277 00:26:04.580 ⇒ 00:26:17.079 Payas Parab: We’ll have to kind of figure that out. But those I do. I think those are like important cuts that they look at. So I wanna make sure we have that somewhere on the roadmap to get those incorporated. I just like at some point is, gonna be like, Hey, I want to see this by offer, or whatever.
278 00:26:17.080 ⇒ 00:26:17.510 Luke Daque: Gotcha.
279 00:26:17.510 ⇒ 00:26:21.559 Payas Parab: That question is gonna come up. And I wanna make sure we’re prepared for it. So somehow, we gotta figure out.
280 00:26:21.770 ⇒ 00:26:27.740 Payas Parab: yeah, like, how do those tags occur in the raw data, and then how do we get those through to the final?
281 00:26:28.050 ⇒ 00:26:31.929 Payas Parab: The final orders? I think it would be orders.
282 00:26:32.230 ⇒ 00:26:34.300 Payas Parab: I want to say orders.
283 00:26:35.430 ⇒ 00:26:35.940 Luke Daque: Yeah, sure.
284 00:26:35.940 ⇒ 00:26:52.580 Payas Parab: Like if if the offer was presented, and then they like add other stuff to their cart, it’s still like that’s the original offer that got them interested. So I think it is like an order level, right like that order officially, is tagged with one of these offers, and that’s like what got them in the door like? Maybe they add other items to the cart right? But
285 00:26:52.790 ⇒ 00:26:54.900 Payas Parab: for the most part we can. You know.
286 00:26:55.730 ⇒ 00:26:56.330 Luke Daque: Yeah.
287 00:26:56.500 ⇒ 00:26:57.600 Luke Daque: makes sense.
288 00:26:58.690 ⇒ 00:27:10.389 Luke Daque: I’ll look into it. I I think this might not be as urgent as the Daily Kpi one. So I can. I can take note of this. But and yeah, add this to like the tasks to look into.
289 00:27:10.870 ⇒ 00:27:11.290 Payas Parab: Awesome.
290 00:27:11.290 ⇒ 00:27:14.330 Luke Daque: Yeah, it’s not as urgent right compared to.
291 00:27:14.330 ⇒ 00:27:19.540 Payas Parab: It’s not as urgent. But I do. I do know it’s gonna come up. So I wanna make sure we get this one going as well. Yeah.
292 00:27:19.710 ⇒ 00:27:21.540 Luke Daque: Gotcha. Okay. Sounds good.
293 00:27:22.370 ⇒ 00:27:23.159 Payas Parab: Excellent.
294 00:27:24.850 ⇒ 00:27:29.090 Payas Parab: yeah. And and I think you know, and this is my my issue as well. Like, maybe we need to like
295 00:27:29.190 ⇒ 00:27:32.580 Payas Parab: just somewhere have like a key. Priorities.
296 00:27:32.960 ⇒ 00:27:33.670 Luke Daque: Yeah.
297 00:27:34.030 ⇒ 00:27:36.909 Payas Parab: I’m wondering if, like Github tickets, or like
298 00:27:38.090 ⇒ 00:27:42.240 Payas Parab: tasks in slack, like any ideas on like what would work well for you.
299 00:27:42.960 ⇒ 00:27:47.650 Luke Daque: I’m fine with Github tickets. But yeah, we will have. We’ll check with maybe
300 00:27:48.753 ⇒ 00:27:54.580 Luke Daque: the team with that, like Nico and Utam, and and maybe also even Ro Robert, to see like
301 00:27:54.590 ⇒ 00:27:56.590 Luke Daque: what would be the best for us. Right?
302 00:27:56.880 ⇒ 00:27:58.349 Luke Daque: Yeah. So what would work?
303 00:27:58.560 ⇒ 00:28:02.410 Luke Daque: But I’m fine with anything like if if we do it in notion or
304 00:28:02.858 ⇒ 00:28:06.199 Luke Daque: tickets, or a different ticketing tool, that’d be fine with me. So.
305 00:28:06.690 ⇒ 00:28:07.650 Payas Parab: Sounds good.
306 00:28:07.650 ⇒ 00:28:11.780 Luke Daque: But, like Github tickets, make sense because we are already using the repo. Anyway.
307 00:28:11.780 ⇒ 00:28:16.979 Payas Parab: Sure. Yeah, yeah, that makes sense. And the Github is connected in there. Right? So would it help if I
308 00:28:17.010 ⇒ 00:28:22.050 Payas Parab: like, made these like things, just issues in Github, like, right now, those things I sent in slack. So we’re not.
309 00:28:22.050 ⇒ 00:28:24.170 Luke Daque: I, yeah, I think that that.
310 00:28:24.170 ⇒ 00:28:28.919 Payas Parab: Let me do that right now then, cause then we can at least start setting up this system. You know where we’re.
311 00:28:28.920 ⇒ 00:28:36.839 Luke Daque: Yeah, and we can directly create tickets out of the issues and like branches out of the tickets as well. So yeah, that’d be cool.
312 00:28:37.245 ⇒ 00:28:40.679 Payas Parab: And then which one? What’s it’s Ryan dash Brainforge is your get a.
313 00:28:40.680 ⇒ 00:28:42.579 Luke Daque: Yes, yes, yes, yes.
314 00:28:47.940 ⇒ 00:28:55.169 Payas Parab: and I am a developer. So I’ll make sure I put some details in there in terms of what we’re looking at and stuff like that. So don’t worry. Alright,
315 00:28:56.870 ⇒ 00:29:12.079 Payas Parab: excellent. So let me go. Make those 3 things that I sent in the slack right now is Github issues. So then you’re able to track, and we can start tracking better in there and then maybe we can like consolidate a few things into like one Pr, so that we don’t just like quickly change like pro.
316 00:29:12.080 ⇒ 00:29:12.620 Luke Daque: Yeah.
317 00:29:12.620 ⇒ 00:29:13.836 Payas Parab: You know what I mean?
318 00:29:14.250 ⇒ 00:29:32.089 Payas Parab: we can consolidate a few issues into it, and and that that feedbacks for me myself where I was like, I’m just making some loose fucking. 5 line changes in a Pr, and it may have broken something, and that’s totally on me, because I feel like, you guys are just like, yeah, it looks fine. But I don’t know. Maybe I did something wrong, like, I’m not that great at this as much as you guys.
319 00:29:33.430 ⇒ 00:29:37.880 Payas Parab: Awesome. Thank you so much. Ryan. Appreciate it. Okay, so I’m gonna put those into Github tickets right now.
320 00:29:41.950 ⇒ 00:29:44.289 Payas Parab: and then we will go from there.
321 00:29:44.650 ⇒ 00:29:45.510 Luke Daque: Yeah, sounds good.
322 00:29:45.510 ⇒ 00:29:55.580 Payas Parab: We’ll have a chat probably later today, about just like how we can improve collaboration. And please make sure you feel free to like feel it’s a 2 way street, right like, hey? Pious! You just like send these things in slack.
323 00:29:55.630 ⇒ 00:30:02.550 Payas Parab: Not super helpful, you know. Like maybe make a ticket. Please make a Github issue feel free to give me that feedback as well. Okay.
324 00:30:02.730 ⇒ 00:30:03.920 Luke Daque: Cool sounds, good.
325 00:30:03.920 ⇒ 00:30:07.129 Payas Parab: Alright, man, appreciate it. Appreciate all that you guys do alright.
326 00:30:07.300 ⇒ 00:30:11.719 Luke Daque: Sure. By the way, can you send me the screenshot of the Daily Kpi? What.
327 00:30:11.720 ⇒ 00:30:12.300 Payas Parab: Yeah, yeah.
328 00:30:12.300 ⇒ 00:30:12.970 Luke Daque: For it.
329 00:30:13.400 ⇒ 00:30:17.830 Payas Parab: Yeah, I’m gonna actually add a link in a screenshot to the the P. The Pr. For you.
330 00:30:17.840 ⇒ 00:30:19.890 Luke Daque: Cool, nice, sounds good. Thank you.
331 00:30:19.890 ⇒ 00:30:22.470 Payas Parab: Or the the issue. Yep, so you have it all. There.
332 00:30:23.300 ⇒ 00:30:24.000 Luke Daque: Cool
333 00:30:24.750 ⇒ 00:30:25.780 Luke Daque: thanks.
334 00:30:25.780 ⇒ 00:30:27.560 Payas Parab: Alright, ma’am, I’ll talk to you later. Alright.
335 00:30:27.610 ⇒ 00:30:29.409 Luke Daque: Talk to you later. Bye, bye.