Meeting Title: DE-Planning-Session Date: 2024-07-08 Meeting participants: Nicolas Sucari, Uttam Kumaran, Atharv Gudi, Patrick Trainer


WEBVTT

1 00:00:54.070 00:00:54.800 Patrick Trainer: Hey! Thorpe!

2 00:00:57.670 00:00:58.700 Atharv Gudi: Hello!

3 00:00:59.140 00:00:59.980 Patrick Trainer: How’s it going.

4 00:01:01.120 00:01:02.635 Atharv Gudi: Good! Good! I

5 00:01:03.500 00:01:04.459 Atharv Gudi: Still.

6 00:01:04.750 00:01:06.780 Atharv Gudi: I think I got the 1st

7 00:01:06.870 00:01:09.760 Atharv Gudi: push and everything. Then I saw your comments as well, so.

8 00:01:10.130 00:01:11.520 Patrick Trainer: Yeah, yeah, yeah.

9 00:01:11.610 00:01:13.207 Patrick Trainer: yeah. So there’s

10 00:01:14.100 00:01:15.170 Patrick Trainer: that

11 00:01:15.350 00:01:21.600 Patrick Trainer: I mean, you didn’t do anything wrong. I think I just instructed. There, I I

12 00:01:22.190 00:01:24.519 Patrick Trainer: messed up in that.

13 00:01:24.590 00:01:29.549 Patrick Trainer: The directory is all kind of weird, but we’ll get that fixed so.

14 00:01:34.250 00:01:35.070 Nicolas Sucari: Hey, guys.

15 00:01:35.330 00:01:35.940 Nicolas Sucari: are you.

16 00:01:36.259 00:01:36.579 Patrick Trainer: Hey!

17 00:01:37.140 00:01:38.530 Patrick Trainer: Doing good! How are you?

18 00:01:40.140 00:01:41.140 Nicolas Sucari: And fine.

19 00:01:42.040 00:01:43.779 Nicolas Sucari: Are you on a standing desk.

20 00:01:45.660 00:01:46.420 Patrick Trainer: Yeah.

21 00:01:46.420 00:01:47.080 Nicolas Sucari: Ice.

22 00:01:47.570 00:01:48.049 Patrick Trainer: Yeah, I mean.

23 00:01:48.050 00:01:50.179 Nicolas Sucari: For your normal desk. But standing. Yeah.

24 00:01:50.180 00:01:54.020 Patrick Trainer: Yeah, I met my normal one, but it. I just moved it up.

25 00:01:55.610 00:01:59.782 Patrick Trainer: they’re also changed it to a different spot in the house to

26 00:02:00.560 00:02:03.020 Patrick Trainer: mix mix things up. So this is.

27 00:02:03.370 00:02:06.096 Patrick Trainer: this will be my new background for a bit.

28 00:02:07.270 00:02:08.199 Nicolas Sucari: That’s nice.

29 00:02:12.620 00:02:13.290 Nicolas Sucari: Hmm.

30 00:02:25.240 00:02:26.470 Patrick Trainer: Boom! What’s going on.

31 00:02:27.920 00:02:30.500 Uttam Kumaran: Hey, guys, sorry I was just moving some stuff around.

32 00:02:32.080 00:02:34.069 Patrick Trainer: You in your new apartment.

33 00:02:34.520 00:02:36.589 Uttam Kumaran: Yes, new house, new house.

34 00:02:36.590 00:02:38.812 Patrick Trainer: Oh, yeah. New house, new house. Excuse me.

35 00:02:43.480 00:02:46.039 Patrick Trainer: how’s it? How’s it? How’s the move been.

36 00:02:47.130 00:02:48.374 Uttam Kumaran: Lou has been

37 00:02:49.280 00:02:50.759 Uttam Kumaran: good. It’s been a lot.

38 00:02:50.770 00:02:54.489 Uttam Kumaran: It’s been hot. It’s been really hot.

39 00:02:55.620 00:02:57.390 Uttam Kumaran: Parents were here.

40 00:02:57.550 00:02:59.790 Uttam Kumaran: And so it’s like a lot of that.

41 00:03:01.250 00:03:06.499 Uttam Kumaran: the house is good. House is coming along. I have a lot of like gardening and stuff to do

42 00:03:06.600 00:03:09.269 Uttam Kumaran: next month, I think. Mainly

43 00:03:09.300 00:03:09.815 Uttam Kumaran: yeah.

44 00:03:10.600 00:03:15.900 Uttam Kumaran: I’m trying to get like the basics down, but it’s when you see things that are out of place.

45 00:03:16.030 00:03:18.309 Uttam Kumaran: I just like can’t have it.

46 00:03:18.380 00:03:21.660 Uttam Kumaran: So the office is better than last week.

47 00:03:21.840 00:03:22.579 Uttam Kumaran: But

48 00:03:23.620 00:03:27.026 Patrick Trainer: And that looks like a pretty good, pretty, good little spot, little.

49 00:03:27.310 00:03:29.310 Uttam Kumaran: Bad. Yeah. I mean, I had this couch

50 00:03:29.660 00:03:30.920 Uttam Kumaran: behind me.

51 00:03:31.410 00:03:31.900 Patrick Trainer: Nice!

52 00:03:32.160 00:03:33.720 Uttam Kumaran: Which is my old couch?

53 00:03:34.496 00:03:37.130 Uttam Kumaran: Which is nice cause then I can

54 00:03:37.580 00:03:41.779 Uttam Kumaran: basically like, How do I just make it so? I don’t leave this room, you know, for as long.

55 00:03:42.050 00:03:44.480 Patrick Trainer: You just put your bed in there and then.

56 00:03:44.875 00:03:46.850 Uttam Kumaran: That’s the that’s the role.

57 00:03:46.850 00:03:48.335 Patrick Trainer: That’s the bed.

58 00:03:49.460 00:03:58.480 Uttam Kumaran: No, it’s just so it’s comfy cause. So you guys know, I mean, I spend like so much time. I was spending a lot of time outside the house at like coffee shops and stuff.

59 00:03:58.530 00:04:06.480 Uttam Kumaran: mainly because it was. I just been working from that apartment for like 2 years, and so go through phases of like being there

60 00:04:06.610 00:04:09.819 Uttam Kumaran: and then like being like, I’m getting depressed. I need to leave.

61 00:04:09.820 00:04:10.270 Patrick Trainer: Right and.

62 00:04:10.270 00:04:19.570 Uttam Kumaran: And then, being like, Well, you’re not even at the apartment you’re paying for. So this time around. I’m trying to spend more time here. And I don’t need to go

63 00:04:19.579 00:04:24.059 Uttam Kumaran: now that I have like a quiet area, basically to work that’s like, separate from the rest of the place. So.

64 00:04:24.060 00:04:27.057 Patrick Trainer: Right? Yeah, no, that’s a that’s a that’s a good call.

65 00:04:27.570 00:04:28.310 Uttam Kumaran: Yeah.

66 00:04:32.241 00:04:34.448 Uttam Kumaran: cool. Let’s talk about

67 00:04:35.830 00:04:38.440 Uttam Kumaran: this week So

68 00:04:38.900 00:04:42.880 Uttam Kumaran: I’m on. I’m on the project tracker. I’m looking at like the de tab

69 00:04:43.562 00:04:46.464 Uttam Kumaran: I guess. Let’s just go through stuff that’s

70 00:04:48.660 00:04:51.589 Uttam Kumaran: in review or block. It looks like nothing

71 00:04:52.100 00:04:55.729 Uttam Kumaran: pressing there on the in progress side.

72 00:04:56.703 00:04:57.810 Uttam Kumaran: I guess

73 00:04:58.080 00:05:02.420 Uttam Kumaran: we yes, last week was kind of like a weird Friday, so we didn’t

74 00:05:02.730 00:05:05.019 Uttam Kumaran: do the Snowflake Key pairs.

75 00:05:05.080 00:05:07.590 Uttam Kumaran: But I wonder, Pat, if we should

76 00:05:08.140 00:05:11.220 Uttam Kumaran: try to just do it like during the week this week.

77 00:05:11.220 00:05:15.780 Patrick Trainer: Yeah, yeah, we can knock that out. I mean, that’s just been kinda

78 00:05:16.090 00:05:22.740 Patrick Trainer: not low priority, but like on back burner. So yeah, we can definitely knock that out this week

79 00:05:23.465 00:05:26.289 Patrick Trainer: the other thing that’s like

80 00:05:26.460 00:05:29.382 Patrick Trainer: semi blocked, I guess would be

81 00:05:30.824 00:05:35.670 Patrick Trainer: author’s 1st pull request it’s blocked on me.

82 00:05:37.310 00:05:39.550 Patrick Trainer: there was. So I

83 00:05:39.730 00:05:42.860 Patrick Trainer: assumed wrong with the

84 00:05:43.890 00:05:48.989 Patrick Trainer: structure of Dbt. Yaml, so apparently you can’t define

85 00:05:49.150 00:05:52.400 Patrick Trainer: your model tests and column tests

86 00:05:52.570 00:05:54.520 Patrick Trainer: in separate.

87 00:05:54.520 00:05:55.430 Uttam Kumaran: Oh, okay.

88 00:05:55.430 00:06:00.189 Patrick Trainer: Like using separate keys or using a light key. So you get like a

89 00:06:00.560 00:06:02.885 Patrick Trainer: like a namespace error. Essentially.

90 00:06:03.410 00:06:05.719 Patrick Trainer: it’s like, or like a duplicate key.

91 00:06:06.160 00:06:12.881 Patrick Trainer: I assumed it was going to merge them. I thought that was the functionality. Maybe it was at 1 point, and they changed that. But

92 00:06:13.380 00:06:16.640 Patrick Trainer: or I’m just totally wrong. Probably that.

93 00:06:17.485 00:06:18.250 Patrick Trainer: But

94 00:06:19.460 00:06:23.949 Patrick Trainer: I am thinking of a way to

95 00:06:25.680 00:06:26.470 Patrick Trainer: like

96 00:06:27.630 00:06:30.150 Patrick Trainer: the issue that I’m

97 00:06:30.480 00:06:32.319 Patrick Trainer: coming with is like.

98 00:06:32.630 00:06:38.639 Patrick Trainer: so we have, like the light, dash, constraint right where we need, like every

99 00:06:38.660 00:06:39.690 Patrick Trainer: file

100 00:06:40.260 00:06:44.240 Patrick Trainer: for a separate yaml, defining each

101 00:06:44.330 00:06:45.650 Patrick Trainer: call column

102 00:06:46.020 00:06:52.259 Patrick Trainer: for light dash specifically. But then we also want to split the tests into something different.

103 00:06:52.763 00:06:54.880 Patrick Trainer: Which we can’t really do.

104 00:06:55.000 00:06:56.410 Patrick Trainer: So it’s like.

105 00:06:56.580 00:06:59.969 Patrick Trainer: because we need 2 things that are like.

106 00:07:00.360 00:07:04.239 Patrick Trainer: like, I feel like the light dash is like constraining

107 00:07:04.570 00:07:11.399 Patrick Trainer: the flexibility of the yaml files, because, like, we need those those column definitions in

108 00:07:12.310 00:07:17.109 Patrick Trainer: for light dash rather than being able to like, just put them into one schema.

109 00:07:18.400 00:07:22.570 Uttam Kumaran: So I wonder if, like we instead, it’s just like

110 00:07:22.930 00:07:30.210 Uttam Kumaran: I mean, the plan is to move off of light dash where I’m working with the ae team to to kind of prioritize that

111 00:07:30.542 00:07:33.039 Uttam Kumaran: I don’t know. It’ll be done this week.

112 00:07:33.943 00:07:40.279 Uttam Kumaran: So I wonder if, like the now that we have a kind of a different understanding of that. I wonder if it’s just like, wait

113 00:07:40.310 00:07:43.680 Uttam Kumaran: for that, and and like move on to something else, or

114 00:07:43.720 00:07:46.819 Uttam Kumaran: because the light desk. Shouldn’t be a constraint soon.

115 00:07:47.340 00:07:48.140 Patrick Trainer: Okay.

116 00:07:48.880 00:07:54.100 Patrick Trainer: okay, yeah, that’s like, that’s what I was thinking. And like, if

117 00:07:55.250 00:07:58.839 Patrick Trainer: so, my idea, like, if light dash.

118 00:07:59.540 00:08:01.989 Patrick Trainer: perfect world light, dash wasn’t here.

119 00:08:02.090 00:08:05.569 Patrick Trainer: we would just also move the

120 00:08:05.970 00:08:07.370 Patrick Trainer: column tests

121 00:08:08.010 00:08:11.270 Patrick Trainer: over to where, like the model tests are.

122 00:08:11.280 00:08:14.209 Patrick Trainer: and then we’d be able to like have

123 00:08:14.640 00:08:17.549 Patrick Trainer: models and columns all in the same

124 00:08:18.890 00:08:35.219 Patrick Trainer: file, basically and so I guess in a nutshell, it’s merging all of those separate yaml files into like logical groups rather than having each one separate, because it’s just like ridiculous to to maintain.

125 00:08:35.600 00:08:36.450 Uttam Kumaran: So.

126 00:08:36.610 00:08:37.590 Uttam Kumaran: oh, Guy.

127 00:08:39.490 00:08:40.330 Uttam Kumaran: yeah.

128 00:08:41.120 00:08:48.390 Uttam Kumaran: what does that mean? You mean having multiple files or having one file, and it’s just like Com. It’s just have like formatted in a different way. Or what do you think.

129 00:08:48.715 00:08:54.580 Patrick Trainer: No, not formatted in a different way. But instead of having like, say, we have, like, we’ve got

130 00:08:54.630 00:09:03.399 Patrick Trainer: all orders. We’ve got order items we’ve got. Like all these different reports in each of those different models has

131 00:09:03.850 00:09:06.150 Patrick Trainer: order items, dot Yaml

132 00:09:06.704 00:09:10.180 Patrick Trainer: all orders, dot Yaml Kim’s report, Dot Yaml.

133 00:09:10.280 00:09:23.140 Patrick Trainer: taking all of those and putting those into like 1 1 schema file. And just so, we’re like managing all of those from one spot and like changing it there.

134 00:09:25.260 00:09:27.630 Patrick Trainer: and going going that route.

135 00:09:28.464 00:09:29.050 Patrick Trainer: Yeah.

136 00:09:29.050 00:09:31.339 Uttam Kumaran: Almost like how the sources file is set up.

137 00:09:31.670 00:09:32.869 Uttam Kumaran: Where’s the whole project?

138 00:09:32.870 00:09:34.469 Patrick Trainer: Yeah, ex, exactly.

139 00:09:41.120 00:09:44.540 Uttam Kumaran: Yeah, I mean, I just I don’t see why

140 00:09:45.270 00:09:46.700 Uttam Kumaran: not

141 00:09:47.700 00:09:50.320 Uttam Kumaran: apart from that file is just gonna be large.

142 00:09:52.060 00:09:53.139 Uttam Kumaran: 8 up.

143 00:09:53.140 00:09:54.050 Patrick Trainer: Couldn’t get.

144 00:09:54.220 00:09:59.719 Patrick Trainer: Yeah, it can get large, but it can also be.

145 00:10:00.460 00:10:00.929 Uttam Kumaran: Please don’t.

146 00:10:01.163 00:10:01.630 Patrick Trainer: Date. It.

147 00:10:01.840 00:10:02.230 Uttam Kumaran: Yeah.

148 00:10:02.230 00:10:10.360 Patrick Trainer: Cleans it up, but it can also be consolidated at the Directory level. So if we just want to do it, for, like.

149 00:10:10.660 00:10:12.040 Patrick Trainer: we have

150 00:10:12.610 00:10:24.589 Patrick Trainer: our like, say, staging Directory. We put our staging models in that staging Dot Yaml. We have our Marts Directory put. We put our marts models into a marts dot gamble.

151 00:10:24.690 00:10:26.600 Patrick Trainer: We have the

152 00:10:26.980 00:10:34.029 Patrick Trainer: yeah like, kpis, right? That’s another one. And we put the the Kpi models in the Kpis dot yaml.

153 00:10:34.140 00:10:35.419 Patrick Trainer: And so it’s like

154 00:10:36.290 00:10:38.579 Patrick Trainer: like we could put everything

155 00:10:38.920 00:10:42.139 Patrick Trainer: in there. But like I think it makes more sense to

156 00:10:42.580 00:10:44.924 Patrick Trainer: group it by directory.

157 00:10:46.290 00:10:47.270 Patrick Trainer: at least just keep.

158 00:10:47.270 00:10:49.710 Uttam Kumaran: Say, okay, let me say, okay with that, because

159 00:10:49.790 00:10:54.220 Uttam Kumaran: the tests themselves doesn’t really matter what these

160 00:10:54.520 00:11:06.550 Uttam Kumaran: like, it doesn’t really matter what the structure of the sequel file is for the test. The tests are more based on what’s the contents of the table? You’re not re like the workflow on the developer side. You’re not really.

161 00:11:07.260 00:11:12.140 Uttam Kumaran: You’re not really like switching back and forth. You’re more switching back and forth from other example. Test

162 00:11:12.490 00:11:13.650 Uttam Kumaran: Amos.

163 00:11:14.272 00:11:23.519 Uttam Kumaran: So that one thing. Second thing, if you’re using like copilot like, it’ll be able to auto. Do those pretty easily. And if everything’s in one file.

164 00:11:23.860 00:11:28.787 Patrick Trainer: Yeah, take the context a lot easier than having everything else open.

165 00:11:29.710 00:11:30.610 Uttam Kumaran: Yeah.

166 00:11:31.160 00:11:36.800 Uttam Kumaran: I mean, I’m I’m fine with that. And then, yeah, just reduces the number of files that are in each thing.

167 00:11:36.960 00:11:39.310 Uttam Kumaran: Right? So yeah, I’m okay with that.

168 00:11:39.710 00:11:40.420 Patrick Trainer: Right

169 00:11:43.106 00:11:46.849 Patrick Trainer: And then I got a better understanding, too, of

170 00:11:47.209 00:11:50.079 Patrick Trainer: what I was wrong about with the test directory.

171 00:11:50.280 00:11:53.760 Patrick Trainer: So there’s like a distinction between

172 00:11:53.940 00:11:55.040 Patrick Trainer: of

173 00:11:55.710 00:11:59.930 Patrick Trainer: like built in or model tests. And then custom tests.

174 00:12:00.110 00:12:03.060 Patrick Trainer: And so you can write basically

175 00:12:03.651 00:12:08.950 Patrick Trainer: like unit sequel that like, if you want to test for, like non null, you’d write.

176 00:12:10.090 00:12:12.620 Patrick Trainer: Select all from

177 00:12:13.280 00:12:16.660 Patrick Trainer: where column is null, like in your sequel.

178 00:12:16.800 00:12:18.579 Patrick Trainer: and then you could drop that

179 00:12:18.600 00:12:19.920 Patrick Trainer: into

180 00:12:20.420 00:12:27.049 Patrick Trainer: the test directory, and it’ll pick that up and run that sequel against the your model tables.

181 00:12:27.310 00:12:32.169 Patrick Trainer: So it’s like they’re they’re custom tests is basically what they’re saying.

182 00:12:33.500 00:12:36.720 Uttam Kumaran: I’m down to move. I’m down to start by moving

183 00:12:37.570 00:12:43.390 Uttam Kumaran: like I don’t. I don’t really see at the moment the need for staging versus

184 00:12:44.898 00:12:48.459 Uttam Kumaran: mart, I think. Just move everything into

185 00:12:50.030 00:12:51.799 Uttam Kumaran: oh, I mean, I guess it’s like

186 00:12:51.850 00:12:54.949 Uttam Kumaran: this is kind of a larger like. Dbt, project thing.

187 00:12:59.430 00:13:07.839 Patrick Trainer: Like I was looking through to like get some inspiration. Whatnot? I started looking through like, get labs, big Dbt, project.

188 00:13:08.020 00:13:08.430 Uttam Kumaran: Yeah.

189 00:13:08.430 00:13:09.230 Patrick Trainer: Tab

190 00:13:09.640 00:13:11.549 Patrick Trainer: and went through there like

191 00:13:11.850 00:13:13.220 Patrick Trainer: yaml spec.

192 00:13:13.400 00:13:16.000 Patrick Trainer: and it’s like 20,000 lines long.

193 00:13:16.823 00:13:17.510 Uttam Kumaran: Really. Yeah.

194 00:13:17.510 00:13:19.240 Patrick Trainer: It. Yeah, it’s massive.

195 00:13:19.430 00:13:22.999 Patrick Trainer: But it also gave me like reassurance that like

196 00:13:23.510 00:13:25.910 Patrick Trainer: that does work in

197 00:13:26.240 00:13:28.100 Patrick Trainer: a developer workflow.

198 00:13:28.300 00:13:28.855 Patrick Trainer: Okay,

199 00:13:30.200 00:13:36.466 Patrick Trainer: it’s kind of what I’ve always done in the past. I’ve always kind of kept it simple that way. But

200 00:13:38.950 00:13:41.020 Patrick Trainer: yeah, there’s there’s just that.

201 00:13:41.500 00:13:47.089 Uttam Kumaran: Okay, let’s do that. I don’t mind. I didn’t think about that. Yeah, I don’t. I think that’s like ideal

202 00:13:47.390 00:13:48.590 Uttam Kumaran: cool. So

203 00:13:48.620 00:13:50.260 Uttam Kumaran: what does it mean for the ticket.

204 00:13:51.237 00:13:58.502 Patrick Trainer: For the ticket. So I created like like a branch ticket that references our thoughts.

205 00:13:59.060 00:14:02.219 Patrick Trainer: pr, and so what we can do

206 00:14:02.460 00:14:04.410 Patrick Trainer: is

207 00:14:04.420 00:14:05.950 Patrick Trainer: we can.

208 00:14:08.160 00:14:14.330 Patrick Trainer: I mean, really just like ship, the scope of our thoughts pr and

209 00:14:14.520 00:14:19.499 Patrick Trainer: restructure into this new it like literally just merge everything into one

210 00:14:19.944 00:14:22.080 Patrick Trainer: and then and then go from there.

211 00:14:22.910 00:14:26.560 Uttam Kumaran: So maybe if you guys so I think if you guys meet

212 00:14:26.810 00:14:31.899 Uttam Kumaran: and we probably will have some time, cause I think we just want to talk up. I just want to talk about that

213 00:14:32.040 00:14:40.979 Uttam Kumaran: email ticket. But you guys, maybe can take the rest of time and just chat about that. I don’t know whether it’s just copy paste, or you could probably run a script to basically

214 00:14:41.560 00:14:43.690 Uttam Kumaran: cat all those files together.

215 00:14:43.690 00:14:44.560 Patrick Trainer: Yeah.

216 00:14:44.780 00:14:46.290 Uttam Kumaran: Or something locally

217 00:14:46.590 00:14:47.740 Uttam Kumaran: right around.

218 00:14:47.870 00:14:56.460 Patrick Trainer: Yeah, either either would work. So it’s like our thought to make like concrete or to make it more concrete. It’d be like

219 00:14:56.580 00:14:58.529 Patrick Trainer: before we were moving.

220 00:14:59.280 00:15:01.660 Patrick Trainer: Just models. Dot test.

221 00:15:01.770 00:15:08.840 Patrick Trainer: Now, we’d just be moving everything but into one file, like even the column definitions and column tests

222 00:15:09.830 00:15:17.600 Patrick Trainer: in instead of like just isolating the test. We’re just now, instead of having multiple yaml files. We’re just gonna have one big one.

223 00:15:18.000 00:15:21.010 Atharv Gudi: One big gamble file, including all the metadata

224 00:15:21.020 00:15:22.520 Atharv Gudi: configurations.

225 00:15:24.020 00:15:28.819 Uttam Kumaran: So the big thing, I think to figure out is like, how does that look in terms of like

226 00:15:31.190 00:15:33.490 Uttam Kumaran: like comments or organization?

227 00:15:33.790 00:15:40.050 Uttam Kumaran: Because, for example, if you still want to go, modify something for one. Yaml, I guess you go search. Just go search that file for

228 00:15:40.570 00:15:41.640 Uttam Kumaran: the table, name.

229 00:15:41.640 00:15:42.839 Patrick Trainer: Model. Name. Yeah.

230 00:15:42.840 00:15:45.210 Uttam Kumaran: Yeah, but having that like

231 00:15:45.530 00:15:47.981 Uttam Kumaran: having those separate would be nice.

232 00:15:49.190 00:15:55.349 Uttam Kumaran: But yeah, that’s I think that’s the only thing there. So yeah, probably a thought, I think if you and pat me, and maybe go through

233 00:15:55.510 00:15:57.060 Uttam Kumaran: doing it for one.

234 00:15:57.430 00:16:05.240 Uttam Kumaran: or doing or like Mer, or just pick a folder, and just like merge a couple and like, agree on the structure. Then it’s probably just doing it, including all of them.

235 00:16:05.280 00:16:06.390 Uttam Kumaran: We’re robbed.

236 00:16:06.480 00:16:12.250 Uttam Kumaran: Yeah, I think this is great. I for me, the the positives are like it just reduces the number of files.

237 00:16:12.290 00:16:13.470 Uttam Kumaran: And

238 00:16:13.890 00:16:18.699 Uttam Kumaran: basically, now we just have, we’re gonna get rid of light dash. So a lot of those

239 00:16:18.710 00:16:20.659 Uttam Kumaran: column definitions and things

240 00:16:20.720 00:16:24.099 Uttam Kumaran: aren’t going to exist. I just wonder what like whether

241 00:16:27.040 00:16:29.300 Uttam Kumaran: light dash is going to.

242 00:16:32.720 00:16:33.960 Uttam Kumaran: not work.

243 00:16:34.130 00:16:35.229 Uttam Kumaran: If we do that.

244 00:16:36.320 00:16:41.119 Patrick Trainer: Like I was trying to think about that, too, and it’s like

245 00:16:42.270 00:16:44.129 Patrick Trainer: at the end of the day, like

246 00:16:44.310 00:16:46.920 Patrick Trainer: Dbt. Is already

247 00:16:47.470 00:16:54.369 Patrick Trainer: merging all of those yaml files like into it’s like into into a manifest right?

248 00:16:54.400 00:16:55.900 Patrick Trainer: Yeah. And

249 00:16:56.230 00:17:00.310 Patrick Trainer: light dash works based off of that manifest.

250 00:17:00.310 00:17:00.820 Uttam Kumaran: Yeah.

251 00:17:00.820 00:17:02.100 Patrick Trainer: And so

252 00:17:03.980 00:17:11.048 Uttam Kumaran: So so maybe it’s like, also, like, I wonder if you should just work on the Pr that does that just moves all the tests into one.

253 00:17:11.630 00:17:12.450 Uttam Kumaran: Yeah, that’s all.

254 00:17:12.450 00:17:13.590 Patrick Trainer: That’s what I’m saying. Yeah.

255 00:17:13.599 00:17:18.469 Uttam Kumaran: Yeah, don’t worry about the call. Don’t worry about the light dash specific stuff.

256 00:17:18.819 00:17:21.519 Uttam Kumaran: and then by the time that Pr is ready

257 00:17:21.679 00:17:28.039 Uttam Kumaran: we’ll we’ll have. We’ll just like kind of like, leave it there until we’re we’re off of light. Dash basically.

258 00:17:28.349 00:17:37.240 Patrick Trainer: Right? Well, so the the issue there is like if we and this is like the light dash, constraint, so like the light dash needs you to

259 00:17:37.779 00:17:45.469 Patrick Trainer: define, like the like, the model key, right? It has, like model name, my model. But if you define

260 00:17:45.509 00:17:48.319 Patrick Trainer: my model in another file.

261 00:17:48.490 00:17:49.020 Uttam Kumaran: Oh!

262 00:17:49.020 00:17:52.160 Patrick Trainer: That’s where that’s where it clashes. So it’s like

263 00:17:53.060 00:17:56.549 Patrick Trainer: light dash is preventing

264 00:17:57.140 00:18:02.809 Patrick Trainer: moving everything. If we keep it in like 2 files like we can’t have 2

265 00:18:03.530 00:18:06.749 Patrick Trainer: similarly named namespaces.

266 00:18:07.400 00:18:09.525 Uttam Kumaran: So we’re not. We’re not even

267 00:18:11.570 00:18:14.539 Uttam Kumaran: like, we’re not making any light dash updates.

268 00:18:15.580 00:18:17.870 Uttam Kumaran: So I wonder whether.

269 00:18:18.320 00:18:20.059 Patrick Trainer: I think it should be fine to be.

270 00:18:20.060 00:18:24.240 Uttam Kumaran: We turn off the compile we’ve we turn off the compile. Validate job

271 00:18:24.600 00:18:25.770 Uttam Kumaran: and

272 00:18:25.790 00:18:30.469 Uttam Kumaran: lightes is still. It’s still hitting Snowflake. So there’s no data refresh issue.

273 00:18:30.600 00:18:35.560 Uttam Kumaran: And then you just keep you just move, just keep moving on it, cause we’re not making any light dash changes.

274 00:18:35.820 00:18:36.470 Patrick Trainer: Okay.

275 00:18:37.680 00:18:39.100 Patrick Trainer: Oh, well, then.

276 00:18:39.480 00:18:44.959 Uttam Kumaran: So then you’re just moving the test to then just ditch. Just do just like ditch the column definitions

277 00:18:45.100 00:18:46.959 Uttam Kumaran: move all the test to one.

278 00:18:47.220 00:18:52.700 Uttam Kumaran: and then we’ll probably try to meet, maybe next week or later, this weekend

279 00:18:53.195 00:18:55.239 Uttam Kumaran: merge it, and just see if it works.

280 00:18:55.240 00:18:57.820 Patrick Trainer: Sweet. Yeah. Yeah. Awesome.

281 00:18:59.840 00:19:00.750 Uttam Kumaran: Okay, cool

282 00:19:01.732 00:19:08.230 Uttam Kumaran: so maybe I’ll so let’s talk about the email. And then maybe I’ll give you guys the rest of this meeting to kind of just chat through that

283 00:19:09.155 00:19:11.179 Uttam Kumaran: so for the

284 00:19:11.680 00:19:13.726 Uttam Kumaran: email, pipeline stuff that

285 00:19:15.450 00:19:17.460 Patrick Trainer: Last part is

286 00:19:18.290 00:19:22.470 Patrick Trainer: I think we talked a little like late last week about

287 00:19:23.440 00:19:32.720 Patrick Trainer: so we need to either run like like a headless chrome, just to like go and click on the button just because it requires

288 00:19:32.730 00:19:33.990 Patrick Trainer: Javascript.

289 00:19:35.200 00:19:38.130 Patrick Trainer: I’ve used that pipeteer

290 00:19:38.150 00:19:42.440 Patrick Trainer: library before that works well, just like allows you to

291 00:19:43.200 00:19:48.740 Patrick Trainer: launch a headless chrome instance and and click that

292 00:19:50.430 00:19:52.040 Patrick Trainer: the

293 00:19:52.470 00:19:55.050 Patrick Trainer: like. The Csv Pdf.

294 00:19:55.540 00:20:04.560 Patrick Trainer: pipeline, etc, like that extractor that’s working great that looks good. The next step there is just like.

295 00:20:04.910 00:20:06.139 Patrick Trainer: after we get the

296 00:20:06.240 00:20:07.610 Patrick Trainer: Csv, like.

297 00:20:07.870 00:20:08.690 Patrick Trainer: I think

298 00:20:08.880 00:20:13.350 Patrick Trainer: we came to agree that we’re gonna like, dump it as a seed.

299 00:20:13.530 00:20:14.040 Patrick Trainer: and.

300 00:20:14.040 00:20:14.660 Uttam Kumaran: Yeah.

301 00:20:14.950 00:20:19.069 Patrick Trainer: Yeah, and then let Dbt handle the loading. I think that’s a

302 00:20:19.600 00:20:24.819 Patrick Trainer: pretty simple slash, like elegant solution. I think that works really well.

303 00:20:26.240 00:20:27.840 Patrick Trainer: and then the

304 00:20:28.350 00:20:29.530 Patrick Trainer: next.

305 00:20:29.580 00:20:31.190 Patrick Trainer: like, I think we do that

306 00:20:31.200 00:20:33.340 Patrick Trainer: the same for

307 00:20:33.800 00:20:46.119 Patrick Trainer: the HTML email. The only part we’re needing to figure out now is just the the clicking on the button, and I think we’ve got a way forward like a path forward. It’s just the implementation step now.

308 00:20:47.850 00:20:52.160 Uttam Kumaran: So why don’t you close out the 2 Csv’s 1st

309 00:20:54.710 00:20:57.180 Uttam Kumaran: and then we can move on to

310 00:20:57.430 00:20:59.220 Uttam Kumaran: the headless, chrome thing.

311 00:20:59.710 00:21:02.159 Uttam Kumaran: Cause that just that just leaves one

312 00:21:02.840 00:21:06.510 Uttam Kumaran: like only the attentive one that basically we need to do that with.

313 00:21:06.690 00:21:08.659 Uttam Kumaran: And then we can tell her, Tom.

314 00:21:09.030 00:21:10.060 Patrick Trainer: Okay. Cool.

315 00:21:10.620 00:21:14.210 Uttam Kumaran: So for so there’s 1 for wise

316 00:21:15.067 00:21:19.610 Uttam Kumaran: what? So one for unisv. There’s 1 for

317 00:21:20.385 00:21:24.830 Uttam Kumaran: direct mail one. There’s direct mail, 2

318 00:21:25.304 00:21:28.829 Uttam Kumaran: so all 3 of those are coming in like, so what like do you wanna

319 00:21:30.360 00:21:34.700 Uttam Kumaran: like? What do you think is best like? If I just flag you on those

320 00:21:34.920 00:21:39.540 Uttam Kumaran: emails. The emails are already in the ticket. Like, what’s.

321 00:21:40.230 00:21:43.420 Patrick Trainer: I think we should have a

322 00:21:43.540 00:21:44.489 Patrick Trainer: like a

323 00:21:45.320 00:21:47.050 Patrick Trainer: dedicated like

324 00:21:47.060 00:21:50.780 Patrick Trainer: brain forge email for like a

325 00:21:51.430 00:21:53.160 Patrick Trainer: like we call it like.

326 00:21:53.590 00:21:55.209 Patrick Trainer: I don’t know, receiving

327 00:21:55.633 00:21:59.746 Patrick Trainer: at brainforce.com, or like some something along those lines.

328 00:22:00.743 00:22:01.889 Patrick Trainer: And then

329 00:22:02.070 00:22:07.229 Patrick Trainer: we will use that as like the the

330 00:22:07.370 00:22:15.999 Patrick Trainer: the S. 3 bucket for all of these like email Csvs to come in instead of like the individual like slack Channel

331 00:22:16.130 00:22:18.530 Patrick Trainer: native emails. Does that make sense.

332 00:22:18.790 00:22:19.363 Uttam Kumaran: Yeah, so.

333 00:22:19.650 00:22:23.990 Nicolas Sucari: Why don’t we use? Why don’t we use the email that we have, for example, pull part.

334 00:22:23.990 00:22:24.830 Uttam Kumaran: Yeah. Stunt.

335 00:22:24.830 00:22:28.819 Nicolas Sucari: At Brain Forge, so that we can separate that for each client.

336 00:22:30.330 00:22:31.560 Patrick Trainer: Oh, yeah, that.

337 00:22:32.270 00:22:33.179 Uttam Kumaran: So let’s do that.

338 00:22:33.180 00:22:35.040 Patrick Trainer: Yeah, that works, too. Yeah.

339 00:22:38.410 00:22:42.770 Nicolas Sucari: I don’t know if we haven’t created the poll cards at Brain Force. AI.

340 00:22:42.880 00:22:47.820 Nicolas Sucari: I think we have a group, but I don’t know if a group is what we need, right?

341 00:22:48.820 00:22:51.460 Uttam Kumaran: You need a separate account with a separate inbox, right?

342 00:22:52.330 00:22:52.910 Nicolas Sucari: Yeah.

343 00:22:55.990 00:23:06.789 Patrick Trainer: That I think would be most clean, because, like, if pool parts is sending other stuff like, I mean, it doesn’t really matter, because it’s like it’s searching through like, I’ve tested it with

344 00:23:07.230 00:23:11.090 Patrick Trainer: my email. And it’s like, there’s more than

345 00:23:11.940 00:23:13.940 Patrick Trainer: it’s just sitting in there.

346 00:23:13.940 00:23:19.440 Uttam Kumaran: Setup. The inbox is set up right now as a group, which means it’s like a fake. It’s like a virtual inbox.

347 00:23:19.740 00:23:20.310 Uttam Kumaran: It just like.

348 00:23:20.310 00:23:21.080 Patrick Trainer: So right.

349 00:23:21.080 00:23:23.680 Uttam Kumaran: Us, and then we can all reply as.

350 00:23:24.130 00:23:24.950 Patrick Trainer: Okay.

351 00:23:24.950 00:23:25.570 Uttam Kumaran: Not really.

352 00:23:25.570 00:23:26.220 Patrick Trainer: Like a separate.

353 00:23:26.220 00:23:28.572 Uttam Kumaran: Box. I can create.

354 00:23:29.210 00:23:30.938 Uttam Kumaran: I can create like,

355 00:23:32.600 00:23:38.340 Uttam Kumaran: full parts that go Etl, or yeah, risk or incoming, or something like that.

356 00:23:38.590 00:23:39.330 Patrick Trainer: Yeah.

357 00:23:39.850 00:23:42.080 Uttam Kumaran: As an account give you that.

358 00:23:42.850 00:23:44.770 Patrick Trainer: That sounds that sounds

359 00:23:45.340 00:23:47.859 Patrick Trainer: good. Yeah, that sounds good to me.

360 00:23:48.980 00:23:53.930 Uttam Kumaran: Is that an account like a service like? Is there a service account way of doing this?

361 00:23:58.030 00:23:58.919 Uttam Kumaran: I don’t know.

362 00:23:59.480 00:24:04.920 Patrick Trainer: So this the service account is, what’s going to be

363 00:24:06.970 00:24:10.534 Patrick Trainer: getting or like going to get the email

364 00:24:11.692 00:24:14.830 Patrick Trainer: in which that’s like that Gmad

365 00:24:15.660 00:24:19.909 Patrick Trainer: script or library. It’s basically what it does is

366 00:24:20.090 00:24:20.890 Patrick Trainer: it

367 00:24:21.630 00:24:25.030 Patrick Trainer: fires off, and then it like, impersonates

368 00:24:25.510 00:24:27.660 Patrick Trainer: that user. Or that account

369 00:24:27.990 00:24:29.739 Patrick Trainer: opens their mailbox.

370 00:24:29.810 00:24:32.510 Patrick Trainer: searches for it grabs it, loads it.

371 00:24:33.437 00:24:36.929 Patrick Trainer: And then like, that’s that’s the deal. So I

372 00:24:39.140 00:24:42.189 Patrick Trainer: I believe it’s yeah. It’s gonna have to be like another

373 00:24:45.340 00:24:48.370 Patrick Trainer: dedicate like, I guess, user in brain forge.

374 00:24:48.770 00:24:52.640 Uttam Kumaran: Okay, okay, I’ll just create. I’ll so I’ll create another one. And then.

375 00:24:54.700 00:25:01.544 Uttam Kumaran: yeah, the the the Csv’s are ready to go. So what I’ll do is I’ll create the new one. And then

376 00:25:02.310 00:25:04.523 Uttam Kumaran: let’s go ahead. And I’m just gonna

377 00:25:05.310 00:25:09.189 Uttam Kumaran: basically start to forward those over to that account.

378 00:25:09.610 00:25:10.240 Patrick Trainer: Cool.

379 00:25:19.870 00:25:22.950 Nicolas Sucari: Once that email is set, we need to like

380 00:25:23.592 00:25:28.129 Nicolas Sucari: add that email to the reports that we are getting from

381 00:25:28.200 00:25:33.639 Nicolas Sucari: other stuff. Right? We need to ask Kim to do that, or anyone else do that, or we can do it.

382 00:25:35.056 00:25:38.230 Uttam Kumaran: It’s just so it’s just post pilot.

383 00:25:38.350 00:25:44.320 Uttam Kumaran: Basically, anytime, we’re like, someone’s like, we want to send you a Csv with this data. It’ll get sent to there.

384 00:25:45.005 00:25:47.480 Uttam Kumaran: So we can add that to notion. And then

385 00:25:48.770 00:25:50.859 Uttam Kumaran: I think I think basically.

386 00:25:51.020 00:25:57.960 Uttam Kumaran: what I’m going to do is forward all of them over, and then slowly, like what we can do is basically have a backlog ticket for

387 00:25:59.560 00:26:02.759 Uttam Kumaran: like having them move it directly to send to that one.

388 00:26:02.920 00:26:08.450 Uttam Kumaran: It doesn’t really matter if it’s coming to me and I’m auto forwarding, then it’s the same thing, because I’ll just forward with attachment.

389 00:26:08.800 00:26:09.560 Patrick Trainer: Right.

390 00:26:13.850 00:26:14.370 Uttam Kumaran: Okay.

391 00:26:18.240 00:26:25.460 Uttam Kumaran: okay, cool. The nice thing about the test is that we’re also gonna have, like, basically, Brian is working on

392 00:26:25.520 00:26:30.559 Uttam Kumaran: kind of like a document about how we do testing and like everything around Sla’s and things like that. So

393 00:26:31.280 00:26:32.729 Uttam Kumaran: it should be good timing.

394 00:26:32.980 00:26:38.579 Uttam Kumaran: So that we’re not like doing this. Basically, like, after that’s already written, he’ll have a good understanding where everything is.

395 00:26:38.810 00:26:42.200 Uttam Kumaran: But yeah, those are the big 2 things is like, if we can get

396 00:26:42.780 00:26:45.660 Uttam Kumaran: if we can get that done, and if we can get the

397 00:26:46.871 00:26:51.760 Uttam Kumaran: test done, then it’s a it should be a good week, and we’ll continue to move on to.

398 00:26:51.840 00:26:53.802 Uttam Kumaran: We’ll continue to move on to

399 00:26:54.640 00:26:57.520 Uttam Kumaran: finish up some of the security stuff, and there is

400 00:26:57.590 00:27:06.099 Uttam Kumaran: there. There is a bunch of backlog tickets, basically. To continue on to ideally like winter would be good to do next.

401 00:27:06.840 00:27:09.999 Patrick Trainer: Yeah, I agree. Get SQL, Fluff fired up.

402 00:27:10.430 00:27:10.790 Uttam Kumaran: Yeah.

403 00:27:12.850 00:27:14.530 Patrick Trainer: That’s really the only like

404 00:27:15.430 00:27:17.480 Patrick Trainer: SQL. Linker, that.

405 00:27:18.060 00:27:19.930 Patrick Trainer: apart from like

406 00:27:21.470 00:27:25.370 Patrick Trainer: the Jeff brains ide like doing it locally. The.

407 00:27:25.770 00:27:31.599 Patrick Trainer: It’s the only like sequel, formatter, slash linter that seems to be worth anything.

408 00:27:32.540 00:27:33.330 Uttam Kumaran: Okay.

409 00:27:36.550 00:27:38.880 Uttam Kumaran: okay, cool. So maybe I’ll leave

410 00:27:39.010 00:27:44.838 Uttam Kumaran: like the rest of this time. Maybe Pat and aar. If you guys want to chat about that testing ticket,

411 00:27:45.860 00:27:51.320 Uttam Kumaran: and then we just update that ticket or or slack for any updates on there.

412 00:27:51.320 00:27:52.599 Patrick Trainer: Yeah, yeah, sounds good.

413 00:27:52.900 00:27:53.820 Patrick Trainer: Move it?

414 00:27:55.540 00:27:59.704 Uttam Kumaran: Okay. Alright. I’ll chat with you guys on slack. Then I’ll leave this and leave this meeting.

415 00:27:59.950 00:28:00.660 Patrick Trainer: See it.

416 00:28:00.660 00:28:02.439 Nicolas Sucari: Thanks, guys. Bye, bye.