Meeting Title: DE-Planning-Session Date: 2024-07-01 Meeting participants: Nicolas Sucari, Uttam Kumaran, Atharv Gudi, Patricktrainer


WEBVTT

1 00:05:44.890 00:05:45.949 patricktrainer: Hey! Arthur!

2 00:05:46.960 00:05:48.710 Atharv Gudi: Hey, Patrick, how’s it going.

3 00:05:48.950 00:05:50.270 patricktrainer: Going good! How are you.

4 00:05:51.310 00:05:51.770 Atharv Gudi: Good.

5 00:05:53.470 00:05:53.970 patricktrainer: That.

6 00:05:55.570 00:05:59.399 Atharv Gudi: Will Nico, and put them also be joining us.

7 00:05:59.700 00:06:02.709 patricktrainer: Yes, yeah, they should be here in a few minutes.

8 00:06:04.010 00:06:04.940 patricktrainer: Top of them.

9 00:06:05.760 00:06:06.340 Uttam Kumaran: A

10 00:06:08.070 00:06:09.479 Uttam Kumaran: hey, Thor! How’s it going.

11 00:06:09.700 00:06:11.490 Atharv Gudi: Hi! How’s it going? I’m good.

12 00:06:17.307 00:06:22.379 Atharv Gudi: Put them. Did we have an another meeting earlier today? I’m not sure if I had.

13 00:06:23.270 00:06:26.843 Uttam Kumaran: Yeah, we did have a meeting this morning.

14 00:06:27.370 00:06:35.703 Uttam Kumaran: but I’m gonna update. I’ll send a note to the intern channel. Basically, we just reviewed with Priya and

15 00:06:36.630 00:06:43.839 Uttam Kumaran: and actually kind of like their progress and everything. Tomorrow we have a meeting. I’m gonna walk through like how to push pull requests.

16 00:06:44.325 00:06:49.010 Uttam Kumaran: Basically how to create a pull request, and then kind of set code up to review

17 00:06:50.410 00:07:00.419 Uttam Kumaran: But today we can. I think in this meeting, we can kind of chat about progress. And this meeting is really just we’re just covering everything that’s kind of going on in like data engineering world.

18 00:07:00.810 00:07:03.659 Uttam Kumaran: So it should be good overview of everything.

19 00:07:03.660 00:07:10.030 Atharv Gudi: Alright do you guys just mind waiting one moment. I just need to kind of book in a pen. I just like to write stuff down.

20 00:07:10.470 00:07:11.770 Uttam Kumaran: Yeah, yeah, no problem.

21 00:07:13.610 00:07:15.974 Uttam Kumaran: I guess. Yeah, we could probably go through

22 00:07:16.590 00:07:22.090 Uttam Kumaran: the Github project view I made. That’s just like de.

23 00:07:22.480 00:07:23.319 Uttam Kumaran: and that has like.

24 00:07:23.745 00:07:24.170 patricktrainer: Mingling

25 00:07:24.540 00:07:25.540 patricktrainer: nice.

26 00:07:26.130 00:07:26.970 patricktrainer: Yeah. I just saw that.

27 00:07:26.970 00:07:27.969 Nicolas Sucari: Yes, of course.

28 00:07:29.050 00:07:33.010 Nicolas Sucari: Yeah. And you I I was seeing this, too, right? A couple of minutes ago.

29 00:07:33.120 00:07:33.820 Nicolas Sucari: Nice.

30 00:07:33.820 00:07:34.690 Uttam Kumaran: Cool.

31 00:07:36.410 00:07:37.259 Nicolas Sucari: I did share.

32 00:07:47.685 00:07:48.350 patricktrainer: Hush!

33 00:07:48.350 00:07:50.020 Atharv Gudi: Yeah. Back.

34 00:07:57.630 00:07:58.230 Nicolas Sucari: Okay?

35 00:07:59.228 00:08:01.219 Nicolas Sucari: Sorry. I just like

36 00:08:01.860 00:08:03.259 Nicolas Sucari: I came here

37 00:08:03.340 00:08:09.460 Nicolas Sucari: a minute later and you already started. Do you wanna start reviewing this utam, this board.

38 00:08:09.460 00:08:15.320 Uttam Kumaran: Yeah, if you want, if we wanna just go through everything and like ready and progress like stuff that’s in play right now.

39 00:08:15.792 00:08:18.420 Uttam Kumaran: Maybe we can start with this stuff that’s like

40 00:08:18.660 00:08:20.180 Uttam Kumaran: in progress.

41 00:08:22.640 00:08:23.090 Nicolas Sucari: Yeah.

42 00:08:23.090 00:08:25.010 Uttam Kumaran: And we can. Yeah, that’d be great.

43 00:08:26.447 00:08:35.390 Nicolas Sucari: Patrick, do you wanna start? I know you were working last week on integrating all those files that we’re receiving through emails and slack.

44 00:08:36.395 00:08:42.429 patricktrainer: Right? Right? Right? Yeah. So I created a package that I

45 00:08:42.580 00:08:51.510 patricktrainer: what should do on yes or Friday. That’s all up and working ready to go. I’m proud to

46 00:08:52.118 00:08:55.580 patricktrainer: like I’ve been doing that all locally, and so I’m about to

47 00:08:56.140 00:09:00.960 patricktrainer: beam it up into like, create a new repo and brain forge and

48 00:09:00.970 00:09:02.569 patricktrainer: put it all up in there

49 00:09:03.277 00:09:05.430 patricktrainer: so that should be ready to go

50 00:09:05.906 00:09:10.789 patricktrainer: and then the second part of that is just going to be

51 00:09:11.230 00:09:13.070 patricktrainer: setting up like the

52 00:09:13.500 00:09:15.110 patricktrainer: stuff for the email.

53 00:09:16.470 00:09:17.830 patricktrainer: Like the

54 00:09:18.040 00:09:20.780 patricktrainer: email credentials. And like those specifics

55 00:09:21.330 00:09:24.960 patricktrainer: but like all the all, the functionality seems to be there

56 00:09:25.460 00:09:26.750 patricktrainer: working out

57 00:09:29.590 00:09:31.080 patricktrainer: And then.

58 00:09:32.150 00:09:34.700 patricktrainer: yeah, so that’s that’s update on that.

59 00:09:34.930 00:09:39.010 patricktrainer: The keepers that just needs to

60 00:09:39.430 00:09:41.670 patricktrainer: be done by everyone.

61 00:09:42.280 00:09:44.790 patricktrainer: that’s just kind of been on like back burner.

62 00:09:48.720 00:09:51.380 Uttam Kumaran: Should we? For key pair stuff

63 00:09:51.980 00:09:55.899 Uttam Kumaran: like pat, what do you think is the best way to do this like?

64 00:09:56.430 00:09:57.710 Uttam Kumaran: I think we just.

65 00:09:57.710 00:09:59.970 patricktrainer: Get everybody to to

66 00:10:00.961 00:10:04.219 patricktrainer: generate their key pairs on their own

67 00:10:04.410 00:10:05.270 patricktrainer: like on their own.

68 00:10:05.270 00:10:07.530 Nicolas Sucari: I think I already I already.

69 00:10:07.530 00:10:09.689 patricktrainer: Okay, you already. Yeah. You already did this

70 00:10:10.896 00:10:13.720 patricktrainer: and so that

71 00:10:14.060 00:10:19.049 patricktrainer: Nika was able to do it without any difficulty. I think we got it

72 00:10:19.640 00:10:20.650 patricktrainer: ready to go for.

73 00:10:20.650 00:10:21.760 Nicolas Sucari: And the reason that

74 00:10:22.810 00:10:25.020 Nicolas Sucari: there is a documentation here.

75 00:10:25.624 00:10:37.660 Nicolas Sucari: Probably. But if you want to check, if this is okay this is what it, what we had on the on the repo to the document I copied from there and paste it here.

76 00:10:38.530 00:10:39.409 Nicolas Sucari: And yeah, yeah.

77 00:10:39.410 00:10:41.779 patricktrainer: Yeah, this looks good and then.

78 00:10:41.780 00:10:43.660 Uttam Kumaran: So can like. So for this dog, like.

79 00:10:43.660 00:10:46.650 patricktrainer: All he just needs to do is like step one and step 2,

80 00:10:46.750 00:10:47.480 patricktrainer: and then.

81 00:10:47.480 00:10:47.820 Nicolas Sucari: Yeah.

82 00:10:47.820 00:10:52.989 patricktrainer: They need to send me step 3, and then I’ll alter their user. And then that’s it.

83 00:10:54.770 00:10:55.570 patricktrainer: yeah.

84 00:10:55.730 00:11:04.839 Uttam Kumaran: So, yeah, I guess couple of things. One is like we. So we have docs here and we have docs in the repo, like what I thought we were just gonna be referencing the docs in the repo.

85 00:11:07.180 00:11:07.730 Nicolas Sucari: Yeah, we can.

86 00:11:07.730 00:11:08.899 patricktrainer: Yeah, I think I read it.

87 00:11:08.900 00:11:09.550 Nicolas Sucari: What’s it? Here?

88 00:11:09.550 00:11:11.019 patricktrainer: In the rebabout. But yeah.

89 00:11:11.840 00:11:23.760 Nicolas Sucari: I just copied from there, so that we have all of the docs here. But I know that yeah. Patrick was working on this doc on the repo. So that’s why I just copy paste it here. But yeah, we can reference it now, if you want.

90 00:11:24.540 00:11:27.689 Uttam Kumaran: Yeah, you can actually, Nika, what you can do is you can just

91 00:11:27.770 00:11:31.900 Uttam Kumaran: literally copy what we have here, create a Pr for.

92 00:11:32.950 00:11:33.780 Nicolas Sucari: Oh, but that’s.

93 00:11:33.780 00:11:34.140 Uttam Kumaran: Is that a.

94 00:11:34.140 00:11:35.210 Nicolas Sucari: I mean if I.

95 00:11:35.720 00:11:38.490 patricktrainer: Yeah, that’s already that’s already done in the repat.

96 00:11:38.490 00:11:38.930 Nicolas Sucari: Yeah.

97 00:11:38.930 00:11:40.480 Uttam Kumaran: Oh, okay. Okay. Cool.

98 00:11:40.940 00:11:41.570 Nicolas Sucari: It seems like.

99 00:11:41.570 00:11:44.460 Uttam Kumaran: I think if you just, I think if you just reference the file.

100 00:11:45.240 00:11:46.360 Uttam Kumaran: then

101 00:11:47.310 00:11:48.940 Uttam Kumaran: it’ll all like.

102 00:11:49.050 00:11:49.745 Uttam Kumaran: basically.

103 00:11:50.820 00:11:52.559 Uttam Kumaran: it’ll just show the preview.

104 00:11:55.140 00:11:56.190 Nicolas Sucari: Think it’s here?

105 00:11:56.600 00:11:58.219 patricktrainer: Right like in notion.

106 00:11:59.400 00:12:00.790 patricktrainer: Yeah, here.

107 00:12:02.030 00:12:06.280 Nicolas Sucari: Do you want me like to? Where do you want it? To live on the repo, or in notion of them?

108 00:12:06.500 00:12:07.540 Uttam Kumaran: On the repo.

109 00:12:08.460 00:12:10.779 Nicolas Sucari: Okay. So I should copy

110 00:12:14.060 00:12:15.730 Nicolas Sucari: profile. No.

111 00:12:15.730 00:12:18.099 patricktrainer: No, I I think you can copy like the link.

112 00:12:18.100 00:12:19.680 Uttam Kumaran: If you copy the URL

113 00:12:20.250 00:12:22.189 Uttam Kumaran: for the page.

114 00:12:31.110 00:12:35.279 Uttam Kumaran: And then, yeah, as long as it’s the same thing, then we don’t need to worry about what’s in.

115 00:12:36.140 00:12:37.230 Uttam Kumaran: It’ll yeah.

116 00:12:37.230 00:12:39.839 patricktrainer: Yeah. And then it just dumps it there. Yeah, nice.

117 00:12:39.840 00:12:40.500 Nicolas Sucari: Cat.

118 00:12:41.520 00:12:42.130 Nicolas Sucari: and this.

119 00:12:42.130 00:12:45.129 patricktrainer: And that should keep that. Should that should keep it all up to date.

120 00:12:45.870 00:12:50.539 Uttam Kumaran: Yeah, cause. So cause ideally, it’s like, when people have dock up states

121 00:12:50.690 00:12:52.690 Uttam Kumaran: on the engineering side.

122 00:12:52.920 00:12:54.770 Uttam Kumaran: then it could just go.

123 00:12:55.250 00:12:57.209 Uttam Kumaran: We will just make a Pr and update it.

124 00:12:58.540 00:13:00.060 Nicolas Sucari: Perfect. Yeah. 30 days

125 00:13:00.570 00:13:01.260 Nicolas Sucari: was fine.

126 00:13:01.260 00:13:02.220 Uttam Kumaran: Okay, cool.

127 00:13:03.443 00:13:06.589 Uttam Kumaran: Yeah. So what we’ll do is, I think.

128 00:13:08.210 00:13:10.400 Uttam Kumaran: pat this week. I’m just gonna set

129 00:13:10.500 00:13:15.629 Uttam Kumaran: like some sort of reminder just to get everybody to do this in slack.

130 00:13:16.150 00:13:19.089 Uttam Kumaran: So, Nico, if you want to update this on the ticket.

131 00:13:19.310 00:13:23.829 Uttam Kumaran: can we just like send this to everybody and internal engineering to basically handle that.

132 00:13:23.830 00:13:24.550 Nicolas Sucari: Yeah.

133 00:13:24.550 00:13:25.550 Uttam Kumaran: On their own

134 00:13:26.170 00:13:26.600 Uttam Kumaran: and.

135 00:13:26.600 00:13:28.149 Nicolas Sucari: I can do it. Okay.

136 00:13:28.450 00:13:30.290 Uttam Kumaran: On Dealey on Friday.

137 00:13:30.560 00:13:33.639 Uttam Kumaran: If there’s any issues, people can

138 00:13:34.220 00:13:35.999 Uttam Kumaran: reach out directly to you. Pat

139 00:13:36.890 00:13:38.029 patricktrainer: Yeah, that sounds good.

140 00:13:38.530 00:13:43.140 Uttam Kumaran: So this is perfect. Yeah, I think this. And just tell people like to do this by the end of the week.

141 00:13:44.420 00:13:49.129 Nicolas Sucari: So they need to follow. Step one and 2, and then send the credentials to you. Right.

142 00:13:49.130 00:13:54.389 patricktrainer: Yeah, step one step 2, and then they send the public key to me.

143 00:13:55.050 00:13:56.669 patricktrainer: And I will

144 00:13:57.440 00:13:59.229 patricktrainer: get that loaded to Smith Lake.

145 00:14:00.300 00:14:02.120 Nicolas Sucari: Perfect. Yeah, sure.

146 00:14:04.830 00:14:10.019 Uttam Kumaran: And then any pat any like questions people have in that process.

147 00:14:10.617 00:14:14.739 Uttam Kumaran: or like, if if they have like, you’re seeing the same question over and over again.

148 00:14:14.990 00:14:16.430 Uttam Kumaran: Just update the Doc.

149 00:14:17.590 00:14:18.439 patricktrainer: Yeah, yeah, yeah.

150 00:14:18.440 00:14:23.380 Uttam Kumaran: Cause we’ll cause we’ll probably do this for for any new Snowflake accounts, anyway. So.

151 00:14:24.520 00:14:25.230 patricktrainer: Right.

152 00:14:25.820 00:14:26.480 Uttam Kumaran: Topo.

153 00:14:33.150 00:14:33.840 Nicolas Sucari: Perfect.

154 00:14:33.960 00:14:34.720 Nicolas Sucari: Yep.

155 00:14:35.050 00:14:41.699 Nicolas Sucari: okay. So we can close. We can add like a due date for Friday. This one. It’s fine.

156 00:14:41.700 00:14:42.984 Uttam Kumaran: Let’s do that.

157 00:14:44.750 00:14:50.851 Nicolas Sucari: Okay? And on the other side, the the what you’re we’ve been working around the actions to

158 00:14:51.350 00:14:56.159 Nicolas Sucari: load the files. Is something we can nail this week. What do you think.

159 00:14:56.880 00:15:01.554 patricktrainer: Yeah, I think that should be. We should be able to finish that this week, for sure.

160 00:15:03.040 00:15:03.395 Nicolas Sucari: Okay.

161 00:15:03.750 00:15:06.509 patricktrainer: I think. Yeah, there’s probably a couple more steps.

162 00:15:06.760 00:15:10.555 patricktrainer: but I’ll add those tickets in just to to track that.

163 00:15:11.220 00:15:19.759 Nicolas Sucari: Yeah. And we just had a meeting with Ryan and Brian before. And I think Ryan mentioned that there were some other files that were

164 00:15:19.850 00:15:24.599 Nicolas Sucari: being sent directly to Utam’s email. So I don’t know Utam. If you were able to check if you are.

165 00:15:24.600 00:15:28.660 Uttam Kumaran: Yeah, I’m gonna I’m gonna in in this ticket. I’m gonna basically

166 00:15:28.670 00:15:30.660 Uttam Kumaran: I already did put like

167 00:15:30.680 00:15:32.920 Uttam Kumaran: the current emails, which was like.

168 00:15:33.100 00:15:36.949 Uttam Kumaran: Case one case 2. I’m just gonna put all of them.

169 00:15:37.020 00:15:39.060 Uttam Kumaran: And then Pat will create like a

170 00:15:39.260 00:15:40.550 Uttam Kumaran: migration.

171 00:15:41.240 00:15:47.609 Uttam Kumaran: I guess we’ll create like a migration step where I’m just gonna list every all the current integrations.

172 00:15:47.670 00:15:50.229 Uttam Kumaran: And then you can track as like a checklist.

173 00:15:51.095 00:15:51.500 patricktrainer: Okay.

174 00:15:52.630 00:16:04.680 Uttam Kumaran: So I’m gonna I’m gonna I’ll do that today. And then I’ll basically say, if anyone else is receiving anything, please add it here, and then we can just run through that again ideally, if, like, once you have your service set up

175 00:16:04.740 00:16:06.250 Uttam Kumaran: and you have docs on like

176 00:16:06.400 00:16:07.570 Uttam Kumaran: how to do this.

177 00:16:07.580 00:16:11.100 Uttam Kumaran: anybody can go actually do the integration. So maybe it’s like

178 00:16:11.190 00:16:13.499 Uttam Kumaran: you test one or test 2, and then.

179 00:16:13.790 00:16:14.510 patricktrainer: Right?

180 00:16:16.420 00:16:17.820 patricktrainer: Yeah, that sounds good.

181 00:16:18.700 00:16:21.099 Uttam Kumaran: Do book. I’ll just. I’m gonna add that here

182 00:16:34.820 00:16:36.660 Uttam Kumaran: any additional.

183 00:16:38.480 00:16:39.200 Uttam Kumaran: Okay, great.

184 00:16:39.200 00:16:39.820 Nicolas Sucari: Okay.

185 00:16:46.400 00:16:52.520 Uttam Kumaran: Cool, I think, maybe one more thing on security. So there’s this ticket. Verify 2 fa on Snowflake.

186 00:16:53.010 00:16:54.869 Uttam Kumaran: Is that Don Pat, or like.

187 00:16:54.870 00:16:58.470 patricktrainer: Right no. So

188 00:16:59.010 00:17:06.220 patricktrainer: I actually, when I was in Snowflake earlier, I got a notification saying to turn on 2 factor for everyone.

189 00:17:06.589 00:17:13.020 patricktrainer: And so like the key pair off, you think that’s like that’s going to take

190 00:17:13.609 00:17:17.310 patricktrainer: place of like when we’re using Dbt.

191 00:17:17.720 00:17:28.520 patricktrainer: all, all of our service accounts, basically like storing any sort of like credential, like in plain text. But when logging onto the Ui

192 00:17:28.973 00:17:32.120 patricktrainer: like, you can’t use like a local key

193 00:17:32.928 00:17:35.740 patricktrainer: like through the browser. So we

194 00:17:35.930 00:17:42.357 patricktrainer: like. What you’ll either have to do is set up like Sso, like Octa, or something which I mean, we’re not doing

195 00:17:42.820 00:17:49.840 patricktrainer: or 2 factor. So you use your username password, and then and then you use, like the authenticator app

196 00:17:50.276 00:17:53.979 patricktrainer: to do that. And then like, then everything should be good.

197 00:17:54.040 00:17:59.509 patricktrainer: But as far as, like all of our service accounts, all of our accounts to

198 00:18:00.216 00:18:10.380 patricktrainer: like when we’re working locally or like, if anybody uses the snow cli, or something like that, all of that should use the key pair.

199 00:18:11.880 00:18:15.330 Uttam Kumaran: Okay, is this just a setting in Snowflake? I would just go ahead and.

200 00:18:15.330 00:18:15.740 patricktrainer: Yes.

201 00:18:15.740 00:18:16.649 Uttam Kumaran: So rip it.

202 00:18:17.040 00:18:21.170 patricktrainer: Yeah, it’s just a setting so yeah, if you wanna do that.

203 00:18:21.220 00:18:27.560 patricktrainer: it should really only take like 5 min. And then I think the next time someone logs in

204 00:18:28.024 00:18:30.249 patricktrainer: it’s gonna prompt them to

205 00:18:30.990 00:18:32.660 patricktrainer: set up 2 factor.

206 00:18:35.010 00:18:37.178 Uttam Kumaran: Okay, cool. Alright. I can take that.

207 00:18:37.660 00:18:40.273 patricktrainer: Yeah. And then we can also set up

208 00:18:41.410 00:18:45.340 patricktrainer: like password requirement rules like

209 00:18:46.590 00:18:47.670 patricktrainer: must be

210 00:18:48.160 00:18:51.070 patricktrainer: 15 characters long, like that kind of stuff.

211 00:18:51.927 00:18:53.690 patricktrainer: Just to ensure

212 00:18:54.140 00:18:55.929 patricktrainer: safe passwords, but

213 00:18:56.160 00:18:58.149 patricktrainer: hopefully everybody knows to

214 00:18:59.290 00:19:01.670 patricktrainer: use good passwords, anyway. But

215 00:19:02.630 00:19:03.649 patricktrainer: you never know.

216 00:19:05.130 00:19:10.070 Uttam Kumaran: Yeah, I think the last. So I think once we do 2 fa key pair, I want to.

217 00:19:10.330 00:19:16.100 Uttam Kumaran: Basically, maybe you and Darv can work on like the security policy Doc.

218 00:19:16.440 00:19:20.170 Uttam Kumaran: ideally using, like Gpd, basically to like write most of that.

219 00:19:20.370 00:19:25.349 Uttam Kumaran: But kind of just want, like a doc on like how we think about security across

220 00:19:25.450 00:19:29.770 Uttam Kumaran: each of our access points, the data as well as like a list of like

221 00:19:29.880 00:19:33.259 Uttam Kumaran: further things we could do. And that just becomes running

222 00:19:33.520 00:19:39.689 Uttam Kumaran: security, Doc, where, if someone’s like, Hey, how are you doing? Security? We can reference that. Or if we have improvements to make.

223 00:19:40.250 00:19:41.899 Uttam Kumaran: we can keep all that

224 00:19:41.920 00:19:50.130 Uttam Kumaran: there and then, as we have improvements or discussion on it, we’ll centralize the discussion and notion the documentation and Github, and then

225 00:19:50.719 00:19:54.309 Uttam Kumaran: this actual tasks will bring him into into project.

226 00:19:54.310 00:20:02.459 patricktrainer: One thing, one thing that we do need to do and this is true for cool parts especially is the

227 00:20:03.970 00:20:06.060 patricktrainer: like the role hierarchy

228 00:20:06.845 00:20:13.154 patricktrainer: like, right now, it’s kind of like all over the place, and everything kind of uses the the same roles

229 00:20:13.950 00:20:15.103 patricktrainer: So like

230 00:20:15.760 00:20:23.809 patricktrainer: ideally like it. It’s like Stella is set up, really. Well, I just like to get pull parts in that in that same

231 00:20:24.230 00:20:25.040 patricktrainer: form.

232 00:20:27.420 00:20:29.000 Uttam Kumaran: Okay, yeah,

233 00:20:30.190 00:20:32.910 Uttam Kumaran: yeah, so like, basically basically.

234 00:20:32.910 00:20:39.430 patricktrainer: Yeah, basically. So that we have, like a where, like individual warehouses for loads.

235 00:20:39.430 00:20:39.790 Uttam Kumaran: Yes.

236 00:20:39.790 00:20:41.670 patricktrainer: Transforming, reporting.

237 00:20:41.860 00:20:42.810 patricktrainer: loading, transform.

238 00:20:42.810 00:20:46.670 Uttam Kumaran: What do you wanna what do you call this like? What do you call this effort around that?

239 00:20:46.870 00:20:49.959 Uttam Kumaran: Because we also have this ticket? That’s like, confirm

240 00:20:50.600 00:20:52.250 Uttam Kumaran: schema structure.

241 00:20:53.090 00:20:56.139 Uttam Kumaran: And I also actually think this would be a good thing, for, like

242 00:20:56.430 00:20:59.110 Uttam Kumaran: author can probably take on some of this.

243 00:20:59.110 00:20:59.560 patricktrainer: Right, yeah.

244 00:20:59.560 00:21:00.400 Uttam Kumaran: Also

245 00:21:00.710 00:21:04.160 Uttam Kumaran: we do. We do have a Dvt project

246 00:21:04.310 00:21:07.930 Uttam Kumaran: sort of like, how we do Dbt projects. But

247 00:21:08.210 00:21:14.780 Uttam Kumaran: I remember you implemented the roles for Stella, we would just do the same thing. The issue we’re gonna have is

248 00:21:15.510 00:21:17.360 Uttam Kumaran: we probably want to do this

249 00:21:17.580 00:21:22.910 Uttam Kumaran: on like after hours. And there’s gonna be stuff that breaks so.

250 00:21:22.910 00:21:23.330 patricktrainer: Right.

251 00:21:23.330 00:21:27.209 Uttam Kumaran: I just wanna like that’s, you know, that’s the thing. I’ve just the only difference.

252 00:21:27.670 00:21:28.680 patricktrainer: Yeah,

253 00:21:30.600 00:21:35.400 patricktrainer: I think there is a ticket in there somewhere. It’s like.

254 00:21:39.760 00:21:45.254 patricktrainer: I mean, really, like, the the paradigm is like our back role based authentication.

255 00:21:46.060 00:21:49.659 patricktrainer: and so we just need to

256 00:21:49.800 00:21:51.590 patricktrainer: tighten that.

257 00:21:51.980 00:21:53.299 patricktrainer: Yeah, just tighten that up.

258 00:21:53.300 00:21:56.535 Uttam Kumaran: Why don’t I create? Why don’t I create a ticket called

259 00:21:57.250 00:21:58.260 Uttam Kumaran: like

260 00:21:58.890 00:22:00.490 Uttam Kumaran: implement?

261 00:22:01.390 00:22:04.070 Uttam Kumaran: Well, I’ll do one thing I’m gonna say.

262 00:22:05.480 00:22:07.410 patricktrainer: Role, hierarchies or something.

263 00:22:08.480 00:22:09.420 Uttam Kumaran: Yeah.

264 00:22:12.330 00:22:15.840 Uttam Kumaran: So you want to do the R back and the warehouse in one shot.

265 00:22:17.580 00:22:24.290 patricktrainer: Yeah, I mean, we can break it into into multiple po parts. But like the the warehouse. And like all of that, it’s all.

266 00:22:24.570 00:22:25.880 Uttam Kumaran: Cause all tied basically.

267 00:22:25.880 00:22:26.430 patricktrainer: Yeah, it’s.

268 00:22:26.430 00:22:26.870 Uttam Kumaran: Yeah.

269 00:22:26.870 00:22:31.810 patricktrainer: Tied so essentially like the way it’s doing is like so that we can

270 00:22:31.980 00:22:47.539 patricktrainer: control lesser roles with higher roles. Not be. Make sure that we can’t be like locked out or not own those objects, or so that all, all of our parent roles own those objects. But then also with the warehouses that doubles into

271 00:22:47.952 00:22:56.149 patricktrainer: kind of like usage tracking so like, who’s using what? Who’s doing? What what has access to what and then.

272 00:22:56.150 00:22:56.490 Uttam Kumaran: Yeah.

273 00:22:56.490 00:23:05.070 patricktrainer: It also really helps in cost controls. So if we notice that like like 5 trans can have its own warehouse. So if we’re not mixing

274 00:23:05.390 00:23:07.709 patricktrainer: 5 trend compute with

275 00:23:08.687 00:23:17.379 patricktrainer: real compute, like, we’ll be able to bisect those 2 and know, like, okay, 5 tran uses this much real uses this much.

276 00:23:17.390 00:23:22.780 patricktrainer: Our transformations use this much like we’re just bucketing it into

277 00:23:22.960 00:23:24.040 patricktrainer: separate

278 00:23:24.430 00:23:25.210 patricktrainer: pieces.

279 00:23:27.240 00:23:31.700 Uttam Kumaran: Yeah. So there’s almost 2 things that I mean, I get. I’ve done this like

280 00:23:31.760 00:23:36.870 Uttam Kumaran: all the time. The thing I really want to get out of this exercise.

281 00:23:37.240 00:23:39.620 Uttam Kumaran: as everything is

282 00:23:39.680 00:23:45.730 Uttam Kumaran: a document, basically about naming conventions. So like how we’re gonna do warehouse enroll.

283 00:23:45.840 00:23:46.390 Uttam Kumaran: naming.

284 00:23:46.390 00:23:47.100 patricktrainer: I was, yeah.

285 00:23:47.100 00:23:48.000 Uttam Kumaran: Yeah,

286 00:23:49.590 00:23:54.139 Uttam Kumaran: so that’s 1. So it’s like warehouse. I’m just gonna create a ticket around this and then

287 00:23:54.290 00:23:56.607 Uttam Kumaran: warehouse naming

288 00:23:57.640 00:24:00.500 Uttam Kumaran: oddly, naming convention

289 00:24:04.780 00:24:07.690 Uttam Kumaran: roles, warehouses.

290 00:24:11.730 00:24:15.530 Uttam Kumaran: and then we’re gonna have like kind of conventions

291 00:24:15.580 00:24:17.230 Uttam Kumaran: or users

292 00:24:17.330 00:24:19.340 Uttam Kumaran: and service accounts

293 00:24:20.140 00:24:20.880 Uttam Kumaran: yup

294 00:24:21.220 00:24:25.960 Uttam Kumaran: with, as it relates to those as it relates to roles and warehouses.

295 00:24:26.400 00:24:27.150 Uttam Kumaran: Yep.

296 00:24:28.180 00:24:32.299 Uttam Kumaran: warehouses. I also want to include basically like

297 00:24:32.830 00:24:38.459 Uttam Kumaran: you actually have. Wait. You have the boilerplate code, right? Like grants.

298 00:24:38.460 00:24:40.599 patricktrainer: All from all from stellar. Yeah.

299 00:24:41.340 00:24:44.699 Uttam Kumaran: Yeah. So we’ll kind of create, maybe some sort of like.

300 00:24:45.400 00:24:47.310 Uttam Kumaran: we’ll basically like, Evolve.

301 00:24:47.320 00:24:48.670 Uttam Kumaran: the Stella

302 00:24:49.915 00:24:51.640 Uttam Kumaran: set up grant

303 00:24:51.950 00:24:53.370 Uttam Kumaran: exact rate.

304 00:24:53.390 00:24:54.683 Uttam Kumaran: a generic

305 00:24:55.770 00:24:58.350 Uttam Kumaran: set of set of grant scripts.

306 00:25:00.220 00:25:03.199 Uttam Kumaran: it doesn’t have to be like 5 train where it’s like.

307 00:25:03.500 00:25:08.290 Uttam Kumaran: put this thing here. And it basically runs everything. But again, it’s like, if if.

308 00:25:08.290 00:25:08.690 patricktrainer: Right.

309 00:25:08.690 00:25:11.630 Uttam Kumaran: Someone decently equipped was like cool.

310 00:25:12.260 00:25:14.929 Uttam Kumaran: We want to set up a new snowflake instance.

311 00:25:15.030 00:25:19.499 Uttam Kumaran: The main steps are one. Here’s how to set up roles warehouses.

312 00:25:19.590 00:25:25.019 Uttam Kumaran: Here’s how to add users which we already have. What’s the proper security which we’re gonna have

313 00:25:25.430 00:25:30.570 Uttam Kumaran: pairing those up. And then we already have a document for basically establishing Dbt.

314 00:25:30.770 00:25:33.300 Uttam Kumaran: and then we also now have documents for like

315 00:25:33.310 00:25:36.619 Uttam Kumaran: like evidence, unreal. So we’re like, almost.

316 00:25:37.200 00:25:40.499 Uttam Kumaran: and at elementary. So we’re almost through the full stack. Basically.

317 00:25:40.740 00:25:41.810 Uttam Kumaran: So

318 00:25:41.830 00:25:50.869 Uttam Kumaran: we have naming conventions for roles, warehouses, conventions for user service accounts of all this Stella set of grants to create a generic thing. And then for pool parts

319 00:25:50.890 00:25:53.509 Uttam Kumaran: in terms of the implementation.

320 00:25:53.530 00:25:57.688 Uttam Kumaran: we’ll have to update roles.

321 00:25:58.990 00:26:04.559 Uttam Kumaran: and yeah, we’ll probably have to update roles. There’s not. There’s only like one role. Right now.

322 00:26:04.900 00:26:05.960 Uttam Kumaran: I’m

323 00:26:06.130 00:26:12.660 Uttam Kumaran: the update roles, create update warehouse names. We’ll have to go into github

324 00:26:13.200 00:26:17.219 Uttam Kumaran: and update. Yeah, change all those secrets.

325 00:26:18.190 00:26:20.290 Uttam Kumaran: We’ll have to

326 00:26:21.310 00:26:22.230 Uttam Kumaran: clean up

327 00:26:22.470 00:26:25.860 Uttam Kumaran: one password because a lot of people are just using my creds.

328 00:26:26.110 00:26:27.320 Uttam Kumaran: So I’ll probably just

329 00:26:28.030 00:26:30.680 Uttam Kumaran: change my password so that it breaks all that.

330 00:26:30.680 00:26:32.210 Nicolas Sucari: Yeah, I don’t know what happened.

331 00:26:32.210 00:26:32.800 Uttam Kumaran: And

332 00:26:34.700 00:26:38.897 Uttam Kumaran: I was gonna say, changes my password. I’ll put my name in here

333 00:26:39.480 00:26:40.660 Uttam Kumaran: change.

334 00:26:41.380 00:26:42.469 Nicolas Sucari: Yeah, but everyone.

335 00:26:42.470 00:26:43.130 Uttam Kumaran: Yeah, we agree.

336 00:26:43.130 00:26:45.719 Nicolas Sucari: 80 accounts for everyone on Snowflake.

337 00:26:45.720 00:26:48.019 Uttam Kumaran: Yeah, it’s just it’s just like.

338 00:26:48.980 00:26:49.300 Nicolas Sucari: Yeah.

339 00:26:49.300 00:26:53.040 Uttam Kumaran: You can’t enforce these things unless you do so.

340 00:26:54.336 00:26:54.730 Uttam Kumaran: Base guy.

341 00:26:54.730 00:26:55.049 Nicolas Sucari: How many.

342 00:26:55.050 00:26:59.428 Uttam Kumaran: If I just change my password, it’s gonna break like everybody’s stuff. And then.

343 00:26:59.720 00:27:01.140 patricktrainer: So somebody’s. Yes.

344 00:27:01.140 00:27:01.470 Nicolas Sucari: Change.

345 00:27:01.470 00:27:02.910 patricktrainer: You can be like, Hey, my stuff didn’t.

346 00:27:03.730 00:27:07.939 Nicolas Sucari: Yeah, change it and remove it from one password, because, if not, everyone is gonna still.

347 00:27:07.940 00:27:09.799 Uttam Kumaran: Yeah, yeah, yeah.

348 00:27:10.620 00:27:16.119 Uttam Kumaran: So clean up one password, I’ll train my password. We want to also

349 00:27:16.430 00:27:17.930 Uttam Kumaran: update

350 00:27:18.160 00:27:19.310 Uttam Kumaran: service

351 00:27:19.890 00:27:22.330 Uttam Kumaran: accounts for real

352 00:27:22.490 00:27:23.810 Uttam Kumaran: evidence.

353 00:27:25.560 00:27:32.200 Uttam Kumaran: fraud, and like Dash, we’re still on there by then update roles. So what else is update roles.

354 00:27:32.210 00:27:33.580 Uttam Kumaran: warehouse names?

355 00:27:34.060 00:27:39.079 Uttam Kumaran: Oh, we have to update roles. And then we have to do role the role assignments, basically

356 00:27:41.990 00:27:43.660 Uttam Kumaran: role assignments.

357 00:27:43.750 00:27:45.640 Uttam Kumaran: update service accounts.

358 00:27:48.139 00:27:48.699 Uttam Kumaran: I’d

359 00:27:51.060 00:27:53.029 Uttam Kumaran: yeah, I think that should be it.

360 00:27:53.310 00:27:53.860 Uttam Kumaran: Right?

361 00:27:56.490 00:27:59.910 Uttam Kumaran: You’re gonna create a, you have a separate service account for elementary.

362 00:28:00.090 00:28:01.919 Uttam Kumaran: or is that just use the Dbt. One.

363 00:28:04.040 00:28:06.779 patricktrainer: believe Ryan, set that up so I’m not.

364 00:28:07.780 00:28:10.699 Uttam Kumaran: Okay, I think it just uses the Dbt stuff.

365 00:28:11.250 00:28:14.770 Uttam Kumaran: cause I think it just runs as part of the Dbt project. Like all the tests.

366 00:28:15.490 00:28:16.849 patricktrainer: Yeah, I think that’s true.

367 00:28:18.060 00:28:18.750 Uttam Kumaran: Okay.

368 00:28:22.870 00:28:27.519 Uttam Kumaran: great the. And then maybe the thing for us on the establishing side is like.

369 00:28:27.960 00:28:32.399 Uttam Kumaran: for example, right now, we just have, like, we almost have the

370 00:28:32.980 00:28:39.040 Uttam Kumaran: like data engineering. And like Etl related stuff, we have modeling related stuff. And we have bi related stuff

371 00:28:39.180 00:28:44.089 Uttam Kumaran: kind of like, if this may be part of the naming conventions. But like, for example, if we were, gonna add.

372 00:28:44.290 00:28:51.689 Uttam Kumaran: like an external person, or if we’re gonna we had another layer basically like how to do the naming and enroll setup.

373 00:28:51.690 00:28:53.439 patricktrainer: Right, right.

374 00:28:54.010 00:29:00.389 Uttam Kumaran: But I think otherwise. Like, for example, like, let’s say, your ser, your service needs to hit Snowflake, or a new service

375 00:29:00.870 00:29:03.419 Uttam Kumaran: like, where does that fall into?

376 00:29:06.750 00:29:07.983 Uttam Kumaran: you know?

377 00:29:09.350 00:29:13.989 Uttam Kumaran: and maybe this is more. This is less about user accounts, more about the service accounts.

378 00:29:14.200 00:29:17.870 Uttam Kumaran: The nice thing is, we don’t have like prod or staging environments.

379 00:29:17.940 00:29:18.950 Uttam Kumaran: for now.

380 00:29:19.570 00:29:20.559 patricktrainer: That’s not like.

381 00:29:20.840 00:29:22.810 Uttam Kumaran: It’s not like another layer there.

382 00:29:22.910 00:29:23.750 Uttam Kumaran: But

383 00:29:24.780 00:29:27.080 Uttam Kumaran: like, once this gets settled.

384 00:29:27.240 00:29:30.500 Uttam Kumaran: I’m actually thinking more about like, how do we create

385 00:29:31.290 00:29:32.830 Uttam Kumaran: dev environments?

386 00:29:33.340 00:29:35.770 Uttam Kumaran: We create staging, and then that way.

387 00:29:36.030 00:29:39.690 Uttam Kumaran: that way. We then have 3 layers, which is like you running stuff.

388 00:29:39.860 00:29:49.549 Uttam Kumaran: We have the Dbt job that executes the Pr. And staging, and then once it gets pushed to main main, gets executed in prod right? Because right now the Pr itself

389 00:29:49.580 00:29:52.330 Uttam Kumaran: is getting running on prod so. But that, I think.

390 00:29:52.480 00:29:55.589 Uttam Kumaran: is like, that’s the ultimate. That’s we’re getting there. So.

391 00:29:55.590 00:29:56.650 patricktrainer: Right, right.

392 00:29:59.500 00:30:06.330 Uttam Kumaran: So maybe we I think we should do like a meeting. Towards like the end of the week

393 00:30:06.530 00:30:10.289 Uttam Kumaran: we just set do 30 min, and we can maybe just all collaborate on the stock

394 00:30:10.350 00:30:11.410 Uttam Kumaran: like live

395 00:30:14.110 00:30:15.809 Uttam Kumaran: that way, it kind of takes

396 00:30:16.080 00:30:21.920 Uttam Kumaran: this out. Cause I I don’t. I think it’s gonna be a lot of back and forth. And I want us to have like a discussion and a

397 00:30:21.990 00:30:24.000 Uttam Kumaran: click, a video on it. And ideas.

398 00:30:24.460 00:30:26.060 Uttam Kumaran: Yep, probably the

399 00:30:26.380 00:30:30.850 Uttam Kumaran: probably the 4 of us is fine. We can aim for some time in the morning.

400 00:30:30.860 00:30:40.240 Uttam Kumaran: Athar. So and I guess, Athar, any questions. Sorry. This is like a lot of like new lingo and new stuff. You have any questions, or

401 00:30:40.540 00:30:42.806 Uttam Kumaran: if any of this makes sense or

402 00:30:43.900 00:30:49.510 Uttam Kumaran: I I just wanna make sure that you’re you kinda get. Why, we’re doing this in particular.

403 00:30:50.210 00:30:53.650 Atharv Gudi: Yeah, I think I was able to follow along. I think for the most part.

404 00:30:54.560 00:31:00.299 Atharv Gudi: I think I can. I’m able to follow along to what you’re you all are speaking about.

405 00:31:00.430 00:31:03.500 Atharv Gudi: I can see why you are doing what you’re doing as well.

406 00:31:04.040 00:31:08.100 Atharv Gudi: so I don’t have any questions just yet, but I feel like

407 00:31:08.230 00:31:13.379 Atharv Gudi: I still need to spend a little more time in the water, so that I know what’s happening.

408 00:31:13.690 00:31:14.579 Atharv Gudi: I guess that discount.

409 00:31:14.580 00:31:15.609 Uttam Kumaran: Yeah, this is.

410 00:31:15.610 00:31:16.350 Atharv Gudi: Yes.

411 00:31:16.750 00:31:18.870 Uttam Kumaran: Yeah, this sort of stuff is like.

412 00:31:19.240 00:31:25.329 Uttam Kumaran: I don’t know if this is pretty, it’s a lot deeper than just data models. And this is more about like

413 00:31:25.510 00:31:27.580 Uttam Kumaran: developer like devops

414 00:31:27.620 00:31:30.130 Uttam Kumaran: security. But again, like

415 00:31:30.680 00:31:35.820 Uttam Kumaran: I would say, it’s like I. I didn’t get a chance to do this stuff until like a little bit

416 00:31:35.890 00:31:38.680 Uttam Kumaran: into the job. And I had to do this stuff basically because

417 00:31:39.390 00:31:42.449 Uttam Kumaran: this just without, if you don’t handle this.

418 00:31:42.600 00:31:51.240 Uttam Kumaran: it causes a lot of issues. And instead of handling this, I just didn’t. And I basically created one role, one warehouse

419 00:31:51.250 00:31:54.250 Uttam Kumaran: because the alternative is like creating a ton

420 00:31:54.340 00:31:56.990 Uttam Kumaran: and then having to back that out. And I

421 00:31:57.060 00:32:04.319 Uttam Kumaran: basically a lot of what we’re doing now is like taking care of stuff I just didn’t really have. I just didn’t have the brain power to handle. But

422 00:32:04.460 00:32:06.119 Uttam Kumaran: I was like, Okay, one day, well.

423 00:32:06.330 00:32:15.840 Uttam Kumaran: have some more people on, and we’ll handle it. So a lot of this is like. And the nice thing is, I hope you’re recognizing that as we go through any of these processes

424 00:32:16.160 00:32:23.659 Uttam Kumaran: I’m always pushing for. Can we write stuff down, not only like the state of the world, and like what is our standard, but then also.

425 00:32:23.680 00:32:28.769 Uttam Kumaran: where are there improvements? Because this stuff is constantly gonna change the security

426 00:32:28.790 00:32:37.480 Uttam Kumaran: stuff. It’s gonna constantly evolve and get more secure, like 2 fa and keep hair fully in the last few years that they really like beefed up in snowflake.

427 00:32:37.780 00:32:44.719 Uttam Kumaran: And then also, you know, the standard way of doing this will work for the most part. But there’s always going to be edge cases.

428 00:32:45.335 00:32:49.469 Uttam Kumaran: So this all stuff that like, will we? Just, I just wanna document and have

429 00:32:49.480 00:32:51.929 Uttam Kumaran: idea about how we do these things. Basically. So.

430 00:32:52.550 00:32:59.099 Atharv Gudi: Okay, yeah, I had one more question. I think it was just about my ticket. Just to confirm.

431 00:32:59.100 00:32:59.900 Uttam Kumaran: Yeah.

432 00:33:00.529 00:33:08.440 Atharv Gudi: I my, I think it was moving test to the test directly. This is in with regard to the pool parts to go directly.

433 00:33:08.820 00:33:11.200 Uttam Kumaran: Yeah. So let’s yeah, let’s

434 00:33:11.460 00:33:14.200 Uttam Kumaran: talk about that right now.

435 00:33:16.060 00:33:18.270 Uttam Kumaran: it’s number 19

436 00:33:18.650 00:33:19.429 Uttam Kumaran: row 19.

437 00:33:19.430 00:33:20.559 Nicolas Sucari: Yeah. Yeah. Here.

438 00:33:22.610 00:33:23.200 Uttam Kumaran: Yeah, so maybe

439 00:33:23.630 00:33:24.260 Uttam Kumaran: talk about the.

440 00:33:24.260 00:33:27.060 patricktrainer: I gotta hop for like 10 min. I’ll be right back.

441 00:33:28.170 00:33:31.490 Uttam Kumaran: Okay, did you guys, Pat, did you and Artar talk about this yet?

442 00:33:31.490 00:33:33.500 patricktrainer: No, not yet not yet.

443 00:33:33.910 00:33:35.270 Uttam Kumaran: Okay. Okay. Good.

444 00:33:36.970 00:33:37.310 patricktrainer: And

445 00:33:37.930 00:33:39.280 patricktrainer: I’ll I’ll be right back.

446 00:33:39.970 00:33:41.029 patricktrainer: Okay, there, bye.

447 00:33:41.331 00:33:48.259 Nicolas Sucari: What I’m just before is the going into this did you create that ticket that you were talking about just right now?

448 00:33:48.890 00:33:50.449 Uttam Kumaran: Ye? Yeah, I I did.

449 00:33:51.950 00:33:53.270 Uttam Kumaran: It’s a

450 00:33:55.020 00:33:56.389 Uttam Kumaran: somewhere here.

451 00:33:58.030 00:33:59.220 Nicolas Sucari: Did you run the desktop.

452 00:33:59.220 00:33:59.710 Uttam Kumaran: 2.

453 00:33:59.710 00:34:00.290 Nicolas Sucari: All right.

454 00:34:01.250 00:34:02.180 Nicolas Sucari: Okay.

455 00:34:02.750 00:34:04.470 Uttam Kumaran: I’m gonna put in backlog.

456 00:34:06.090 00:34:06.770 Nicolas Sucari: Okay.

457 00:34:08.159 00:34:08.960 Nicolas Sucari: great

458 00:34:10.866 00:34:20.970 Nicolas Sucari: and one more thing, I was setting up evidence with Jared Patrick last week, and we ran into a lot of issues. And when we talk with the evidence team.

459 00:34:21.557 00:34:23.109 Nicolas Sucari: they asked, yeah.

460 00:34:23.110 00:34:27.929 Uttam Kumaran: Yeah, I actually just talked to Jared about, he said. He said he figured out he’s figured out using codespaces.

461 00:34:28.770 00:34:30.049 Nicolas Sucari: Yeah, he’s using course, face.

462 00:34:30.050 00:34:32.170 Uttam Kumaran: I don’t know what those spaces is.

463 00:34:33.060 00:34:44.754 Nicolas Sucari: Code space is kind of inside github like Vs code inside. Github, yeah, I don’t know. It’s kind of more laggy like you’re not. Gonna you’re not running locally

464 00:34:45.130 00:34:52.319 Nicolas Sucari: but it works. And he’s he’s using that because he’s finding an issue about that. dB, I don’t know

465 00:34:52.360 00:34:54.209 Nicolas Sucari: but what we found out.

466 00:34:54.219 00:34:54.899 Uttam Kumaran: Is the other piece.

467 00:34:54.900 00:34:56.000 Nicolas Sucari: Evidence team.

468 00:34:57.850 00:34:59.770 Nicolas Sucari: I don’t know, I think. Yes.

469 00:35:00.090 00:35:00.690 Uttam Kumaran: Okay.

470 00:35:01.730 00:35:09.239 Nicolas Sucari: I don’t know but what we found out with the evidence team is that we need to update the evidence version

471 00:35:09.633 00:35:22.970 Nicolas Sucari: to make it work. So I don’t know if that’s kind of something that everyone needs to do, locally or it’s something that we need to change on the full request that we have on the code that we come on. Github

472 00:35:23.400 00:35:24.379 Nicolas Sucari: just wanted to ask.

473 00:35:24.380 00:35:27.320 Uttam Kumaran: Yeah. So that’s something that yeah, that’s something that

474 00:35:27.520 00:35:38.209 Uttam Kumaran: this team will handle, which is like version updates, we already have one that’s an upgrade to Dbt 1 point, but we also should do one that’s around evidence.

475 00:35:40.240 00:35:41.670 Nicolas Sucari: Because I was wrong.

476 00:35:41.670 00:35:42.129 Uttam Kumaran: No, I mean.

477 00:35:42.130 00:35:43.160 Nicolas Sucari: So, if like, when I.

478 00:35:43.160 00:35:44.180 Uttam Kumaran: Joshua, I don’t know.

479 00:35:44.460 00:35:45.669 Uttam Kumaran: Yeah. Go ahead.

480 00:35:46.300 00:35:57.762 Nicolas Sucari: Yeah, I was. I was when I cloned the ripple. I was using the like. The the version of evidence that we have. There was 35.1. I think something like that. And

481 00:35:58.350 00:36:06.919 Nicolas Sucari: the guy from evidence told us to update that to 38 something. So yeah, probably we should everyone do that. So we don’t run into any issue.

482 00:36:08.880 00:36:09.310 Uttam Kumaran: Do we have.

483 00:36:09.310 00:36:10.019 Nicolas Sucari: Don’t worry. Thank you.

484 00:36:10.020 00:36:13.223 Uttam Kumaran: What about did you? You set up a docs for? How to

485 00:36:14.080 00:36:15.830 Uttam Kumaran: create evidence and stuff.

486 00:36:16.770 00:36:17.350 Nicolas Sucari: Yeah.

487 00:36:18.870 00:36:24.219 Nicolas Sucari: I need to update with that stuff that we discussed. But I have it here.

488 00:36:25.572 00:36:28.429 Nicolas Sucari: Setting up elementary.

489 00:36:28.930 00:36:29.970 Nicolas Sucari: I come here.

490 00:36:31.240 00:36:32.120 Uttam Kumaran: Ok, selling a.

491 00:36:32.120 00:36:33.269 Nicolas Sucari: Because, okay.

492 00:36:33.450 00:36:42.830 Nicolas Sucari: yeah, with with Jared, we did everything but when you click, start evidence, sometimes you run into errors. So we were like debugging with the evidence team.

493 00:36:43.292 00:36:53.360 Nicolas Sucari: I need to add, like those cases like how to try to figure out. It’s deleting node modules starting every day, everything again evidence, and that.

494 00:36:53.360 00:36:55.960 Uttam Kumaran: Yeah, there’s gonna be something where, like.

495 00:36:56.890 00:37:02.230 Uttam Kumaran: yeah, I wanna try. I don’t know whether everybody’s using like virtual environments or not.

496 00:37:02.820 00:37:09.550 Uttam Kumaran: But this is like, we probably need to do a whole doc on like how to set up virtual environments. Basically, the problem that happens.

497 00:37:09.550 00:37:09.940 Nicolas Sucari: Okay.

498 00:37:09.940 00:37:13.620 Uttam Kumaran: Is that you have different packages across different clients, and your laptop.

499 00:37:13.620 00:37:14.300 Nicolas Sucari: Yeah.

500 00:37:14.300 00:37:19.919 Uttam Kumaran: And it’s just a nightmare. So I usually set up virtual environments for every client.

501 00:37:20.040 00:37:21.060 Uttam Kumaran: And I switch between.

502 00:37:21.060 00:37:21.910 Nicolas Sucari: But I’m.

503 00:37:21.910 00:37:26.050 Uttam Kumaran: Cause. Basically, you could have separate pi, a virtual environment is basically.

504 00:37:26.220 00:37:27.150 Uttam Kumaran: if you just think about it.

505 00:37:27.150 00:37:27.690 Nicolas Sucari: Yeah, I know. And.

506 00:37:27.690 00:37:32.380 Uttam Kumaran: Separate instance of your machine. So I have different packages between each.

507 00:37:32.826 00:37:37.110 Uttam Kumaran: I don’t know whether I think we’ll probably add some steps to that. So

508 00:37:37.170 00:37:43.710 Uttam Kumaran: I’m gonna I’m just gonna create a ticket. That’s like update evidence version. I don’t really know whether this is

509 00:37:43.800 00:37:45.080 Uttam Kumaran: updating

510 00:37:45.530 00:37:46.780 Uttam Kumaran: the version.

511 00:37:46.830 00:37:50.789 Uttam Kumaran: Let’s like you dwell from locally. It’s updating the one that we have

512 00:37:51.140 00:37:54.320 Uttam Kumaran: in production. But either way we can figure that out.

513 00:37:54.938 00:37:57.030 Uttam Kumaran: I’m just gonna note to stop.

514 00:37:59.680 00:38:01.637 Uttam Kumaran: Stocks are here.

515 00:38:04.770 00:38:06.250 Nicolas Sucari: Okay, great.

516 00:38:06.600 00:38:13.359 Nicolas Sucari: I mean, because locally, we are running evidence. And we are running real locally right before

517 00:38:13.740 00:38:15.010 Nicolas Sucari: pushing anything

518 00:38:15.330 00:38:15.980 Nicolas Sucari: right?

519 00:38:15.980 00:38:19.960 Uttam Kumaran: But both of them have. Both of them have versions like your local.

520 00:38:19.960 00:38:20.960 Nicolas Sucari: Yeah, I don’t.

521 00:38:21.130 00:38:24.829 Uttam Kumaran: And the one that this version, similarly to evidence.

522 00:38:25.719 00:38:29.489 Uttam Kumaran: We were hosting evidence like real. They’re hosting it

523 00:38:29.730 00:38:34.129 Uttam Kumaran: evidence. We actually pay like 15 bucks a month, and we’re self hosting it.

524 00:38:35.760 00:38:37.600 Uttam Kumaran: I,

525 00:38:38.690 00:38:44.200 Uttam Kumaran: if we’re start like the thing is, I I did. I just wanted to handle it because I didn’t know whether we were gonna

526 00:38:44.650 00:38:45.670 Uttam Kumaran: use it

527 00:38:46.100 00:38:46.860 Uttam Kumaran: like.

528 00:38:47.230 00:38:50.290 Uttam Kumaran: I didn’t know whether we were gonna end up using it full time.

529 00:38:50.550 00:38:54.179 Uttam Kumaran: But if we do, then I may actually just switch to there

530 00:38:54.380 00:38:56.429 Uttam Kumaran: cloud version. I don’t know how much

531 00:38:57.090 00:38:58.979 Uttam Kumaran: the Cloud version boss.

532 00:39:01.150 00:39:03.420 Nicolas Sucari: If we are hosting it, we can like.

533 00:39:03.420 00:39:05.129 Uttam Kumaran: $10 per month

534 00:39:05.220 00:39:09.440 Uttam Kumaran: per viewer. Yeah, we’re basically hosting it for 15 bucks.

535 00:39:09.650 00:39:11.699 Uttam Kumaran: These guys are saying 15

536 00:39:12.550 00:39:15.010 Uttam Kumaran: per month per viewer, which is like.

537 00:39:15.430 00:39:18.140 Uttam Kumaran: I just don’t really think we need to pay for that.

538 00:39:19.943 00:39:23.749 Uttam Kumaran: I’m already hosting it for like 10 bucks a month or 15 bucks a month.

539 00:39:24.700 00:39:27.810 Nicolas Sucari: Okay, if we are hosting it, can we like

540 00:39:28.290 00:39:30.089 Nicolas Sucari: customize? The yeah. URL.

541 00:39:31.340 00:39:36.630 Uttam Kumaran: Yeah, yeah, we are. Yeah, yeah, we can. Right now, it’s on like, netlify

542 00:39:37.190 00:39:37.760 Uttam Kumaran: like a.

543 00:39:37.760 00:39:38.410 Nicolas Sucari: Okay.

544 00:39:38.410 00:39:40.930 Uttam Kumaran: Full parts ago, not like metlify dot app

545 00:39:43.280 00:39:44.920 Uttam Kumaran: But that may be another

546 00:39:45.310 00:39:47.169 Uttam Kumaran: doc for us to set up

547 00:39:51.750 00:39:52.550 Uttam Kumaran: but it’s like.

548 00:39:52.550 00:39:53.249 Nicolas Sucari: Need to go.

549 00:39:53.250 00:39:54.040 Uttam Kumaran: Yeah, if you

550 00:39:54.460 00:39:56.530 Uttam Kumaran: but this is for. But this is for row.

551 00:39:58.630 00:39:58.975 Nicolas Sucari: Okay?

552 00:39:59.710 00:40:03.920 Nicolas Sucari: And you said, real, we are okay, real. We are not hosting it. We are hosting evidence.

553 00:40:03.920 00:40:10.369 Uttam Kumaran: Real. We’re not hosting. I can’t do anything about the URL evidence. Our evidence thing is in. Yeah. So if you look at this, URL.

554 00:40:10.370 00:40:12.380 Nicolas Sucari: Yeah, and that’s fine. Okay?

555 00:40:12.380 00:40:18.529 Uttam Kumaran: So it’s a app. I mean, ideally, what I want to do is put this behind brain forge.

556 00:40:19.040 00:40:20.279 Nicolas Sucari: Exactly. Yeah.

557 00:40:20.690 00:40:24.050 Uttam Kumaran: So you go to like you go to like pool parts to go

558 00:40:24.110 00:40:26.230 Uttam Kumaran: like, how does that elementary one?

559 00:40:26.240 00:40:30.389 Uttam Kumaran: It’s like cool parts of go dot like brainforge.ai slash.

560 00:40:31.060 00:40:31.810 Uttam Kumaran: Yeah, exactly.

561 00:40:31.810 00:40:32.859 Nicolas Sucari: Like something like that.

562 00:40:32.860 00:40:33.820 Uttam Kumaran: Evidence.

563 00:40:33.820 00:40:34.210 Nicolas Sucari: Especially.

564 00:40:34.210 00:40:35.810 Uttam Kumaran: Yeah, let’s see, like

565 00:40:36.400 00:40:39.129 Uttam Kumaran: that’s like, mostly vanity. But I would like

566 00:40:39.280 00:40:44.319 Uttam Kumaran: ideally, it’s almost like you can go to Brain Forge. You can say, client login, and you can go to like your

567 00:40:45.080 00:40:49.939 Uttam Kumaran: you frankly should just go to your notion, Page, and then you can go from there, or something, but, like.

568 00:40:50.840 00:40:54.810 Uttam Kumaran: like, you know, these are all. These are all dreams of mine that I have so.

569 00:40:55.128 00:41:00.219 Nicolas Sucari: There is an easy one to try to figure that out. That is, use link tree.

570 00:41:01.900 00:41:05.999 Uttam Kumaran: Yeah, we could do link tree, too. But link tree, like I don’t want it to look jank.

571 00:41:07.100 00:41:27.929 Nicolas Sucari: No, no, but okay, no. I was just thinking, because if we are like using all of these tools, and we can like have, like our customized URL, where it says, like Brainforce, dot Pullparts, or something like that, it will be like more kind of I don’t know professional delivery to cost to the clients. But that’s fine. Just

572 00:41:28.120 00:41:29.000 Nicolas Sucari: thought.

573 00:41:30.310 00:41:32.488 Uttam Kumaran: Yeah. So let’s talk about

574 00:41:33.500 00:41:34.240 Uttam Kumaran: a stars.

575 00:41:34.240 00:41:36.050 Nicolas Sucari: Sorry. Let’s go with, yeah.

576 00:41:36.050 00:41:36.859 Atharv Gudi: Yeah, yeah, no. Problem.

577 00:41:36.860 00:41:37.720 Uttam Kumaran: Yeah, I’m

578 00:41:37.880 00:41:39.280 Uttam Kumaran: so

579 00:41:40.790 00:41:43.579 Uttam Kumaran: moving all tests to the test. Yeah.

580 00:41:43.740 00:41:47.379 Atharv Gudi: So I took a look at the pool parts ago. I found

581 00:41:47.480 00:41:49.750 Atharv Gudi: a test directory. I found

582 00:41:49.820 00:41:52.520 Atharv Gudi: a couple of tests, and I think the models

583 00:41:53.990 00:41:55.140 Atharv Gudi: somewhere.

584 00:41:56.510 00:41:56.900 Uttam Kumaran: Yeah.

585 00:41:56.900 00:41:57.550 Nicolas Sucari: So one ash.

586 00:41:57.550 00:41:58.749 Uttam Kumaran: Let me even give you.

587 00:41:58.830 00:42:07.969 Uttam Kumaran: Yeah, let me give you even an overview of this ticket, and I’ll actually just write a little bit. Yeah, let me share, and then I’ll even. I’ll even show you kind of like

588 00:42:08.180 00:42:11.550 Uttam Kumaran: what the goal is. Of this.

589 00:42:14.530 00:42:19.330 Uttam Kumaran: so the initial. So the primary goal of what this relates to is

590 00:42:19.850 00:42:21.409 Uttam Kumaran: like at the moment

591 00:42:26.030 00:42:27.999 Uttam Kumaran: at the moment, hold on one sec.

592 00:42:31.420 00:42:33.330 Uttam Kumaran: At the moment where

593 00:42:33.740 00:42:35.100 Uttam Kumaran: writing

594 00:42:40.220 00:42:41.980 Uttam Kumaran: Ryan and all past

595 00:42:45.810 00:42:47.300 Uttam Kumaran: individual

596 00:42:48.270 00:42:49.110 Uttam Kumaran: bio.

597 00:42:49.980 00:42:51.450 Uttam Kumaran: each model.

598 00:42:57.290 00:43:01.690 Uttam Kumaran: the animal file where alongside

599 00:43:03.450 00:43:04.250 Uttam Kumaran: model

600 00:43:05.700 00:43:06.730 Uttam Kumaran: curry.

601 00:43:13.130 00:43:14.859 Uttam Kumaran: The project.

602 00:43:16.270 00:43:21.289 Uttam Kumaran: The goal here is to centralize these.

603 00:43:21.600 00:43:23.900 Uttam Kumaran: and that’s

604 00:43:24.400 00:43:25.940 Uttam Kumaran: directory.

605 00:43:27.030 00:43:29.919 Uttam Kumaran: or both, for

606 00:43:30.330 00:43:32.180 Uttam Kumaran: a few reasons.

607 00:43:32.560 00:43:33.710 Uttam Kumaran: one

608 00:43:34.480 00:43:35.720 Uttam Kumaran: boat ownership

609 00:43:36.530 00:43:38.350 Uttam Kumaran: and do

610 00:43:40.560 00:43:41.400 Uttam Kumaran: or

611 00:43:41.900 00:43:43.890 Uttam Kumaran: being able to

612 00:43:44.880 00:43:46.070 Uttam Kumaran: breaker

613 00:43:46.730 00:43:47.960 Uttam Kumaran: certain

614 00:43:48.630 00:43:49.790 Uttam Kumaran: closed.

615 00:43:49.930 00:43:53.390 Uttam Kumaran: based on changes to this directory.

616 00:43:54.280 00:43:56.450 Uttam Kumaran: 3. To separate

617 00:43:58.440 00:43:59.320 Uttam Kumaran: out.

618 00:44:02.070 00:44:04.260 Uttam Kumaran: to separate out

619 00:44:04.710 00:44:07.489 Uttam Kumaran: what lives in each

620 00:44:07.960 00:44:09.450 Uttam Kumaran: camera file.

621 00:44:09.920 00:44:11.870 Uttam Kumaran: Currently, the Yaml

622 00:44:12.400 00:44:14.429 Uttam Kumaran: file is a petrol

623 00:44:16.070 00:44:18.440 Uttam Kumaran: for a lot of things

624 00:44:24.160 00:44:25.880 Uttam Kumaran: big. And

625 00:44:26.690 00:44:27.420 Uttam Kumaran: you know.

626 00:44:28.091 00:44:30.599 Uttam Kumaran: so in terms of like

627 00:44:32.910 00:44:34.100 Uttam Kumaran: acceptance.

628 00:44:34.410 00:44:36.930 Uttam Kumaran: So one is, I’m

629 00:44:37.660 00:44:40.649 Uttam Kumaran: there are. So there

630 00:44:40.870 00:44:43.940 Uttam Kumaran: are no tests that live

631 00:44:44.820 00:44:47.199 Uttam Kumaran: the slash models

632 00:44:51.090 00:44:52.215 Uttam Kumaran: to

633 00:44:55.950 00:45:00.009 Uttam Kumaran: The second thing is that we have

634 00:45:00.500 00:45:03.819 Uttam Kumaran: documentation on where that’s

635 00:45:04.130 00:45:06.070 Uttam Kumaran: live and

636 00:45:06.510 00:45:08.440 Uttam Kumaran: how to update them?

637 00:45:09.600 00:45:15.900 Uttam Kumaran: 3. So they’re not testing all we have documentation where tests live and where to update them.

638 00:45:16.390 00:45:18.279 Uttam Kumaran: Is there anything else?

639 00:45:25.160 00:45:28.469 Uttam Kumaran: I think that’s it. So

640 00:45:28.490 00:45:30.110 Uttam Kumaran: save this for now.

641 00:45:30.330 00:45:32.350 Uttam Kumaran: and let me show you

642 00:45:33.148 00:45:35.380 Uttam Kumaran: what in particular is

643 00:45:35.630 00:45:38.540 Uttam Kumaran: going on. So if I go to

644 00:45:39.320 00:45:41.599 Uttam Kumaran: go to the repo.

645 00:45:48.800 00:45:55.739 Uttam Kumaran: So right now, for example, if I go to like, if I go to models and I go to like shopify customers.

646 00:45:55.980 00:45:58.060 Uttam Kumaran: you’re gonna see here.

647 00:45:58.781 00:46:02.289 Uttam Kumaran: This may not be a good one.

648 00:46:02.850 00:46:05.329 Uttam Kumaran: Let’s go to all orders. For example.

649 00:46:06.650 00:46:08.839 Uttam Kumaran: you’re gonna see here, there are

650 00:46:09.460 00:46:14.070 Uttam Kumaran: expectation test here, volume, anomaly test here.

651 00:46:15.470 00:46:20.819 Uttam Kumaran: So what we actually want to do is the problem is in the same file.

652 00:46:20.930 00:46:24.440 Uttam Kumaran: You have information about the table.

653 00:46:24.500 00:46:27.849 Uttam Kumaran: You have some like tags on the model

654 00:46:28.377 00:46:34.000 Uttam Kumaran: and then you also the test. Here we additionally have stuff that’s relevant for light dash, which is a bi tool

655 00:46:34.020 00:46:35.160 Uttam Kumaran: that we’re using

656 00:46:35.543 00:46:40.200 Uttam Kumaran: and the problem is, this is like a catch-all for all of these related to this table.

657 00:46:40.510 00:46:43.828 Uttam Kumaran: The problem with that is is that if you go to

658 00:46:44.810 00:46:46.910 Uttam Kumaran: I don’t. I can’t see.

659 00:46:47.060 00:46:47.730 Uttam Kumaran: It’s

660 00:46:49.390 00:46:51.289 Uttam Kumaran: I go to.

661 00:46:55.610 00:46:57.589 Uttam Kumaran: Where is like the top area.

662 00:47:02.920 00:47:04.789 Uttam Kumaran: If I go to

663 00:47:05.000 00:47:05.940 Uttam Kumaran: sees it.

664 00:47:11.170 00:47:13.280 Uttam Kumaran: if I go to

665 00:47:15.100 00:47:16.410 Uttam Kumaran: pull request.

666 00:47:16.670 00:47:19.010 Uttam Kumaran: and I go to like this, for example.

667 00:47:20.065 00:47:24.320 Uttam Kumaran: This is not a good one. If I go to.

668 00:47:25.870 00:47:27.150 Uttam Kumaran: let’s see

669 00:47:39.870 00:47:41.830 Uttam Kumaran: trying to look at a specific

670 00:47:41.950 00:47:43.560 Uttam Kumaran: Pr that we could.

671 00:47:52.610 00:47:56.989 Uttam Kumaran: Okay, actually, yeah, this is probably good one. So if I go to look at

672 00:47:59.170 00:48:02.619 Uttam Kumaran: there are these 2 tests that run that are light dash related.

673 00:48:03.610 00:48:06.320 Uttam Kumaran: And they’re actually running unnecessarily, because

674 00:48:06.380 00:48:09.600 Uttam Kumaran: these changes have nothing to do with lightash.

675 00:48:09.730 00:48:12.579 Uttam Kumaran: But they are updates to this Yaml file

676 00:48:12.760 00:48:20.429 Uttam Kumaran: and the current workflow. The way it works is anytime a yaml file is updated. We trigger a series of jobs.

677 00:48:20.610 00:48:29.219 Uttam Kumaran: including jobs related to light dash. However, because these are tests. They should have nothing to do with light dash. But the jobs still run.

678 00:48:29.320 00:48:37.829 Uttam Kumaran: One of the ancillary reasons we’re doing this is also so we can continue just to run the jobs that are necessary when we make updates to files.

679 00:48:38.599 00:48:43.290 Uttam Kumaran: So the goal will be to take everything that’s in

680 00:48:43.350 00:48:45.929 Uttam Kumaran: models and move it to tests

681 00:48:45.990 00:48:48.710 Uttam Kumaran: right now, where there is nothing in tests.

682 00:48:49.220 00:48:52.089 Uttam Kumaran: But I think if I’m not mistaken.

683 00:48:52.880 00:48:56.450 Uttam Kumaran: if you just make a model file

684 00:48:56.990 00:49:02.929 Uttam Kumaran: model Yaml here with the same name. It should get picked up right? Or is there anything else we need to do.

685 00:49:07.060 00:49:08.000 patricktrainer: I hide.

686 00:49:08.420 00:49:10.769 patricktrainer: Don’t think so. I think.

687 00:49:13.360 00:49:14.510 patricktrainer: there is

688 00:49:14.630 00:49:17.969 patricktrainer: something that I was looking at over the weekend

689 00:49:19.410 00:49:24.669 patricktrainer: with regards to like DVD tests and the testing directory. It’s like they

690 00:49:24.710 00:49:27.239 patricktrainer: change something where, like, they’re

691 00:49:28.570 00:49:39.589 patricktrainer: delineating, or they’re they’re separating generic tests from custom tests and generic tests. I believe, have to live under

692 00:49:39.680 00:49:41.130 patricktrainer: the model key.

693 00:49:41.430 00:49:44.790 patricktrainer: But, as you were saying, like, when

694 00:49:45.380 00:49:51.179 patricktrainer: you create like that Yaml file, and you have like models, slash like your.

695 00:49:51.300 00:49:57.210 patricktrainer: and then underneath that, like my model key, or whatever that will override

696 00:49:58.900 00:50:01.669 patricktrainer: other configs, and like the different

697 00:50:01.860 00:50:04.079 patricktrainer: yaml files, so.

698 00:50:04.080 00:50:04.425 Uttam Kumaran: Okay.

699 00:50:06.430 00:50:08.130 patricktrainer: One thing that we

700 00:50:08.250 00:50:16.789 patricktrainer: could do instead of like so really like what I see is like, the main problem with our tests now is that there

701 00:50:17.260 00:50:28.369 patricktrainer: really mixed with all of that like light dash stuff, and then they’re also all, each in a separate file, which is like a pain. Yes, work with right.

702 00:50:28.440 00:50:29.860 patricktrainer: And so like.

703 00:50:30.070 00:50:33.960 patricktrainer: even if we’re not able to, let’s say, move

704 00:50:34.570 00:50:37.509 patricktrainer: like if it doesn’t allow us to move some of the like

705 00:50:37.700 00:50:39.250 patricktrainer: generic tests

706 00:50:39.913 00:50:50.650 patricktrainer: into the Test Directory, I think, having all of the tests just be in its own self contained. File like that would be like a world of difference.

707 00:50:50.780 00:50:55.060 patricktrainer: and then have, like the light dash, column definitions.

708 00:50:55.180 00:50:59.870 patricktrainer: and all of that like live in their own place. You see what I’m saying.

709 00:51:00.700 00:51:03.291 Uttam Kumaran: Yeah, yeah, no. I agree.

710 00:51:04.260 00:51:08.703 Uttam Kumaran: And again, I think they they have a test directory for a reason there. And

711 00:51:09.270 00:51:13.520 Uttam Kumaran: and we’re starting to do a ton of work on tests. So just to make it clear.

712 00:51:13.830 00:51:16.120 Uttam Kumaran: like, also, like, I want to make the

713 00:51:16.140 00:51:18.359 Uttam Kumaran: code review process a lot easier.

714 00:51:18.790 00:51:19.130 patricktrainer: So.

715 00:51:19.130 00:51:23.280 Uttam Kumaran: If there’s test, if there’s update for tests, it’ll basically just go to the Ae team

716 00:51:23.490 00:51:25.340 Uttam Kumaran: to have a review

717 00:51:25.787 00:51:28.010 Uttam Kumaran: and then also, like again, we can

718 00:51:28.200 00:51:34.360 Uttam Kumaran: probably set something up to actually like, run those tasks. And there’s a probably more ci updates we can make. So

719 00:51:35.680 00:51:36.620 Uttam Kumaran: yeah.

720 00:51:37.990 00:51:39.479 patricktrainer: Right, right.

721 00:51:43.298 00:51:48.380 Uttam Kumaran: So a thar of any questions. That’s a lot I think it’s probably gonna require some

722 00:51:48.400 00:51:51.239 Uttam Kumaran: time between you and Patrick.

723 00:51:51.240 00:51:51.800 Atharv Gudi: And.

724 00:51:51.800 00:51:57.450 Uttam Kumaran: Obviously some of it can happen, Async, and slack. But yeah, you have any questions there.

725 00:51:59.260 00:52:00.330 Atharv Gudi: I

726 00:52:00.600 00:52:15.710 Atharv Gudi: think so. All of these tests should be all of the anything that’s comes under tests in each of these yaml folders. It needs to be taken out and put in a separate yaml folder in the test Directory.

727 00:52:16.700 00:52:17.660 Uttam Kumaran: Yes,

728 00:52:18.760 00:52:22.780 Uttam Kumaran: I think maybe it’s probably best to to walk through with Patrick.

729 00:52:22.800 00:52:24.530 Uttam Kumaran: Yeah. One example.

730 00:52:24.927 00:52:30.639 Uttam Kumaran: Both the creation of the file. And like basically how to move it over. And then.

731 00:52:30.840 00:52:34.340 Uttam Kumaran: Pat, I think the best thing is like, if you just do Dvt run

732 00:52:34.570 00:52:35.870 Uttam Kumaran: locally, and it.

733 00:52:36.390 00:52:36.930 patricktrainer: Right.

734 00:52:36.930 00:52:39.889 Uttam Kumaran: Picked up or Dbt build, and it gets picked up

735 00:52:40.040 00:52:44.449 Uttam Kumaran: by elementary. Then we’re like good. And then it’s basically just the migration from there.

736 00:52:45.100 00:52:47.100 patricktrainer: Right, right, right, right.

737 00:52:50.090 00:52:54.840 Uttam Kumaran: So I thought I’d probably just grab time with Pat. And then the other thing is, I would read

738 00:52:54.950 00:52:55.990 Uttam Kumaran: this

739 00:52:56.770 00:53:02.389 Uttam Kumaran: Dvt, I would basically Google do what? Exactly what I did, which is Google search Dvt test and read the top 3 couple of things.

740 00:53:02.390 00:53:03.050 Atharv Gudi: Okay.

741 00:53:03.440 00:53:07.339 Uttam Kumaran: That’ll give you. That’ll give you a decent enough overview of like

742 00:53:07.770 00:53:09.909 Uttam Kumaran: what’s going on in the test.

743 00:53:10.270 00:53:12.449 Uttam Kumaran: And basically a test is like

744 00:53:12.740 00:53:17.009 Uttam Kumaran: it’s like, for example, if you want to say there can’t be duplicate ids in this table.

745 00:53:17.290 00:53:20.960 Uttam Kumaran: The so the query you need to write is just like select.

746 00:53:21.360 00:53:22.320 Uttam Kumaran: count

747 00:53:22.730 00:53:28.450 Uttam Kumaran: like, select id, and then count star. You can basically say, does this id show up

748 00:53:28.480 00:53:30.069 Uttam Kumaran: more than once in this table?

749 00:53:30.260 00:53:34.640 Uttam Kumaran: What Dbt does is allows you just to basically templatize that really easily.

750 00:53:34.770 00:53:39.300 Uttam Kumaran: Once you run your model, it will run that query. And if it returns back

751 00:53:39.340 00:53:42.999 Uttam Kumaran: a result that’s like multiple, it’ll fail.

752 00:53:43.220 00:53:49.870 Uttam Kumaran: Fail. Meaning it’s it’s like, there’s just some way of capturing that failure. And then you can take one or more actions from it.

753 00:53:50.275 00:53:54.699 Uttam Kumaran: Take that example, and then, like, blow it up, you can do a whole host of other things.

754 00:53:55.148 00:53:58.259 Uttam Kumaran: But that’s the that’s like the simplest example.

755 00:53:58.510 00:54:07.700 Uttam Kumaran: So we’re we’re testing for all sorts of stuff we’re testing, like other duplicate values does like, does the sum from one table equal to sum from another table?

756 00:54:09.130 00:54:31.659 Uttam Kumaran: you know, we’re doing like, basically, what this helps us do is just when we make updates. We know that not affecting other tables, not introducing bugs, or if, like the if, there’s bad data that comes in doesn’t change anything, and it just makes the Ae. Teams lives really easier. And then it gives Jacob and the analysis folks like a lot of peace of mind. So that’s the

757 00:54:32.720 00:54:34.920 Uttam Kumaran: that’s the overall goal.

758 00:54:35.310 00:54:39.429 Atharv Gudi: Okay, yeah, I think I’ll try to

759 00:54:39.880 00:54:44.509 Atharv Gudi: put something up with. I’ll get in touch with Patrick about this after the

760 00:54:45.200 00:54:48.110 Atharv Gudi: Well, I think Nico’s having a github

761 00:54:48.340 00:54:49.569 Atharv Gudi: pushing thing to.

762 00:54:49.570 00:54:50.789 Uttam Kumaran: Yeah, I’m I’m running.

763 00:54:50.790 00:54:51.130 Nicolas Sucari: More of.

764 00:54:51.130 00:54:55.420 Uttam Kumaran: Tomorrow. I’m basically, I’m basically just gonna show everyone how to push code. But

765 00:54:55.850 00:54:58.629 Uttam Kumaran: I would grab Thomas. Don’t worry about that like

766 00:54:59.044 00:55:02.489 Uttam Kumaran: that I’m gonna I’m gonna just walk everybody through pushing

767 00:55:02.630 00:55:07.709 Uttam Kumaran: like a Pr to our internal channel, to our internal repo, the brain forward actions.

768 00:55:07.850 00:55:11.169 Uttam Kumaran: And then you guys will know how to create a pull request

769 00:55:11.360 00:55:13.960 Uttam Kumaran: how to get it reviewed. But

770 00:55:14.200 00:55:17.738 Uttam Kumaran: you can meet with Patrick anytime today if we’re still working

771 00:55:18.210 00:55:18.630 Atharv Gudi: Yeah.

772 00:55:18.630 00:55:20.415 Uttam Kumaran: Or whatever he has time, or whatever.

773 00:55:20.670 00:55:28.887 Atharv Gudi: I’ve like, so far since whenever I have used Github, it’s always been through Vs code that I’ve been pushing and pulling code.

774 00:55:29.720 00:55:31.220 Atharv Gudi: So yeah.

775 00:55:31.220 00:55:32.360 patricktrainer: Yeah, that’s fine.

776 00:55:32.360 00:55:43.760 Uttam Kumaran: It’s not that. It’s not that bad. And I’m I’m just I. I’m not what I’m sharing. What I wanna just make sure is that everybody everybody’s in some one or more phases of like they use get a desktop or Vs code. So if you already know.

777 00:55:43.880 00:55:50.980 Uttam Kumaran: then it’ll just be a refresher, basically but the thing is that what may be new is like, how to get people to review

778 00:55:51.581 00:55:57.870 Uttam Kumaran: and how to add comments on how to respond to comments. Basically like how to do pull requests in a team.

779 00:55:58.302 00:56:00.320 Uttam Kumaran: Because none of the the code for our

780 00:56:00.430 00:56:03.400 Uttam Kumaran: for our clients. Everything has to get reviewed

781 00:56:03.590 00:56:05.040 Uttam Kumaran: before it gets merged.

782 00:56:05.110 00:56:09.139 Uttam Kumaran: So compared to like, if you’re working on a hobby project or something, and you could just keep

783 00:56:09.240 00:56:12.140 Uttam Kumaran: pushing stuff. There will be blockers, and then

784 00:56:12.330 00:56:15.719 Uttam Kumaran: we also run tests and whole hosts of things. It’s a little bit of like advanced.

785 00:56:16.120 00:56:16.730 Atharv Gudi: Okay.

786 00:56:16.730 00:56:18.690 Uttam Kumaran: Pr workshop.

787 00:56:19.270 00:56:28.069 Atharv Gudi: Yeah, hopefully, I’ve done the engineering onboarding properly, cause I think I do some things to my tweet, some things to my Vs code and all, and stuff.

788 00:56:29.180 00:56:36.510 Uttam Kumaran: That’s fine. Yeah, we’ll we’ll we’ll figure all that out. There’ll be small things that I’m sure we’ll have to figure out so.

789 00:56:37.420 00:56:46.080 Atharv Gudi: Okay, cool. I think that’s I. Those were the big questions. I had my ticket and the engineering onboarding stuff that I had.

790 00:56:48.550 00:56:54.311 Uttam Kumaran: Okay, cool. So maybe we just just to look back at stuff for this week. So the big things is

791 00:56:57.370 00:57:06.659 Uttam Kumaran: we want to create the security policy document. We want to finish up 2 fa and the key pairs ideally this week. This is basically kind of just like small things.

792 00:57:07.187 00:57:10.290 Uttam Kumaran: Tests will what Darby are gonna handle that.

793 00:57:10.300 00:57:14.749 Uttam Kumaran: And then I think the big thing, Pat is just continue working on the email stuff.

794 00:57:15.646 00:57:19.429 Uttam Kumaran: The moving test should clear up the running light, dash

795 00:57:19.900 00:57:23.780 Uttam Kumaran: on each pull, request thing. And then there’s some stuff on

796 00:57:23.870 00:57:28.740 Uttam Kumaran: Nixon regarding like moving data into Stella, but that those are all Stella specific.

797 00:57:29.050 00:57:30.810 Uttam Kumaran: So nothing

798 00:57:31.170 00:57:32.360 Uttam Kumaran: for y’all.

799 00:57:33.140 00:57:36.929 Uttam Kumaran: There’s 1 thing about concerning schema structure and snowflake

800 00:57:38.630 00:57:42.110 Uttam Kumaran: but I don’t. I think this is probably okay.

801 00:57:44.160 00:57:46.130 Uttam Kumaran: Oh, he has a thing for

802 00:57:46.420 00:57:48.480 Uttam Kumaran: right? Okay? So this, this will happen.

803 00:57:50.590 00:57:57.560 Uttam Kumaran: other than that, I think we’re fine. And then I want to book a meeting for this on Thursday. So I’m just gonna tag me

804 00:58:02.670 00:58:03.379 Uttam Kumaran: this sort of space.

805 00:58:03.380 00:58:04.740 Nicolas Sucari: Then there is one we can bring.

806 00:58:04.740 00:58:06.119 Uttam Kumaran: Suffer to suffer next week.

807 00:58:06.250 00:58:07.040 Uttam Kumaran: Yeah.

808 00:58:07.660 00:58:08.140 Nicolas Sucari: Okay.

809 00:58:08.140 00:58:08.610 Uttam Kumaran: Go ahead!

810 00:58:08.610 00:58:16.830 Nicolas Sucari: One more thing. What? What you are like discussing with this guy Justin from Eunice on

811 00:58:17.225 00:58:26.559 Nicolas Sucari: using like the their new model, do? Do we need to change something, or what we have on the integration from Unis, so that probably we should.

812 00:58:26.560 00:58:29.690 Uttam Kumaran: No, that’s they want us to create like a shop.

813 00:58:29.940 00:58:32.219 Uttam Kumaran: They want us to create a shopify.

814 00:58:32.360 00:58:35.640 Uttam Kumaran: app like, I just need to go into shopify and do something

815 00:58:35.810 00:58:38.149 Uttam Kumaran: kind of totally out of the scope.

816 00:58:38.525 00:58:43.949 Uttam Kumaran: It’s done nothing. It’s it’s just like I have to go to shopify and hit some buttons and approve them something. So

817 00:58:44.546 00:58:46.519 Uttam Kumaran: okay, I’ll go. I’ll create it. I’m gonna.

818 00:58:46.520 00:58:46.890 Nicolas Sucari: I mean.

819 00:58:46.890 00:58:52.180 Uttam Kumaran: I’m gonna I’ll create a ticket for that. But I’ll probably just handle it and try to document how I did it.

820 00:58:53.310 00:59:02.389 Nicolas Sucari: Yeah, okay, no. I was just wondering if that is what’s gonna change, how we get the data from shopify or anything. But if that’s not the case. That’s fine.

821 00:59:03.680 00:59:04.510 Uttam Kumaran: Yeah. No.

822 00:59:07.550 00:59:07.940 Nicolas Sucari: Okay.

823 00:59:07.940 00:59:08.800 Uttam Kumaran: Cool.

824 00:59:12.100 00:59:18.693 Uttam Kumaran: Okay, great. I thought. Please. If you’re blocked by anything, send a note to internal engineering. Nobody.

825 00:59:19.110 00:59:24.070 Uttam Kumaran: nobody will look at you any sort of way everybody has like is.

826 00:59:24.150 00:59:30.070 Uttam Kumaran: everybody’s like working on a ton of things. So I’m sure everybody would be down to help. So anytime you have an issue. Please slack in there.

827 00:59:30.240 00:59:33.290 Atharv Gudi: Perfect. Alright, I’ll I’ll be sure to reach out.

828 00:59:34.620 00:59:35.330 Atharv Gudi: Gone.

829 00:59:37.680 00:59:38.680 Atharv Gudi: Okay, guys.

830 00:59:39.420 00:59:40.780 Uttam Kumaran: With chat and slack.

831 00:59:41.610 00:59:42.340 patricktrainer: Awesome.

832 00:59:43.600 00:59:44.410 patricktrainer: Yep.

833 00:59:44.830 00:59:45.930 Nicolas Sucari: Thanks guys.

834 00:59:45.930 00:59:46.430 Uttam Kumaran: Okay.

835 00:59:46.430 00:59:47.290 patricktrainer: The uncle.

836 00:59:47.290 00:59:48.490 Nicolas Sucari: Bye, bye, session.

837 00:59:48.490 00:59:49.250 Atharv Gudi: Aye.