Meeting Title: Email-downloader-review Date: 2024-07-24 Meeting participants: Nicolas Sucari, Patrick Trainer, Uttam Kumaran


WEBVTT

1 00:00:29.640 00:00:31.320 Patrick Trainer: Yo. Oh.

2 00:00:32.070 00:00:33.389 Patrick Trainer: shit! Hold on!

3 00:00:41.690 00:00:42.750 Patrick Trainer: Can you hear me?

4 00:00:44.025 00:00:44.290 Uttam Kumaran: So.

5 00:00:44.290 00:00:45.110 Patrick Trainer: Thumbs down.

6 00:00:56.900 00:00:59.319 Patrick Trainer: Okay, I think. Okay, I can hear you much.

7 00:00:59.320 00:00:59.730 Uttam Kumaran: Okay.

8 00:00:59.730 00:01:06.730 Patrick Trainer: Cat cat sat on my keyboard before I got over here and was like fucking everything up so.

9 00:01:08.600 00:01:09.220 Uttam Kumaran: Unit.

10 00:01:09.220 00:01:10.340 Patrick Trainer: You know how it is.

11 00:01:10.340 00:01:11.140 Uttam Kumaran: Yeah.

12 00:01:12.400 00:01:16.350 Uttam Kumaran: I finally have my setup, like in like a 90%.

13 00:01:17.976 00:01:20.040 Uttam Kumaran: Mode, which is great.

14 00:01:20.040 00:01:21.700 Patrick Trainer: Nice. Nice.

15 00:01:24.080 00:01:27.820 Patrick Trainer: Yeah, you pretty much like settled in for the most part.

16 00:01:28.280 00:01:31.980 Uttam Kumaran: Yeah, everything and everything. It’s just there’s like projects.

17 00:01:32.200 00:01:33.540 Uttam Kumaran: I gotta do.

18 00:01:33.540 00:01:34.250 Patrick Trainer: Yeah.

19 00:01:34.250 00:01:38.399 Uttam Kumaran: So just going like one by one, I mean, like I’ve been.

20 00:01:38.560 00:01:42.640 Uttam Kumaran: It’s just been a lot of travel, so I haven’t had like free weekends, last few weekends, but now.

21 00:01:42.640 00:01:43.460 Patrick Trainer: Yeah, yeah.

22 00:01:43.460 00:01:45.629 Uttam Kumaran: Next month will be a lot of just like.

23 00:01:45.800 00:01:49.138 Uttam Kumaran: okay, cool. Let’s clean the garage entirely. Okay, cool.

24 00:01:49.880 00:01:52.799 Uttam Kumaran: Do this room. And like, yeah.

25 00:01:53.000 00:01:54.049 Patrick Trainer: Will be sweet.

26 00:01:54.300 00:02:06.309 Patrick Trainer: Yeah, you’ve got a pretty good like setup where you’re at now. It’s like you. You’re you’re like, right in line with those like Youtube bloggers where they got the stuff behind them and the.

27 00:02:06.800 00:02:08.930 Uttam Kumaran: Well, I just had this couch, and.

28 00:02:08.930 00:02:10.250 Patrick Trainer: Dude. It’s perfect.

29 00:02:10.259 00:02:16.369 Uttam Kumaran: And it’s it’s fits really well. And my friend came over yesterday after work to come like talk about some work stuff.

30 00:02:16.519 00:02:20.619 Uttam Kumaran: and we just like chat chill here, and I was like showing him some stuff, and it was like great.

31 00:02:21.153 00:02:25.799 Uttam Kumaran: And my dad was like you should put in there, because in case you want to like, hang on the couch. And I’m like.

32 00:02:26.439 00:02:32.109 Uttam Kumaran: Yeah, I guess you’re right. I just like want to sleep here and not go in the other room.

33 00:02:34.930 00:02:36.569 Uttam Kumaran: probably fine, but.

34 00:02:36.950 00:02:38.550 Patrick Trainer: Yeah, just live in the office.

35 00:02:38.770 00:02:39.490 Uttam Kumaran: Yeah.

36 00:02:40.050 00:02:41.499 Patrick Trainer: It’s not bad.

37 00:02:43.870 00:02:45.620 Patrick Trainer: not bad.

38 00:02:48.360 00:02:51.609 Patrick Trainer: Yeah, if you notice. Me.

39 00:02:52.100 00:02:56.050 Patrick Trainer: Yolo, pushing to Main on on stuff from

40 00:02:56.060 00:02:58.160 Patrick Trainer: just I’m doing actions.

41 00:03:01.660 00:03:03.089 Uttam Kumaran: I assume so.

42 00:03:03.300 00:03:04.060 Patrick Trainer: Yeah.

43 00:03:07.440 00:03:09.480 Patrick Trainer: it’s funny, like, if you look at

44 00:03:10.370 00:03:13.470 Patrick Trainer: like my, you know how like Github has, like the little

45 00:03:13.600 00:03:15.720 Patrick Trainer: squares like your get history.

46 00:03:16.260 00:03:17.460 Patrick Trainer: If you look at

47 00:03:17.790 00:03:19.350 Patrick Trainer: like my.

48 00:03:19.960 00:03:21.570 Patrick Trainer: those blocks like

49 00:03:21.700 00:03:24.640 Patrick Trainer: the really super dark ones I was like.

50 00:03:24.660 00:03:27.724 Patrick Trainer: I bet I was working on action Sunday.

51 00:03:30.580 00:03:32.909 Uttam Kumaran: It’s still kind of wild that like

52 00:03:33.430 00:03:34.540 Uttam Kumaran: can’t like

53 00:03:35.060 00:03:38.929 Uttam Kumaran: the local stuff doesn’t work, or I don’t know. Yeah, it’s just

54 00:03:39.730 00:03:41.000 Uttam Kumaran: it’s annoying.

55 00:03:41.000 00:03:42.530 Patrick Trainer: Wait. Which one’s this for.

56 00:03:43.780 00:03:47.009 Uttam Kumaran: No, no, I mean I I remember we were talking about like, okay.

57 00:03:47.010 00:03:48.430 Patrick Trainer: Colorado, yeah.

58 00:03:48.430 00:03:49.550 Uttam Kumaran: Stuff, but.

59 00:03:50.240 00:03:50.880 Patrick Trainer: Yeah.

60 00:03:51.320 00:03:53.879 Patrick Trainer: yeah, I don’t know if that’s just like a.

61 00:03:54.510 00:03:56.599 Uttam Kumaran: Figure. We’ll have to figure it out later.

62 00:03:56.980 00:03:57.947 Patrick Trainer: Yeah, if it’s

63 00:03:59.210 00:04:06.979 Patrick Trainer: I’ve been trying to think about like, why, that problem is the way it is. And it’s like, maybe because it is actually like

64 00:04:07.420 00:04:11.269 Patrick Trainer: we’re trying to run stuff up in Github’s own environment.

65 00:04:12.570 00:04:14.689 Patrick Trainer: It’s a maybe that just adds

66 00:04:15.260 00:04:16.420 Patrick Trainer: complexity.

67 00:04:17.170 00:04:20.730 Uttam Kumaran: Well, you just. You just can’t make like small mistakes. And that’s the problem.

68 00:04:20.730 00:04:23.880 Patrick Trainer: Right? Right? Right? Like there needs to be a better.

69 00:04:24.750 00:04:26.369 Patrick Trainer: I don’t know like debugging environment.

70 00:04:26.370 00:04:27.530 Nicolas Sucari: Hi, guys.

71 00:04:28.110 00:04:29.820 Nicolas Sucari: Sorry for very long.

72 00:04:31.230 00:04:32.760 Patrick Trainer: Not okay.

73 00:04:35.016 00:04:36.370 Nicolas Sucari: Really sorry.

74 00:04:37.970 00:04:41.830 Nicolas Sucari: I I don’t know if anyone else is gonna join. Actually.

75 00:04:41.890 00:04:45.680 Nicolas Sucari: this meeting was the idea of this meeting was just to

76 00:04:45.760 00:04:49.440 Nicolas Sucari: you, Patrick, to give us like a demo of what you’ve been working on on the

77 00:04:49.710 00:04:51.300 Nicolas Sucari: downloader.

78 00:04:52.160 00:04:52.530 Uttam Kumaran: Hey! Liz!

79 00:04:52.530 00:04:53.050 Nicolas Sucari: Yeah.

80 00:04:53.050 00:04:59.059 Uttam Kumaran: I think Jacob just got sick today, and then I totally forgot Ryan was, gonna be out this week.

81 00:04:59.722 00:05:06.669 Uttam Kumaran: But we will send them this, and then we’ll end up using pro hopefully this recording or something for a write up.

82 00:05:07.680 00:05:11.450 Patrick Trainer: Yeah, so let me actually

83 00:05:14.260 00:05:16.379 Patrick Trainer: get Github open.

84 00:05:39.350 00:05:40.210 Patrick Trainer: Okay?

85 00:05:42.600 00:05:44.110 Patrick Trainer: So there’s a couple

86 00:05:44.160 00:05:46.730 Patrick Trainer: gonna work or like, walk through.

87 00:05:46.970 00:05:49.250 Patrick Trainer: because there’s a couple of

88 00:05:49.390 00:05:52.739 Patrick Trainer: different components to it. There’s like the main

89 00:05:52.900 00:05:56.180 Patrick Trainer: package, which, like, allows us to

90 00:05:56.640 00:06:00.079 Patrick Trainer: even use the Api. And then there’s like a

91 00:06:01.656 00:06:10.830 Patrick Trainer: I guess, like another like script or application that then calls that package to then to then process the emails.

92 00:06:11.150 00:06:15.669 Patrick Trainer: And then on top of that, there’s an action which

93 00:06:15.710 00:06:17.390 Patrick Trainer: allows us to to

94 00:06:17.460 00:06:24.999 Patrick Trainer: call it, and dispatch events, to be able to like Trigger, this workflow, to go off of

95 00:06:25.180 00:06:27.230 Patrick Trainer: and so

96 00:06:27.810 00:06:28.980 Patrick Trainer: share

97 00:06:29.770 00:06:33.979 Patrick Trainer: kind of like, walk you through 1st in the

98 00:06:34.240 00:06:35.819 Patrick Trainer: hit. This little

99 00:06:36.460 00:06:38.580 Patrick Trainer: what you might call it the

100 00:06:38.790 00:06:40.100 Patrick Trainer: Zoom Bar.

101 00:06:40.120 00:06:41.320 Patrick Trainer: It’s always in the way.

102 00:06:43.800 00:06:46.159 Patrick Trainer: take you 1st through the like. The

103 00:06:47.000 00:06:49.700 Patrick Trainer: very low level stuff. And which is

104 00:06:50.070 00:06:51.819 Patrick Trainer: the attachment downloader.

105 00:06:53.430 00:07:01.409 Patrick Trainer: basically what this is is like a library that allows us to interface with the Google Api.

106 00:07:01.810 00:07:03.050 Patrick Trainer: And so

107 00:07:03.130 00:07:10.109 Patrick Trainer: there’s 4 different modules in it. There’s like operations. There’s service downloader and a logger

108 00:07:11.060 00:07:14.400 Patrick Trainer: log is pretty self-explanatory. It’s just setting up logging.

109 00:07:14.750 00:07:16.150 Patrick Trainer: A

110 00:07:16.700 00:07:24.509 Patrick Trainer: this is just defining the package then meat and potatoes is around like this service module.

111 00:07:24.790 00:07:27.099 Patrick Trainer: And then this is where

112 00:07:27.380 00:07:31.690 Patrick Trainer: we’re actually doing like authentication

113 00:07:32.645 00:07:38.879 Patrick Trainer: and like getting the credentials and then returning the gmail service the Gmail service.

114 00:07:39.080 00:07:41.880 Patrick Trainer: Here is what’s actually going to

115 00:07:43.950 00:07:44.770 Patrick Trainer: I’m

116 00:07:45.210 00:07:48.160 Patrick Trainer: give us those methods needed to

117 00:07:48.520 00:07:50.630 Patrick Trainer: work with the the Gmail Api.

118 00:07:51.606 00:08:00.720 Patrick Trainer: Then we have these like these operations. Which is where, like the actions at like. So after we have the Gmail service

119 00:08:02.170 00:08:04.450 Patrick Trainer: we need to be able to operate

120 00:08:04.720 00:08:18.650 Patrick Trainer: on whatever we’re using in this case, like we’re using like the Gmail service. So these operations are how we like go through and look for messages or attachments. And basically stuff that’s

121 00:08:18.690 00:08:28.029 Patrick Trainer: we’re programmatically doing that we would normally be doing on. Sorry. Can you hear my cat behind me? Is it? Okay? Cool? Just like crying.

122 00:08:29.570 00:08:31.480 Patrick Trainer: really annoying.

123 00:08:33.770 00:08:40.470 Patrick Trainer: So we have the these Gmail operations, and you can see that we’re like passing in the service to it.

124 00:08:40.610 00:08:43.499 Patrick Trainer: And then this is like where we’re

125 00:08:44.159 00:08:48.499 Patrick Trainer: like calling these different operations that we need to get

126 00:08:48.780 00:08:50.350 Patrick Trainer: so like

127 00:08:51.120 00:08:52.606 Patrick Trainer: to get

128 00:08:54.000 00:08:58.445 Patrick Trainer: messages or emails. So you can think of messages as emails.

129 00:08:59.280 00:09:01.510 Patrick Trainer: there’s like basically a query

130 00:09:02.052 00:09:08.070 Patrick Trainer: and that needs to get executed, which corresponds directly to to here.

131 00:09:08.843 00:09:15.570 Patrick Trainer: And we’re going through. We’re grabbing all of the messages and we need to get their ids.

132 00:09:15.800 00:09:25.789 Patrick Trainer: and so part of, like the fetches, fetches, a message is getting its specific ids. So this is returning, fetch messages, is returning a list

133 00:09:25.990 00:09:37.459 Patrick Trainer: of message ids and then fetch message is where it’s actually where we’re actually grabbing an individual message given its message. Id

134 00:09:39.190 00:09:43.490 Patrick Trainer: And this is the same idea with attachments.

135 00:09:43.842 00:09:49.969 Patrick Trainer: So if a message has an attachment it’s also going to have like an attachment id to it

136 00:09:50.629 00:09:57.670 Patrick Trainer: which is partially part of like the message response that we get from here.

137 00:09:58.020 00:09:59.820 Patrick Trainer: So we pass in

138 00:09:59.850 00:10:03.700 Patrick Trainer: a message. Id. We grab its attachment. Id.

139 00:10:04.000 00:10:08.290 Patrick Trainer: and then that’s where we’re actually returning the attachment

140 00:10:09.007 00:10:14.152 Patrick Trainer: and then over here is where we can save that attachment and then define

141 00:10:14.700 00:10:15.620 Patrick Trainer: where

142 00:10:16.880 00:10:20.070 Patrick Trainer: that attachment gets saved.

143 00:10:20.630 00:10:27.670 Patrick Trainer: Got like an input. And output directory that it that it goes into and then, just like some helper methods of like.

144 00:10:28.960 00:10:32.439 Patrick Trainer: did it complete. So this is like the

145 00:10:32.630 00:10:35.759 Patrick Trainer: like, if we want to go in and say, like.

146 00:10:36.060 00:10:41.289 Patrick Trainer: we want to like, find these messages or attachments. Or if there’s like

147 00:10:43.340 00:10:45.530 Patrick Trainer: like other

148 00:10:45.930 00:11:02.509 Patrick Trainer: things that can happen in an email like this is what we’re going to extend. Like, if there’s yeah, if there’s other stuff that, like other functionality of like us needing to go through and use this. Use the Gmail Api, for this is what we would extend.

149 00:11:02.650 00:11:03.265 Patrick Trainer: Okay,

150 00:11:04.120 00:11:07.919 Patrick Trainer: And so this is all pretty like

151 00:11:08.150 00:11:11.440 Patrick Trainer: nice to use in the sense that it is

152 00:11:11.500 00:11:15.790 Patrick Trainer: like fully packaged and so like, what

153 00:11:16.070 00:11:26.069 Patrick Trainer: you end up doing and what our calling workflows end up doing is just like Pip installing from here, Pip, installing this gmad.

154 00:11:26.488 00:11:32.970 Patrick Trainer: and that installs the package in in your python path, and then you’re just able to kind of call it

155 00:11:33.917 00:11:35.830 Patrick Trainer: like in a

156 00:11:36.420 00:11:38.150 Patrick Trainer: like as a library.

157 00:11:38.390 00:11:40.700 Uttam Kumaran: Right now. We’re just taking all attachments.

158 00:11:41.390 00:11:47.120 Patrick Trainer: Right? Right? Right? So we we’re getting like the attachments. But we’re also we’re

159 00:11:47.430 00:11:48.420 Patrick Trainer: a

160 00:11:48.970 00:11:53.369 Patrick Trainer: like in part of the the message query like that we’re looking for here.

161 00:11:53.380 00:11:54.880 Patrick Trainer: or that we

162 00:11:55.100 00:12:03.360 Patrick Trainer: input there. We’re defining like file types which are basically like. So if we want to return, like all the Csv

163 00:12:03.420 00:12:05.979 Patrick Trainer: attachments. That’s where we can do that.

164 00:12:06.090 00:12:07.250 Patrick Trainer: Okay, okay?

165 00:12:08.350 00:12:15.869 Patrick Trainer: And then so that will lead us to. So needing to actually use.

166 00:12:16.645 00:12:17.319 Patrick Trainer: This

167 00:12:17.730 00:12:21.790 Patrick Trainer: package or library, or whatever you want to call it, to

168 00:12:23.000 00:12:26.750 Patrick Trainer: actually go and like, implement the the steps.

169 00:12:26.800 00:12:28.930 Patrick Trainer: And so that’s

170 00:12:29.040 00:12:30.860 Patrick Trainer: in this email

171 00:12:30.930 00:12:32.070 Patrick Trainer: processor.

172 00:12:32.665 00:12:35.569 Patrick Trainer: and so what we can see here.

173 00:12:35.660 00:12:36.930 Patrick Trainer: I mean, these are all

174 00:12:37.060 00:12:40.330 Patrick Trainer: empty. So don’t think I’ve actually written docs yet.

175 00:12:41.227 00:12:50.969 Patrick Trainer: But yeah, what this is doing. It’s like we have our requirements, which we’re pulling from here. You can see that.

176 00:12:52.010 00:12:55.789 Patrick Trainer: But as we go into our actual

177 00:12:57.490 00:13:04.859 Patrick Trainer: library here, or package here. We have a couple of different things. So we’ve got the attachment processor.

178 00:13:05.050 00:13:08.440 Patrick Trainer: a message body processor, and then the

179 00:13:09.004 00:13:13.440 Patrick Trainer: link downloader and I’ll walk through these

180 00:13:14.386 00:13:15.160 Patrick Trainer: real quick

181 00:13:15.908 00:13:24.759 Patrick Trainer: and so, and then we also have this config file. But it’s empty, because that’s actually forget all about what I was saying, There.

182 00:13:25.593 00:13:32.209 Patrick Trainer: we also have the ability to like, we can call this as a module itself.

183 00:13:32.970 00:13:33.670 Patrick Trainer: And

184 00:13:34.200 00:13:35.800 Patrick Trainer: we can.

185 00:13:36.755 00:13:44.274 Patrick Trainer: Basically so that allows us to call it from the command line and supply arguments to it.

186 00:13:45.250 00:13:48.840 Patrick Trainer: like calling it directly. So what I mean by that is like

187 00:13:49.230 00:13:55.572 Patrick Trainer: calling it like Python M. Slash attachments. And that’s going to kick off

188 00:13:56.400 00:14:00.200 Patrick Trainer: like this here and like looking for

189 00:14:03.120 00:14:08.299 Patrick Trainer: basically like, here we have these arguments. If it’s an argument, it’s going to, then

190 00:14:08.310 00:14:11.060 Patrick Trainer: kick off like our process attachment.

191 00:14:13.270 00:14:16.930 Patrick Trainer: Yeah, what you might call it that we’re importing from here.

192 00:14:17.120 00:14:23.130 Patrick Trainer: so we could take a look into that all this really is doing is

193 00:14:23.170 00:14:27.270 Patrick Trainer: so it takes in our our config

194 00:14:27.743 00:14:30.990 Patrick Trainer: looks at what we’re doing, and then

195 00:14:31.370 00:14:36.160 Patrick Trainer: configures for our I’ll and I’ll get to what this config looks like, too.

196 00:14:36.725 00:14:43.700 Patrick Trainer: Making sure that our directory actually exists. And then logging all this stuff as well.

197 00:14:44.200 00:14:46.399 Patrick Trainer: And then in here

198 00:14:47.185 00:14:55.064 Patrick Trainer: this is where we’re actually, then this is where we actually call from that gmad

199 00:14:56.080 00:14:59.259 Patrick Trainer: package that we were just on like the main one

200 00:14:59.926 00:15:05.279 Patrick Trainer: and so we have this interface of it of like, create downloader

201 00:15:06.226 00:15:08.693 Patrick Trainer: and let me actually

202 00:15:09.950 00:15:11.730 Patrick Trainer: think if I can go

203 00:15:12.550 00:15:13.280 Patrick Trainer: that.

204 00:15:18.650 00:15:22.849 Patrick Trainer: Okay? So we have, let’s look for this create downloader.

205 00:15:31.370 00:15:35.000 Patrick Trainer: So we have this create downloader method.

206 00:15:35.350 00:15:39.104 Patrick Trainer: which this is what takes in

207 00:15:40.400 00:15:46.769 Patrick Trainer: or when we pass in the stuff from like the module that we called, like the python M

208 00:15:46.930 00:15:48.720 Patrick Trainer: run attachment.

209 00:15:49.060 00:15:52.539 Patrick Trainer: We have this downloader method which is going to

210 00:15:53.500 00:15:54.300 Patrick Trainer: return

211 00:15:55.190 00:16:01.689 Patrick Trainer: based on the type that we passed in. It’s going to return the appropriate

212 00:16:01.760 00:16:09.620 Patrick Trainer: type of downloader and so if we look in here we can. It returns this class.

213 00:16:10.296 00:16:11.530 Patrick Trainer: Which then

214 00:16:12.670 00:16:14.920 Patrick Trainer: implement this download method?

215 00:16:15.955 00:16:20.399 Patrick Trainer: Which then allows us to actually like download

216 00:16:20.900 00:16:28.370 Patrick Trainer: attachments? Because there’s like there’s the payloads are different, and how they have to be parsed

217 00:16:29.640 00:16:33.209 Patrick Trainer: and so like we can compare that to

218 00:16:33.360 00:16:35.420 Patrick Trainer: like the message body downloader.

219 00:16:35.866 00:16:39.793 Patrick Trainer: It’s just a little bit different in that. We’re just getting

220 00:16:40.610 00:16:46.519 Patrick Trainer: like the text from an email. And then having to the way we have to

221 00:16:46.570 00:16:47.840 Patrick Trainer: return those.

222 00:16:48.040 00:16:50.640 Patrick Trainer: So in this like, if we want to

223 00:16:50.770 00:16:52.630 Patrick Trainer: like download different

224 00:16:54.510 00:17:00.069 Patrick Trainer: types of messages, or we can just create

225 00:17:00.250 00:17:03.390 Patrick Trainer: these additional like downloader classes.

226 00:17:03.880 00:17:07.209 Patrick Trainer: And we won’t have to change any sort of

227 00:17:08.520 00:17:20.890 Patrick Trainer: like. We keep the interface the same and so that like, we can just kind of like plug and play and make some like really loosely decoupled or loosely coupled from like.

228 00:17:21.079 00:17:28.670 Patrick Trainer: if we were to call this directly. So this it’s just kind of like a, it’s like a dispatcher method works out pretty nice.

229 00:17:30.280 00:17:31.440 Patrick Trainer: and so

230 00:17:31.740 00:17:34.960 Patrick Trainer: this is supplying all of those inputs.

231 00:17:35.090 00:17:38.230 Patrick Trainer: It’s going. And it it gets that

232 00:17:39.690 00:17:40.820 Patrick Trainer: from there

233 00:17:40.880 00:17:42.429 Patrick Trainer: this is the same deal.

234 00:17:43.250 00:17:43.920 Patrick Trainer: Like

235 00:17:44.020 00:17:45.759 Patrick Trainer: kind of like hints at

236 00:17:45.910 00:17:46.590 Patrick Trainer: like.

237 00:17:46.930 00:17:50.269 Patrick Trainer: because the interfaces are super clean.

238 00:17:50.540 00:17:54.019 Patrick Trainer: Really, the code is like super reasonable, which is nice.

239 00:17:55.560 00:17:57.869 Patrick Trainer: and we have, like the link downloader.

240 00:17:58.985 00:18:05.794 Patrick Trainer: This was actually kind of interesting to do so in the attentive emails they send us.

241 00:18:06.970 00:18:16.030 Patrick Trainer: it’s basically an HTML page that has a bunch of like Urls posted in it. One of them is a button called download button.

242 00:18:16.570 00:18:21.379 Patrick Trainer: What this is doing is so we’re getting like the file

243 00:18:22.262 00:18:24.710 Patrick Trainer: or sorry we’re get

244 00:18:25.460 00:18:31.129 Patrick Trainer: getting the stuff in the mess like from the messages and we’re getting that HTML,

245 00:18:31.970 00:18:38.990 Patrick Trainer: and we are using like beautiful soup we go through, and we’re trying to find all of the

246 00:18:39.150 00:18:41.290 Patrick Trainer: Href a links

247 00:18:41.750 00:18:48.059 Patrick Trainer: and one of their buttons that they always have. It says Download. Csv.

248 00:18:48.160 00:18:52.199 Patrick Trainer: and so we’re looking for that parsing that and grabbing that link

249 00:18:52.320 00:18:55.270 Patrick Trainer: it actually grabbing all of the links in there

250 00:18:56.016 00:19:00.909 Patrick Trainer: and then we go, and we’re wanting to download that file So

251 00:19:01.110 00:19:02.540 Patrick Trainer: the file that

252 00:19:02.620 00:19:06.550 Patrick Trainer: we’re downloading, or that’s linked in that like.

253 00:19:07.200 00:19:11.580 Patrick Trainer: get Csv or download Csv button.

254 00:19:11.740 00:19:17.537 Patrick Trainer: It’s a URL for like like send grid so that they could track that which then.

255 00:19:17.900 00:19:18.870 Uttam Kumaran: Converts to it.

256 00:19:19.220 00:19:19.930 Patrick Trainer: What’s that?

257 00:19:20.080 00:19:21.489 Uttam Kumaran: Is it off or no?

258 00:19:21.490 00:19:23.319 Patrick Trainer: No, no, no, no! So.

259 00:19:23.320 00:19:26.279 Uttam Kumaran: That’s what that’s my worry. That was my main worry. Really.

260 00:19:26.280 00:19:34.319 Patrick Trainer: Well, so you know it. Actually it it yes and no. So and I’ll I’ll explain that. So

261 00:19:34.860 00:19:36.610 Patrick Trainer: when you

262 00:19:36.880 00:19:39.769 Patrick Trainer: 1st hit like you

263 00:19:40.760 00:19:42.350 Patrick Trainer: like. Make a request

264 00:19:42.440 00:19:43.749 Patrick Trainer: to that. Yeah, URL.

265 00:19:44.740 00:19:45.500 Patrick Trainer: It

266 00:19:46.170 00:19:56.920 Patrick Trainer: sends you through like sendgrids thing, and then it goes through a tentative. It like it has a bunch of like 300 redirects. And so in that

267 00:19:57.548 00:20:00.481 Patrick Trainer: what we’re able to do or

268 00:20:01.680 00:20:04.520 Patrick Trainer: when those redirects are happening.

269 00:20:05.266 00:20:12.789 Patrick Trainer: It’s appending like the redirects URL to its header files.

270 00:20:12.840 00:20:14.820 Patrick Trainer: right? So it knows where to go

271 00:20:15.567 00:20:22.260 Patrick Trainer: and so what we’re doing. And it’s we send like a request

272 00:20:22.646 00:20:27.699 Patrick Trainer: and we get that response. But then, because it’s appended the new URL to it.

273 00:20:28.120 00:20:39.449 Patrick Trainer: we’re grabbing this redirect yeah, URL, that it’s that it’s using. And that redirect yeah. URL, that they’re using is a direct link to their S 3 bucket.

274 00:20:39.980 00:20:41.179 Uttam Kumaran: Oh, okay.

275 00:20:41.890 00:20:45.159 Patrick Trainer: And and so and so, yeah, right?

276 00:20:45.170 00:20:46.999 Patrick Trainer: And so we

277 00:20:48.670 00:20:52.419 Patrick Trainer: get to basically like, skip their off

278 00:20:53.346 00:20:59.879 Patrick Trainer: and we have the direct link to like the the artifact in their s. 3 bucket.

279 00:21:00.150 00:21:00.950 Uttam Kumaran: And you’re the same.

280 00:21:00.950 00:21:11.179 Patrick Trainer: And and so, yeah, that’s and that’s what we’re getting there. And so from there, like, we download the

281 00:21:12.031 00:21:21.259 Patrick Trainer: download the file. We have this like, get this file name from the URL cause. It’s in the this long query string

282 00:21:21.780 00:21:26.410 Patrick Trainer: which is this? And basically so it gives us this report.

283 00:21:27.990 00:21:31.230 Patrick Trainer: like this is like the long URL. It’s like S.

284 00:21:31.310 00:21:35.520 Patrick Trainer: A. 10 of S. 3 generated reports, I imagine, like

285 00:21:36.140 00:21:54.500 Patrick Trainer: our account number 80 56 and then it has this and a whole bunch of like query strings shit so this drops everything after the query string so that we have just here. And then we split

286 00:21:54.710 00:21:58.650 Patrick Trainer: on the slashes take the last

287 00:21:58.810 00:22:01.980 Patrick Trainer: index. So we only return

288 00:22:02.419 00:22:11.430 Patrick Trainer: the name of the Csv. And so that’s how we’re dynamically naming the Csv’s or grabbing the the Csv’s name from

289 00:22:12.910 00:22:15.020 Patrick Trainer: that downloaded filed. It’s like yeah.

290 00:22:15.020 00:22:15.370 Uttam Kumaran: Yeah.

291 00:22:15.370 00:22:16.420 Patrick Trainer: Whatever

292 00:22:17.010 00:22:19.580 Patrick Trainer: the name of the file is and what it’s downloaded.

293 00:22:19.650 00:22:26.269 Patrick Trainer: if that doesn’t work. There’s I have this like fallback method, where it’s just going to create

294 00:22:26.530 00:22:35.308 Patrick Trainer: a 256 hash of what the file name is. Just so we have a unique file each time.

295 00:22:36.330 00:22:37.200 Patrick Trainer: It’s

296 00:22:38.250 00:22:42.935 Patrick Trainer: I. It it had. It’s fallen back a number number of times, but like

297 00:22:43.310 00:22:46.083 Patrick Trainer: it, it should just this should just.

298 00:22:46.480 00:22:47.120 Uttam Kumaran: Yeah.

299 00:22:47.858 00:22:50.001 Patrick Trainer: And then it writes, and

300 00:22:51.010 00:23:09.599 Patrick Trainer: the downloads, the file and that’s kind of like where all this like processing is coming in. This is like the calling of these functions up here. And then in part of this, what it’s doing, it’s creating these like temporary directories. This method just like cleans up

301 00:23:09.740 00:23:12.059 Patrick Trainer: everything at the end.

302 00:23:12.745 00:23:18.919 Patrick Trainer: And so to kind of like, show this all in how it works

303 00:23:19.180 00:23:21.329 Patrick Trainer: or all put together.

304 00:23:22.640 00:23:24.170 Patrick Trainer: We have

305 00:23:24.790 00:23:26.609 Patrick Trainer: this process email.

306 00:23:28.760 00:23:29.410 Uttam Kumaran: Relaxed.

307 00:23:29.860 00:23:33.957 Patrick Trainer: And so it’s like we have. I can just go with like the dispatch.

308 00:23:35.020 00:23:39.610 Patrick Trainer: So we have, like our attachment. Query, which is

309 00:23:40.117 00:23:47.260 Patrick Trainer: like we have, we set a default of like just grabbing these Csv’s. So what this looks like in Gmail

310 00:23:47.290 00:23:49.460 Patrick Trainer: would be like that.

311 00:23:49.830 00:23:54.049 Patrick Trainer: And then we have the email body? Query.

312 00:23:54.594 00:23:57.870 Patrick Trainer: this is like where we’re setting from the person

313 00:23:59.450 00:24:04.420 Patrick Trainer: and like, when we call it like, I have it just coming from you. You don’t mind. Need to

314 00:24:04.680 00:24:06.390 Patrick Trainer: make this more primed.

315 00:24:06.780 00:24:09.739 Patrick Trainer: But so if we like, come in here

316 00:24:10.070 00:24:12.970 Patrick Trainer: and we search for file name. Csv.

317 00:24:13.190 00:24:15.489 Patrick Trainer: it’s gonna filter down our inbox.

318 00:24:15.990 00:24:18.014 Patrick Trainer: So we get

319 00:24:19.470 00:24:21.600 Patrick Trainer: all of the emails that have

320 00:24:21.640 00:24:23.700 Patrick Trainer: a Csv in it, right?

321 00:24:23.880 00:24:25.350 Patrick Trainer: And so.

322 00:24:27.000 00:24:27.860 Patrick Trainer: we have

323 00:24:28.350 00:24:29.780 Patrick Trainer: our main job

324 00:24:30.552 00:24:35.640 Patrick Trainer: here, which we’re going checking out the repo, setting up python.

325 00:24:35.880 00:24:39.479 Patrick Trainer: the the Github Cli authenticating there.

326 00:24:39.770 00:24:43.909 Patrick Trainer: and what we’re needing to do to

327 00:24:45.395 00:24:46.370 Patrick Trainer: a

328 00:24:46.640 00:24:49.340 Patrick Trainer: to use this like library

329 00:24:49.430 00:24:51.462 Patrick Trainer: whatnot, we need to

330 00:24:52.110 00:24:55.129 Patrick Trainer: basically need to like Ssh into it.

331 00:24:55.770 00:25:06.819 Patrick Trainer: Which this has a poiky. That’s I mean, not very important, so we authenticate into here allows us to to pip, install our requirements.

332 00:25:07.875 00:25:09.600 Patrick Trainer: We have a

333 00:25:10.199 00:25:11.359 Patrick Trainer: service, Count.

334 00:25:12.270 00:25:13.130 Patrick Trainer: And

335 00:25:13.580 00:25:16.719 Patrick Trainer: our service count like credentials.

336 00:25:17.220 00:25:20.630 Patrick Trainer: and that is saved in

337 00:25:20.940 00:25:22.600 Patrick Trainer: as a secret.

338 00:25:23.423 00:25:32.029 Patrick Trainer: Which then gets so we echo that secret, and then we’re writing it to this file at Runtime.

339 00:25:32.170 00:25:32.550 Uttam Kumaran: Okay.

340 00:25:32.790 00:25:35.690 Patrick Trainer: And so that keeps everything out of out of.

341 00:25:36.035 00:25:36.380 Uttam Kumaran: Here.

342 00:25:36.702 00:25:38.640 Patrick Trainer: Because, yeah, we can expose that

343 00:25:39.261 00:25:43.580 Patrick Trainer: and then we have. We’re setting up our config.

344 00:25:43.630 00:25:47.360 Patrick Trainer: which pretty much the same deal.

345 00:25:47.420 00:25:49.649 Patrick Trainer: It’s where. So we have.

346 00:25:49.650 00:25:54.379 Uttam Kumaran: And this is all like in the during the run, like none. None of this is persistent.

347 00:25:54.380 00:25:56.950 Patrick Trainer: Right? Yeah, it’s all built at Runtime

348 00:25:58.303 00:26:01.720 Patrick Trainer: and so we have.

349 00:26:01.920 00:26:08.300 Patrick Trainer: we’re looking for creating a directory. We’re checking. If that config file exists.

350 00:26:09.950 00:26:13.329 Patrick Trainer: looking for it in the this repos route.

351 00:26:14.830 00:26:15.690 Patrick Trainer: If it

352 00:26:16.870 00:26:20.990 Patrick Trainer: doesn’t, or it, it creates the config file there.

353 00:26:21.130 00:26:22.939 Patrick Trainer: And so what

354 00:26:24.226 00:26:28.869 Patrick Trainer: it’s doing. And I can actually show a config file over here.

355 00:26:29.120 00:26:33.019 Patrick Trainer: We have these like different keys. So we get like the Gmail Api.

356 00:26:33.110 00:26:34.280 Patrick Trainer: the loader

357 00:26:34.830 00:26:39.160 Patrick Trainer: processor. And then, like our logging information. And so basically, what

358 00:26:39.470 00:26:42.540 Patrick Trainer: is in the secret is

359 00:26:42.910 00:26:49.219 Patrick Trainer: like, imagine this without with all the values removed from it. And so it’s just.

360 00:26:49.220 00:26:50.449 Uttam Kumaran: Oh, I see. Okay.

361 00:26:50.450 00:26:52.436 Patrick Trainer: Yeah, so it’s just these blank keys.

362 00:26:52.960 00:26:56.050 Patrick Trainer: And so that’s what this is doing.

363 00:26:56.240 00:27:04.209 Patrick Trainer: It’s using these said scripts, which is just like, if you’ve ever used, said. It’s the stream editor. It’s like a

364 00:27:04.610 00:27:06.420 Patrick Trainer: a bash thing.

365 00:27:06.540 00:27:09.079 Patrick Trainer: and it’s replacing.

366 00:27:09.400 00:27:12.399 Patrick Trainer: So it’s looking for the the service account file

367 00:27:12.580 00:27:15.796 Patrick Trainer: and then replacing it with,

368 00:27:16.730 00:27:17.470 Patrick Trainer: A,

369 00:27:18.170 00:27:20.850 Patrick Trainer: our stuff that we created up here.

370 00:27:21.482 00:27:23.729 Patrick Trainer: Same deal subject email

371 00:27:24.470 00:27:25.500 Patrick Trainer: skittin

372 00:27:26.210 00:27:28.257 Patrick Trainer: pulling from our secrets.

373 00:27:29.220 00:27:32.069 Patrick Trainer: replacing that like Max, results key

374 00:27:33.220 00:27:34.150 Patrick Trainer: with

375 00:27:34.260 00:27:35.150 Patrick Trainer: our

376 00:27:35.310 00:27:36.550 Patrick Trainer: inputs

377 00:27:37.200 00:27:40.420 Patrick Trainer: of Max results like from up here.

378 00:27:41.501 00:27:46.980 Patrick Trainer: And so it’s just like it’s writing all of our input variables into

379 00:27:47.900 00:27:51.010 Patrick Trainer: its own kind of like ephemeral config file

380 00:27:51.939 00:27:56.160 Patrick Trainer: and then, just for like sanity’s sake, it

381 00:27:56.170 00:27:58.660 Patrick Trainer: outputs the directories and whatnot

382 00:27:59.257 00:28:05.689 Patrick Trainer: or outputs like the contents of it, excluding like secrets of of how, of all of that works?

383 00:28:06.763 00:28:17.679 Patrick Trainer: And then this is where we’re coming through and actually like processing or calling like the the Gmad library and how we’re processing it. So this is that like.

384 00:28:17.720 00:28:23.529 Patrick Trainer: we’re calling the email processor with attachments. So here we’re getting the attachments

385 00:28:23.710 00:28:25.769 Patrick Trainer: here, we’re getting the email bodies.

386 00:28:25.800 00:28:29.180 Patrick Trainer: And then here we’re downloading the links.

387 00:28:30.800 00:28:33.169 Patrick Trainer: and then this was kind of like

388 00:28:33.520 00:28:35.100 Patrick Trainer: a hack, because it wouldn’t

389 00:28:35.230 00:28:37.109 Patrick Trainer: take in a

390 00:28:37.470 00:28:40.270 Patrick Trainer: allow for Colon. So have this

391 00:28:40.280 00:28:42.499 Patrick Trainer: ridiculous Regex to take out.

392 00:28:43.006 00:28:43.739 Patrick Trainer: Colt once.

393 00:28:45.600 00:28:51.690 Patrick Trainer: And so that just that just renames the file and then it uploads that artifact.

394 00:28:52.554 00:28:53.279 Patrick Trainer: And

395 00:28:54.160 00:28:55.550 Patrick Trainer: and it’s

396 00:28:55.830 00:29:02.099 Patrick Trainer: from their output directory and then outputs that artifacts

397 00:29:02.436 00:29:07.000 Patrick Trainer: run id. So then we can use like other actions to to grab it.

398 00:29:07.390 00:29:12.769 Patrick Trainer: So what this looks like in practice, we can run it from here, actually.

399 00:29:12.850 00:29:17.330 Patrick Trainer: And so if we come into the process email workflow.

400 00:29:17.797 00:29:21.842 Patrick Trainer: We have the dispatch event so we can run it ourselves.

401 00:29:22.210 00:29:22.890 Uttam Kumaran: Nice.

402 00:29:22.890 00:29:24.689 Patrick Trainer: We have these

403 00:29:25.650 00:29:31.050 Patrick Trainer: like values that we can do here. And so like, if we

404 00:29:31.680 00:29:35.820 Patrick Trainer: say like from Bootam, this one is from

405 00:29:37.390 00:29:38.170 Patrick Trainer: you.

406 00:29:40.250 00:29:42.570 Uttam Kumaran: I think that one might be from the pool parts. Email. Yeah.

407 00:29:42.570 00:29:45.059 Patrick Trainer: Yeah, so let’s actually do that.

408 00:29:46.900 00:29:48.549 Uttam Kumaran: And then this one isn’t.

409 00:29:48.660 00:29:50.700 Uttam Kumaran: Oh, there is an attachment on this one. Yeah.

410 00:29:50.820 00:29:51.470 Patrick Trainer: Yes!

411 00:29:55.210 00:29:55.989 Patrick Trainer: Did that.

412 00:29:57.240 00:29:59.999 Uttam Kumaran: I think you need to. Do you think you’ll need to do from?

413 00:30:00.700 00:30:01.700 Uttam Kumaran: Oh, yeah.

414 00:30:04.179 00:30:10.609 Patrick Trainer: And so we can run this workflow. It’s gonna inject those values into it.

415 00:30:12.360 00:30:14.540 Patrick Trainer: And then we could come through here.

416 00:30:15.860 00:30:17.399 Patrick Trainer: It’s gonna fire up!

417 00:30:21.400 00:30:23.490 Patrick Trainer: Go through this setup.

418 00:30:25.650 00:30:26.710 Patrick Trainer: dedicate!

419 00:30:28.600 00:30:31.549 Patrick Trainer: Here’s where we’re installing.

420 00:30:31.970 00:30:32.470 Uttam Kumaran: Yeah.

421 00:30:32.470 00:30:33.930 Patrick Trainer: The gmad stuff.

422 00:30:34.060 00:30:34.700 Uttam Kumaran: Yeah.

423 00:30:36.170 00:30:41.170 Nicolas Sucari: Hey? A question, Patrick, if you’re not logged in into your Gmail account.

424 00:30:41.390 00:30:46.840 Nicolas Sucari: is this like authenticating automatically into that? Or how is like accessing.

425 00:30:46.840 00:30:54.940 Patrick Trainer: Yeah. So this is using a a like a service account that’s tied to like the brain forge like at large. So it’s like, it’s not using

426 00:30:55.360 00:30:59.609 Patrick Trainer: our credentials or anything. It’s using a a service accounts.

427 00:30:59.610 00:31:00.290 Nicolas Sucari: Straight.

428 00:31:00.290 00:31:01.260 Patrick Trainer: All of their stuff.

429 00:31:02.210 00:31:11.460 Patrick Trainer: And so, as we can see here this is grab this attachment, this conversions list 2,024, 7, 15,

430 00:31:11.560 00:31:13.589 Patrick Trainer: which, if we look here.

431 00:31:14.647 00:31:15.750 Patrick Trainer: look at that.

432 00:31:16.030 00:31:16.410 Uttam Kumaran: No.

433 00:31:16.410 00:31:21.020 Patrick Trainer: That. What’s that called? Yeah, so that’s that’s that.

434 00:31:22.457 00:31:25.670 Patrick Trainer: And you got these cool little

435 00:31:25.870 00:31:34.669 Patrick Trainer: well bars and whatnot, and have like logging and all of these good that good information.

436 00:31:36.250 00:31:38.463 Patrick Trainer: this is where it’s like creating that

437 00:31:38.780 00:31:39.630 Uttam Kumaran: Content.

438 00:31:40.377 00:31:43.239 Patrick Trainer: Where it’s going. And then it’s

439 00:31:44.090 00:31:49.929 Patrick Trainer: finding this renaming it if it has like a weird name processes those emails

440 00:31:50.409 00:31:58.569 Patrick Trainer: looking for finding those links and it has renamed like the file like took out that colon there

441 00:31:58.610 00:32:01.630 Patrick Trainer: to this conversions list

442 00:32:01.930 00:32:03.180 Patrick Trainer: at Csv.

443 00:32:03.950 00:32:08.700 Patrick Trainer: And then it’s going to upload that process content

444 00:32:09.095 00:32:10.820 Patrick Trainer: which is then going to

445 00:32:11.170 00:32:16.280 Patrick Trainer: allow us to. It. Has that that artifact you have this artifact id

446 00:32:16.758 00:32:24.210 Patrick Trainer: and then everything that’s just like clean up. And so if we come back to the summary, we now have this artifact.

447 00:32:24.410 00:32:26.769 Patrick Trainer: it’s produced at Runtime.

448 00:32:27.090 00:32:29.490 Patrick Trainer: which let’s come up here.

449 00:32:33.000 00:32:34.380 Uttam Kumaran: Is that? Csv, yeah.

450 00:32:35.360 00:32:36.300 Patrick Trainer: And.

451 00:32:37.620 00:32:38.200 Uttam Kumaran: Cool.

452 00:32:38.490 00:32:39.590 Patrick Trainer: Boom.

453 00:32:40.480 00:32:41.050 Nicolas Sucari: Alright!

454 00:32:41.050 00:32:42.260 Patrick Trainer: There’s all the data.

455 00:32:43.740 00:32:45.280 Patrick Trainer: and so

456 00:32:45.390 00:32:49.449 Patrick Trainer: where we go from here is into

457 00:32:54.900 00:32:59.969 Patrick Trainer: into pool parts. This is what I’ve been working on this this morning

458 00:33:00.380 00:33:03.250 Patrick Trainer: we have this new workflow.

459 00:33:03.250 00:33:05.500 Uttam Kumaran: Okay, this is gonna, be my next question, yeah.

460 00:33:05.500 00:33:11.769 Patrick Trainer: Yeah, we have this new workflow, this trigger email processing and commit.

461 00:33:12.290 00:33:15.469 Patrick Trainer: So what this is doing. This has 2 jobs

462 00:33:16.210 00:33:17.470 Patrick Trainer: where it’s

463 00:33:18.680 00:33:19.670 Patrick Trainer: calls

464 00:33:20.170 00:33:22.420 Patrick Trainer: that processor workflow. That process.

465 00:33:22.420 00:33:22.750 Uttam Kumaran: I see.

466 00:33:22.750 00:33:25.930 Patrick Trainer: Emails job or workflow that I just showed

467 00:33:26.758 00:33:30.291 Patrick Trainer: and then these are it supplying

468 00:33:32.240 00:33:35.150 Patrick Trainer: the the input variables for it.

469 00:33:35.320 00:33:36.900 Patrick Trainer: And then

470 00:33:37.010 00:33:45.863 Patrick Trainer: that runs that and like lets it do its thing, creates the artifacts and then remember, it’s outputting

471 00:33:46.460 00:33:47.370 Patrick Trainer: the

472 00:33:47.830 00:33:51.349 Patrick Trainer: artifact id that we can use to then download

473 00:33:52.178 00:33:55.171 Patrick Trainer: what this comes in and does

474 00:33:56.780 00:34:00.179 Patrick Trainer: this is, we’re downloading our artifact.

475 00:34:00.984 00:34:03.650 Patrick Trainer: This is going to be changed.

476 00:34:03.650 00:34:04.230 Uttam Kumaran: So that okay.

477 00:34:04.587 00:34:05.660 Patrick Trainer: Like a holdover.

478 00:34:06.206 00:34:10.569 Patrick Trainer: But then this is what like needs email processing that output

479 00:34:10.860 00:34:15.530 Patrick Trainer: and the run id, which was what was output in the the

480 00:34:15.770 00:34:17.330 Patrick Trainer: last.

481 00:34:23.130 00:34:23.980 Patrick Trainer: There we go.

482 00:34:25.739 00:34:27.869 Patrick Trainer: which is like output in

483 00:34:28.730 00:34:31.539 Patrick Trainer: this. It’s like, so we have that.

484 00:34:31.940 00:34:33.723 Uttam Kumaran: Yeah, so you can.

485 00:34:34.170 00:34:39.329 Patrick Trainer: Yeah, so we can like grab the artifact like, cause, where is

486 00:34:40.955 00:34:42.710 Patrick Trainer: process content. Yeah.

487 00:34:42.850 00:34:46.630 Patrick Trainer: So like, this is the artifact. Id in this like

488 00:34:47.340 00:34:51.940 Patrick Trainer: outputs it. And this allows us to basically go to this.

489 00:34:52.290 00:34:56.630 Patrick Trainer: Yeah, URL to grab that from from here.

490 00:34:57.380 00:34:59.580 Patrick Trainer: And so it uses that

491 00:34:59.590 00:35:01.869 Patrick Trainer: to download the artifacts.

492 00:35:02.600 00:35:08.019 Patrick Trainer: And then this is kind of the the cool part of it, because we have

493 00:35:08.240 00:35:10.510 Patrick Trainer: this cool parts repo

494 00:35:10.570 00:35:15.750 Patrick Trainer: checked out in the context of our workflow or this action

495 00:35:16.460 00:35:21.090 Patrick Trainer: we’re going to use get to commit it back to itself.

496 00:35:21.560 00:35:24.130 Uttam Kumaran: Yeah, back to like you’re gonna put that in seed.

497 00:35:24.130 00:35:28.999 Patrick Trainer: Right? Right? Right? And so we are just setting up like

498 00:35:29.200 00:35:32.829 Patrick Trainer: to start get config. So we know what’s doing.

499 00:35:32.860 00:35:36.190 Patrick Trainer: It’s then going to add the processed content

500 00:35:36.320 00:35:37.689 Patrick Trainer: from this path.

501 00:35:37.810 00:35:40.140 Patrick Trainer: and we’re going to

502 00:35:40.350 00:35:42.200 Patrick Trainer: commit that

503 00:35:42.390 00:35:43.550 Patrick Trainer: to the repo

504 00:35:43.610 00:35:45.810 Patrick Trainer: and then push it.

505 00:35:46.270 00:35:51.270 Patrick Trainer: and then it’ll just be shown up in in here.

506 00:35:54.300 00:35:56.269 Patrick Trainer: let’s actually

507 00:35:56.650 00:35:58.890 Patrick Trainer: the last

508 00:35:59.420 00:36:02.310 Patrick Trainer: issue that I ran into.

509 00:36:03.163 00:36:05.376 Patrick Trainer: There’s a slight one

510 00:36:07.100 00:36:08.300 Patrick Trainer: Don’t think that’s

511 00:36:11.170 00:36:11.880 Patrick Trainer: wit.

512 00:36:19.870 00:36:20.890 Patrick Trainer: Where am I going?

513 00:36:21.280 00:36:23.259 Patrick Trainer: Let’s actually edit

514 00:36:24.010 00:36:24.970 Patrick Trainer: this

515 00:36:25.820 00:36:27.779 Patrick Trainer: so that we can

516 00:36:28.705 00:36:32.499 Patrick Trainer: just test it and see if we can do it. Live

517 00:36:34.231 00:36:38.349 Patrick Trainer: secrets inherent. So we can just edit this file directly

518 00:36:38.910 00:36:40.819 Patrick Trainer: we’ve got.

519 00:36:44.960 00:36:46.090 Patrick Trainer: I

520 00:36:49.000 00:36:50.160 Patrick Trainer: G mad

521 00:36:50.890 00:36:51.850 Patrick Trainer: email

522 00:36:53.030 00:36:54.150 Patrick Trainer: processor

523 00:36:55.400 00:36:56.900 Patrick Trainer: that.

524 00:36:57.000 00:37:00.069 Patrick Trainer: Okay? And then this is going to

525 00:37:01.600 00:37:03.010 Patrick Trainer: that. Let’s

526 00:37:03.080 00:37:04.399 Patrick Trainer: commit this.

527 00:37:05.290 00:37:06.440 Patrick Trainer: A,

528 00:37:08.020 00:37:10.190 Patrick Trainer: I’m just going to yellow it.

529 00:37:14.030 00:37:17.860 Patrick Trainer: So that’s going to probably kick off this damn

530 00:37:18.596 00:37:19.530 Patrick Trainer: light. Dash.

531 00:37:19.530 00:37:20.256 Uttam Kumaran: Yeah, okay.

532 00:37:20.620 00:37:21.520 Patrick Trainer: Alloy.

533 00:37:22.510 00:37:24.440 Patrick Trainer: Yeah, but that’s fine.

534 00:37:24.540 00:37:27.199 Patrick Trainer: And then we’re coming in here

535 00:37:28.060 00:37:29.339 Patrick Trainer: and let’s

536 00:37:30.280 00:37:32.010 Patrick Trainer: actually give this a shot

537 00:37:32.480 00:37:34.590 Patrick Trainer: of running from

538 00:37:36.480 00:37:37.580 Patrick Trainer: ports repo.

539 00:37:38.040 00:37:41.020 Patrick Trainer: And so you see, this is

540 00:37:41.310 00:37:46.220 Patrick Trainer: calling our process emails workflow. That’s from

541 00:37:47.105 00:37:47.820 Patrick Trainer: the

542 00:37:49.310 00:37:52.549 Patrick Trainer: oh, sorry from the the processor

543 00:37:55.660 00:37:57.520 Patrick Trainer: And so it’s installing

544 00:37:57.850 00:37:58.790 Patrick Trainer: a

545 00:37:59.410 00:38:00.860 Patrick Trainer: all that good stuff

546 00:38:08.520 00:38:10.590 Patrick Trainer: might error out on the config.

547 00:38:11.530 00:38:12.270 Patrick Trainer: Yeah.

548 00:38:14.230 00:38:22.060 Patrick Trainer: So one of the things that I’m having to do is because it’s like inheriting the secrets from this repo. I’m just having to

549 00:38:23.390 00:38:28.260 Patrick Trainer: add a couple of secrets there. And so this, this is the last step.

550 00:38:28.670 00:38:32.690 Patrick Trainer: But we know it all works because we can run it through our

551 00:38:34.250 00:38:35.690 Patrick Trainer: like the processor

552 00:38:35.960 00:38:42.039 Patrick Trainer: here, I’m just need to finish setting up this the secret. So we have like.

553 00:38:43.690 00:38:53.859 Patrick Trainer: not deploy keys. But these secrets and variables so in here, like we have like, it’s config, yaml, and service account. I just need to add these to the pool parts repo.

554 00:38:54.220 00:38:56.210 Patrick Trainer: and then we should be

555 00:38:56.930 00:38:58.760 Patrick Trainer: good to go.

556 00:38:58.860 00:39:01.220 Patrick Trainer: because it’s just it’s looking for.

557 00:39:04.030 00:39:04.360 Uttam Kumaran: Okay.

558 00:39:04.690 00:39:08.449 Patrick Trainer: Config file there, which it just doesn’t exist yet.

559 00:39:09.000 00:39:16.689 Uttam Kumaran: So what’s the? So the config file? That’s so you’re gonna have to have, we’re gonna basically have one per repo. And then that’s gonna trigger

560 00:39:16.890 00:39:18.580 Uttam Kumaran: this. And then

561 00:39:18.810 00:39:24.779 Uttam Kumaran: I guess once it’s in seed, I’ll kind of have Brian or Ryan take it from there. Basically.

562 00:39:24.820 00:39:27.400 Uttam Kumaran: they may have some questions about like.

563 00:39:29.019 00:39:34.930 Uttam Kumaran: yeah, I guess. Also, how do we know? How do we not reprocess the old ones.

564 00:39:34.930 00:39:35.830 Nicolas Sucari: Same files.

565 00:39:37.040 00:39:39.556 Patrick Trainer: Right? Right? Right? So

566 00:39:40.930 00:39:45.150 Patrick Trainer: what I’ve been thinking with with that is.

567 00:39:45.960 00:39:47.489 Patrick Trainer: we can either

568 00:39:48.420 00:39:54.699 Patrick Trainer: depending on like the file that we have, we can either name them all the same thing, and just like replace

569 00:39:54.820 00:39:55.910 Patrick Trainer: the file like.

570 00:39:55.910 00:39:56.250 Uttam Kumaran: Yeah.

571 00:39:56.609 00:39:56.969 Patrick Trainer: One.

572 00:39:58.630 00:40:00.877 Patrick Trainer: or we can do

573 00:40:01.590 00:40:02.610 Patrick Trainer: some other

574 00:40:03.150 00:40:05.010 Patrick Trainer: sort of funky stuff.

575 00:40:07.830 00:40:13.011 Patrick Trainer: I think the replacing the name of the file makes makes sense.

576 00:40:13.410 00:40:14.680 Nicolas Sucari: Replacing.

577 00:40:14.940 00:40:18.439 Nicolas Sucari: replacing the name of a file. I mean every time you run.

578 00:40:18.440 00:40:19.759 Patrick Trainer: File, it sends, yeah.

579 00:40:20.000 00:40:25.689 Nicolas Sucari: Yeah, okay. But if if you replace the file every time you run these these jobs are you not gonna be like.

580 00:40:25.740 00:40:35.609 Nicolas Sucari: make it, work again, download every every file, and replace it. Is it not possible just to ignore? If you already have, like the same name, file there.

581 00:40:37.030 00:40:42.479 Patrick Trainer: pro, probably. Yeah, pro. Probably we just just hadn’t gotten there yet.

582 00:40:42.480 00:40:46.240 Uttam Kumaran: Yeah, I I don’t have like a clear idea of like what to do yet.

583 00:40:46.430 00:40:46.950 Patrick Trainer: Yeah.

584 00:40:48.100 00:40:48.810 Uttam Kumaran: But.

585 00:40:48.810 00:40:50.010 Nicolas Sucari: Okay, like.

586 00:40:50.010 00:40:53.850 Patrick Trainer: Right now, like, that’s kinda like my plan, and where I’m headed

587 00:40:55.430 00:40:57.160 Patrick Trainer: that could change

588 00:40:57.886 00:41:00.870 Patrick Trainer: and we could down the line like.

589 00:41:01.840 00:41:05.300 Patrick Trainer: think like maybe seeds isn’t the best way to do it.

590 00:41:07.110 00:41:09.559 Patrick Trainer: and if that’s the case,

591 00:41:10.350 00:41:12.979 Patrick Trainer: then we can either like.

592 00:41:12.980 00:41:14.770 Uttam Kumaran: Send that file somewhere. Yeah, we can just send.

593 00:41:14.770 00:41:17.189 Patrick Trainer: Send that file somewhere else, or like

594 00:41:17.877 00:41:20.660 Patrick Trainer: pipe it in to like the snowflake

595 00:41:20.670 00:41:23.620 Patrick Trainer: cli, or something like that, and

596 00:41:23.680 00:41:25.539 Patrick Trainer: connect to Snowflake that way?

597 00:41:26.450 00:41:27.300 Patrick Trainer: Or

598 00:41:28.330 00:41:30.740 Patrick Trainer: yeah, or send it somewhere else.

599 00:41:30.740 00:41:36.770 Uttam Kumaran: I think having in the repo is good. I just don’t cause we only have. We probably like 5 workflows.

600 00:41:36.870 00:41:46.449 Uttam Kumaran: not sure exactly. Off the top of my head like how those files need to be named for them to come in to see cause they have, like long time stamps or stuff like.

601 00:41:46.450 00:41:47.200 Patrick Trainer: Right.

602 00:41:47.510 00:41:51.729 Uttam Kumaran: I don’t know how. Dbt! I guess we could write a little macro to automatically pick up

603 00:41:52.670 00:41:54.950 Uttam Kumaran: like anything and seed

604 00:41:58.462 00:42:03.947 Uttam Kumaran: but I’ll leave that to. Yeah. I’ll leave that to Brian Ryan, basically to figure out.

605 00:42:04.970 00:42:06.980 Patrick Trainer: Yeah. And so

606 00:42:07.290 00:42:10.560 Patrick Trainer: yeah, this like, I’d say, we’re at like

607 00:42:10.790 00:42:15.529 Patrick Trainer: 95%. I just need to add this config.

608 00:42:16.154 00:42:17.519 Patrick Trainer: To the secret.

609 00:42:17.940 00:42:19.586 Patrick Trainer: And then

610 00:42:20.840 00:42:24.170 Patrick Trainer: and then we’ll be playing around with the

611 00:42:24.210 00:42:25.450 Patrick Trainer: seed files.

612 00:42:25.740 00:42:26.330 Uttam Kumaran: Okay.

613 00:42:27.280 00:42:33.939 Uttam Kumaran: okay, cool. Yeah. I guess once you’re done with the seed file stuff, I think, basically, we can hand it off to

614 00:42:34.638 00:42:37.670 Uttam Kumaran: Vp and Brian to figure out from there

615 00:42:37.920 00:42:45.699 Uttam Kumaran: cool and then ideally again, like, we’re not using this for Stella. But I think we’re going to be using this, probably for any client that needs to send in

616 00:42:45.960 00:42:47.599 Uttam Kumaran: emails. And we can’t

617 00:42:47.670 00:42:51.250 Uttam Kumaran: process another way. So it’s a sec.

618 00:42:51.890 00:42:53.219 Patrick Trainer: Yeah, yeah, yeah.

619 00:42:56.692 00:43:01.229 Uttam Kumaran: Cool, and then we have this recording, so I’ll save this the notion, and then

620 00:43:01.490 00:43:02.750 Uttam Kumaran: probably have

621 00:43:04.170 00:43:07.230 Uttam Kumaran: probably have AI come up with little summary, and then

622 00:43:07.725 00:43:16.290 Uttam Kumaran: I I guess the last thing is just the finish up, whatever those. So even on Friday we can meet with, or or Monday, we can meet with the

623 00:43:16.680 00:43:26.819 Uttam Kumaran: rest of the team. And basically say, like, these seats are not coming in and then let’s just make sure we can. Basically the tickets. Nico, that will be is to deprecate

624 00:43:26.880 00:43:28.939 Uttam Kumaran: the Google sheets

625 00:43:29.842 00:43:32.510 Uttam Kumaran: for those and transition over here. So.

626 00:43:33.990 00:43:40.099 Nicolas Sucari: Yeah, I think we need to to see how this starts working. And also we can remove

627 00:43:40.170 00:43:44.240 Nicolas Sucari: what we are receiving through slack channels. Probably, too, if this is working.

628 00:43:44.300 00:43:45.720 Nicolas Sucari: Yeah.

629 00:43:46.270 00:43:48.294 Nicolas Sucari: my only question, Patrick, is,

630 00:43:48.800 00:43:53.639 Nicolas Sucari: Can we schedule this to be like to trigger itself every week?

631 00:43:54.700 00:43:57.109 Nicolas Sucari: Or it’s something that we need to do manually.

632 00:43:57.350 00:44:04.149 Patrick Trainer: No so like if we you’re not looking at my screen, but like just how we’re

633 00:44:04.390 00:44:15.910 Patrick Trainer: clicking that dispatch event like by ourselves. We can just set it like on a cron schedule, and it can. We can set it whenever we want it to run.

634 00:44:15.930 00:44:17.800 Patrick Trainer: or we can

635 00:44:18.270 00:44:25.560 Patrick Trainer: have it run after like triggered by something else. Any of the Github events can trigger this.

636 00:44:26.370 00:44:27.240 Nicolas Sucari: It’s great.

637 00:44:27.560 00:44:30.219 Nicolas Sucari: Yeah, it will reduce a lot of manual work.

638 00:44:31.160 00:44:32.010 Patrick Trainer: Hopefully, so.

639 00:44:33.890 00:44:34.510 Nicolas Sucari: Yep.

640 00:44:34.960 00:44:35.740 Nicolas Sucari: great.

641 00:44:37.250 00:44:47.450 Uttam Kumaran: Okay, cool. I guess. Yeah, let’s close this out this week. And then I think ideally, we can. We can have them next week work when Ryan’s back basically work on the migration

642 00:44:50.170 00:44:53.900 Uttam Kumaran: and then that should like kinda probably kick off a couple of more things.

643 00:44:54.493 00:45:07.580 Uttam Kumaran: And then the rest of this week. I think tomorrow I think we’ll we’ll close up the our back stuff, and then we also are having some issues with like the mart Dbt mart rename.

644 00:45:07.660 00:45:10.239 Uttam Kumaran: And I, basically, I basically like.

645 00:45:10.440 00:45:15.580 Uttam Kumaran: just want to know what we decided on and then make sure that that’s happening because there’s a bunch of duplicate tables.

646 00:45:15.760 00:45:17.610 Uttam Kumaran: So tomorrow, also.

647 00:45:17.960 00:45:22.293 Uttam Kumaran: maybe tomorrow, we can just do like a whole bunch of tech that stuff when we’re on the call together basically.

648 00:45:22.500 00:45:23.740 Patrick Trainer: That sounds good to me.

649 00:45:23.740 00:45:26.219 Nicolas Sucari: Yeah, I just, I just sent you a message on that one.

650 00:45:27.780 00:45:35.119 Nicolas Sucari: because we just had an issue. The the team’s weekly report issue. That it’s not updating is because of that. It was pointing at the wrong schema.

651 00:45:35.690 00:45:38.980 Uttam Kumaran: Okay, yeah, I just wanna know whether we removed.

652 00:45:39.210 00:45:42.591 Uttam Kumaran: I’m gonna ask Bryce, but better whether we removed the

653 00:45:43.440 00:45:44.170 Uttam Kumaran: the DVD.

654 00:45:44.170 00:45:45.050 Nicolas Sucari: The end. There.

655 00:45:45.460 00:45:47.401 Uttam Kumaran: The and then what?

656 00:45:48.240 00:45:55.260 Uttam Kumaran: basically. What? If you can just write down like what it is right now and like how it works basically before that meeting.

657 00:45:56.460 00:45:58.129 Nicolas Sucari: Yeah, that would be great.

658 00:45:58.130 00:46:13.501 Uttam Kumaran: Okay, said, Yeah, dude, this is great. I mean, like again, this, this takes care of like us having to go through 5 trend. And basically, now for every client we’re just gonna we’ll just be able to create service accounts or anything that comes in this way. We’ll just say, Send us this way

659 00:46:14.050 00:46:14.800 Patrick Trainer: Yeah.

660 00:46:15.110 00:46:16.030 Uttam Kumaran: Yeah, it’s like.

661 00:46:16.930 00:46:17.870 Patrick Trainer: Sweet.

662 00:46:17.870 00:46:18.550 Nicolas Sucari: Ice.

663 00:46:19.540 00:46:22.570 Uttam Kumaran: Okay. Cool. Alright.

664 00:46:22.570 00:46:23.380 Patrick Trainer: Some coffee.

665 00:46:23.590 00:46:25.662 Uttam Kumaran: Yeah. Get some coffee.

666 00:46:27.280 00:46:30.100 Uttam Kumaran: Cool. Okay? Alright. Alright guys, thank you.

667 00:46:30.100 00:46:31.890 Nicolas Sucari: Thank you very much, guys. Bye, bye.