Meeting Title: Pool-Parts-DE-Download-meeting Date: 2024-07-12 Meeting participants: Bryce Codell, Nicolas Sucari, Patrick Trainer
WEBVTT
1 00:01:20.940 ⇒ 00:01:22.079 Bryce Codell: What’s up, Patrick.
2 00:01:26.830 ⇒ 00:01:28.219 Patrick Trainer: Hey? How’s it going.
3 00:01:28.920 ⇒ 00:01:30.460 Bryce Codell: Not too bad, man. How are you?
4 00:01:30.720 ⇒ 00:01:36.832 Patrick Trainer: Too bad but went to the gym, and now I’m trying to stop sweating.
5 00:01:37.723 ⇒ 00:01:38.690 Bryce Codell: It’s like.
6 00:01:38.960 ⇒ 00:01:39.500 Bryce Codell: but.
7 00:01:39.500 ⇒ 00:01:44.180 Patrick Trainer: What it’s 10, and just about a hundred degrees right now.
8 00:01:44.930 ⇒ 00:01:45.520 Bryce Codell: Yeah.
9 00:01:45.520 ⇒ 00:01:47.459 Patrick Trainer: It’s Louisiana life.
10 00:01:47.460 ⇒ 00:01:48.610 Bryce Codell: Yeah, sure, yeah.
11 00:01:48.610 ⇒ 00:01:49.559 Patrick Trainer: Are you doing?
12 00:01:50.480 ⇒ 00:01:54.109 Bryce Codell: Oh, doing? Alright lot of life changes coming up now.
13 00:01:54.110 ⇒ 00:01:56.919 Patrick Trainer: Yeah, yeah. Congrats on the engagement man.
14 00:01:56.920 ⇒ 00:01:58.865 Bryce Codell: Thank you, man, I appreciate it.
15 00:01:59.190 ⇒ 00:02:03.340 Patrick Trainer: Awesome. And the I actually. So I was on
16 00:02:03.430 ⇒ 00:02:09.940 Patrick Trainer: scrolling through Linkedin last night. And I I saw data engineer position with Ashby.
17 00:02:10.899 ⇒ 00:02:12.099 Patrick Trainer: And I’ve
18 00:02:12.633 ⇒ 00:02:17.286 Patrick Trainer: I don’t. I don’t. I? I was gonna text it to you. But yeah, I don’t know.
19 00:02:17.560 ⇒ 00:02:22.579 Bryce Codell: Yeah, are you? Are you? I? You’re working full time for brain forward. Yeah.
20 00:02:22.580 ⇒ 00:02:26.499 Patrick Trainer: Pretty much. Yeah, yeah, yeah. But no, I just I just thought it was funny that it was.
21 00:02:26.870 ⇒ 00:02:27.940 Patrick Trainer: Yeah, he.
22 00:02:28.000 ⇒ 00:02:32.129 Patrick Trainer: you said you’re leaving. And then they’re like, Oh, yeah, there there we go.
23 00:02:32.420 ⇒ 00:02:34.110 Bryce Codell: That is my backfill.
24 00:02:34.210 ⇒ 00:02:35.230 Bryce Codell: them that.
25 00:02:35.230 ⇒ 00:02:37.529 Patrick Trainer: That was like you were the only.
26 00:02:38.610 ⇒ 00:02:39.050 Nicolas Sucari: You guys.
27 00:02:39.050 ⇒ 00:02:39.709 Patrick Trainer: Or Daddy.
28 00:02:39.710 ⇒ 00:02:40.030 Bryce Codell: Yeah.
29 00:02:40.030 ⇒ 00:02:41.130 Patrick Trainer: Hey? Nico, huh!
30 00:02:41.130 ⇒ 00:02:42.149 Bryce Codell: Todd Nico.
31 00:02:43.070 ⇒ 00:02:44.260 Bryce Codell: Why, you.
32 00:02:44.310 ⇒ 00:02:49.748 Bryce Codell: I was the only engineer there, or the only data engineer for the last, like 2 and a half years
33 00:02:50.210 ⇒ 00:02:51.079 Patrick Trainer: It’s a lot.
34 00:02:51.270 ⇒ 00:02:52.140 Bryce Codell: Yeah, yeah.
35 00:02:52.140 ⇒ 00:02:52.960 Patrick Trainer: Whole lot.
36 00:02:52.960 ⇒ 00:02:56.819 Bryce Codell: We were like 30 when I joined. We’re like 110 now.
37 00:02:57.160 ⇒ 00:03:00.249 Patrick Trainer: They run like super lean, too, from what I understand.
38 00:03:00.250 ⇒ 00:03:08.468 Bryce Codell: Yeah, yeah, we do. But now they are hiring a data engineer to backfill me. And they’re hiring a dedicated analyst.
39 00:03:09.170 ⇒ 00:03:12.039 Bryce Codell: so basically splitting up my role into 2.
40 00:03:13.020 ⇒ 00:03:13.930 Bryce Codell: So.
41 00:03:14.390 ⇒ 00:03:16.300 Patrick Trainer: You know you know what you’re worth now.
42 00:03:16.300 ⇒ 00:03:17.240 Bryce Codell: Yeah, right?
43 00:03:17.590 ⇒ 00:03:20.060 Bryce Codell: Yeah, yeah. The times they are a change.
44 00:03:20.060 ⇒ 00:03:22.710 Patrick Trainer: Yeah. No kidding, no kidding.
45 00:03:24.080 ⇒ 00:03:24.960 Bryce Codell: Nan.
46 00:03:25.510 ⇒ 00:03:32.400 Bryce Codell: anyway. So Patrick, I was hoping we could do kind of like a high level walkthrough of
47 00:03:33.244 ⇒ 00:03:44.470 Bryce Codell: the data engineering like world and infrastructure world in brain forge. I’ve worked a little bit at some of the Github actions and workflows that you’ve set up
48 00:03:44.480 ⇒ 00:03:48.870 Bryce Codell: in the pool parts repo. There are a lot. It is very.
49 00:03:48.870 ⇒ 00:03:56.559 Patrick Trainer: Yeah, yeah, there, there, there’s a lot I didn’t. There were a lot when I got here, and I’m trying to.
50 00:03:58.120 ⇒ 00:04:08.859 Patrick Trainer: like my goal is to make them not a lot but the overall idea is to be as like modular, modular, and componentized.
51 00:04:08.930 ⇒ 00:04:11.420 Patrick Trainer: That’s even a word component like as possible.
52 00:04:11.420 ⇒ 00:04:12.150 Bryce Codell: Month.
53 00:04:12.970 ⇒ 00:04:13.970 Patrick Trainer: And so
54 00:04:14.540 ⇒ 00:04:37.799 Patrick Trainer: what I’ve been building recently is a lot of like custom actions, and then, like reusable reusable workflows that call those custom actions, and so that we’re able to then like, compose and stack on top of each other and like kind of have like the nuts and bolts already there and then. All we have to do is like
55 00:04:38.280 ⇒ 00:04:41.979 Patrick Trainer: screw those in like Ikea furniture. Essentially
56 00:04:42.797 ⇒ 00:04:49.611 Patrick Trainer: and so I do have. I guess I’d be helpful if I like. Shared my screen, too.
57 00:04:50.280 ⇒ 00:04:53.520 Patrick Trainer: wouldn’t it? So let’s go
58 00:04:54.870 ⇒ 00:04:55.920 Patrick Trainer: here.
59 00:04:57.210 ⇒ 00:04:58.416 Patrick Trainer: Okay, so
60 00:04:59.370 ⇒ 00:05:00.080 Patrick Trainer: just
61 00:05:00.260 ⇒ 00:05:02.120 Patrick Trainer: brain Forge there.
62 00:05:02.820 ⇒ 00:05:06.380 Patrick Trainer: Better remember where I have these
63 00:05:07.060 ⇒ 00:05:11.640 Patrick Trainer: thoughts. So I guess what we can do on on like. The 1st
64 00:05:14.030 ⇒ 00:05:18.029 Patrick Trainer: kind of like idea is of the
65 00:05:18.790 ⇒ 00:05:21.680 Patrick Trainer: kind of like our Ci pr.
66 00:05:23.470 ⇒ 00:05:29.660 Patrick Trainer: promoting from like our our dev to prod workflow of like when
67 00:05:29.780 ⇒ 00:05:34.900 Patrick Trainer: we’re developing. And then we push the main, and how dpt runs off of that.
68 00:05:35.230 ⇒ 00:05:36.670 Patrick Trainer: And so
69 00:05:37.320 ⇒ 00:05:42.396 Patrick Trainer: what we have is essentially like an environment where
70 00:05:43.260 ⇒ 00:05:44.850 Patrick Trainer: kind of like testing
71 00:05:44.870 ⇒ 00:05:49.899 Patrick Trainer: ground, where whenever you open a Pr
72 00:05:49.910 ⇒ 00:05:56.419 Patrick Trainer: that’s going to trigger a workflow that’s going to essentially like, run dbt.
73 00:05:56.806 ⇒ 00:06:06.080 Patrick Trainer: but what it’s going to do is it’s kind of like Dbts it’s it’s like that notion of slim Ci that so that instead of
74 00:06:06.950 ⇒ 00:06:11.079 Patrick Trainer: running all of the models, we’re only going to run the models that have changed.
75 00:06:11.190 ⇒ 00:06:18.910 Patrick Trainer: And so the way that we’re leveraging. This is so. We have a Dbt running like on a cron job every few hours or so.
76 00:06:18.920 ⇒ 00:06:28.279 Patrick Trainer: and in that it’s creating this artifact. This the the manifest essentially. And so we’re storing that that that manifest.
77 00:06:28.430 ⇒ 00:06:38.749 Patrick Trainer: And that’s really the crux of how our Ci flow works. And so if you kind of like. Follow it here like you’re the developer. You open the Pr
78 00:06:39.250 ⇒ 00:06:40.819 Patrick Trainer: opening that Pr
79 00:06:40.860 ⇒ 00:06:45.710 Patrick Trainer: or committing to that Pr is going to trigger the workflow.
80 00:06:46.060 ⇒ 00:06:50.409 Patrick Trainer: and that workflow runner is going to hit the artifact storage.
81 00:06:50.420 ⇒ 00:06:51.470 Patrick Trainer: Grab that.
82 00:06:52.100 ⇒ 00:06:54.830 Patrick Trainer: and then it’s going to take that diff.
83 00:06:55.060 ⇒ 00:06:56.300 Patrick Trainer: And then
84 00:06:56.540 ⇒ 00:07:03.829 Patrick Trainer: Dbt. Bron is going to do its thing that’s gonna use that manifest to determine what to run.
85 00:07:04.746 ⇒ 00:07:07.869 Patrick Trainer: Checks are going to go, and things look good.
86 00:07:09.330 ⇒ 00:07:13.555 Patrick Trainer: Then you’ll be able to merge the Pr. When the Pr.
87 00:07:14.310 ⇒ 00:07:19.109 Patrick Trainer: gets merged into the main branch. That’s going to kind of do the same thing.
88 00:07:19.290 ⇒ 00:07:35.260 Patrick Trainer: Trigger the workflow again. But in the prod environment. So essentially, we’re just like switching targets. It’s is really what’s happening. And I’ll and I’ll get into the kind of like the components of of where we’re going with this. But that’s kind of like the high level
89 00:07:36.030 ⇒ 00:07:41.749 Patrick Trainer: and I guess kind of like going back a bit. It’s we are
90 00:07:42.600 ⇒ 00:07:55.306 Patrick Trainer: managing dbt, fully using Github actions. It’s like, of course you have, like your local development and all that. But in terms of like Ci and pro runs
91 00:07:55.900 ⇒ 00:07:57.360 Patrick Trainer: Github is
92 00:07:57.370 ⇒ 00:08:12.160 Patrick Trainer: absolutely killing it and past role we use get lab, very similar idea, their workflow syntax, and all of that. It’s a little nicer and cleaner but all in all, it’s it’s it’s the same deal.
93 00:08:13.533 ⇒ 00:08:14.980 Patrick Trainer: And so.
94 00:08:15.530 ⇒ 00:08:16.530 Bryce Codell: Can I ask her a quick question.
95 00:08:16.530 ⇒ 00:08:17.769 Patrick Trainer: Yeah, yeah, yeah, yeah, please.
96 00:08:17.770 ⇒ 00:08:21.542 Bryce Codell: What are you using for artifact storage? Does Github offer.
97 00:08:21.920 ⇒ 00:08:26.629 Patrick Trainer: So yeah. So whenever let’s go
98 00:08:26.820 ⇒ 00:08:29.070 Patrick Trainer: back, whenever
99 00:08:29.690 ⇒ 00:08:31.070 Patrick Trainer: we run
100 00:08:32.539 ⇒ 00:08:34.559 Patrick Trainer: like one of these actions.
101 00:08:40.010 ⇒ 00:08:42.029 Patrick Trainer: it’s going to
102 00:08:43.059 ⇒ 00:08:44.059 Patrick Trainer: actually
103 00:08:47.000 ⇒ 00:08:48.219 Patrick Trainer: build on Pr.
104 00:08:51.100 ⇒ 00:08:51.800 Patrick Trainer: Bull
105 00:08:52.190 ⇒ 00:08:58.150 Patrick Trainer: Github does have its own artifact storage here. I just there’s so many I gotta find this
106 00:09:12.730 ⇒ 00:09:18.609 Patrick Trainer: I’ll find it. But so it’s it’s essentially it’s actually I can just show
107 00:09:18.800 ⇒ 00:09:22.050 Patrick Trainer: the code that makes a lot more sense.
108 00:09:25.340 ⇒ 00:09:28.910 Patrick Trainer: so I’ll start at kind of like
109 00:09:29.050 ⇒ 00:09:31.709 Patrick Trainer: the base, like component
110 00:09:31.780 ⇒ 00:09:34.429 Patrick Trainer: layers. So we have these like custom
111 00:09:34.470 ⇒ 00:09:38.690 Patrick Trainer: actions right? And so we’ve got our
112 00:09:40.860 ⇒ 00:09:42.620 Patrick Trainer: our setup action.
113 00:09:42.870 ⇒ 00:09:43.350 Bryce Codell: Hmm.
114 00:09:43.943 ⇒ 00:09:51.400 Patrick Trainer: Which is like a composite action we’re coming through. We’re installing Dbt.
115 00:09:51.680 ⇒ 00:09:54.300 Patrick Trainer: verifying that that it all works out.
116 00:09:54.940 ⇒ 00:09:58.030 Patrick Trainer: installing those dependencies.
117 00:09:58.200 ⇒ 00:10:02.040 Patrick Trainer: Caching Dvt packages. And then
118 00:10:02.080 ⇒ 00:10:07.249 Patrick Trainer: this is just kind of like some debugging information that’s gonna print to standard out
119 00:10:08.175 ⇒ 00:10:14.360 Patrick Trainer: and then. So we have. Next component is our dbt, run component.
120 00:10:15.198 ⇒ 00:10:18.069 Patrick Trainer: All of these take these inputs
121 00:10:18.651 ⇒ 00:10:22.358 Patrick Trainer: which are little like up higher in the stack
122 00:10:23.280 ⇒ 00:10:27.229 Patrick Trainer: Reminder, like we’re at the very bottom. But as you can see
123 00:10:27.270 ⇒ 00:10:35.649 Patrick Trainer: that all this is is just like one command, and then up the stack. Where that’s when we’re
124 00:10:36.370 ⇒ 00:10:39.320 Patrick Trainer: kind of putting the Legos altogether.
125 00:10:39.320 ⇒ 00:10:40.740 Bryce Codell: Yep. Makes sense.
126 00:10:41.225 ⇒ 00:10:49.000 Patrick Trainer: And so we have this, the Dbt run action as well as the Dbt test action.
127 00:10:49.572 ⇒ 00:10:52.390 Patrick Trainer: And that’s what that’s all doing here.
128 00:10:52.730 ⇒ 00:10:53.960 Patrick Trainer: From
129 00:10:54.030 ⇒ 00:10:55.220 Patrick Trainer: there
130 00:10:55.250 ⇒ 00:10:58.440 Patrick Trainer: we have our workflows.
131 00:10:58.520 ⇒ 00:11:00.349 Patrick Trainer: and so
132 00:11:00.430 ⇒ 00:11:10.980 Patrick Trainer: if we come into. So we have kind of like 2. We have like the reusable right now it’s poorly named, but the reusable workflow.
133 00:11:11.912 ⇒ 00:11:16.939 Patrick Trainer: And what this is is essentially like. This is the
134 00:11:19.330 ⇒ 00:11:29.970 Patrick Trainer: like the the, the main calling workflow, that where all of your tasks are are going to be defined right? So like we have our
135 00:11:30.320 ⇒ 00:11:31.590 Patrick Trainer: Dbt run
136 00:11:31.770 ⇒ 00:11:33.500 Patrick Trainer: our dbt test.
137 00:11:34.119 ⇒ 00:11:38.669 Patrick Trainer: And then this is where kind of all of our like secrets
138 00:11:39.190 ⇒ 00:11:41.850 Patrick Trainer: are handled as well as like.
139 00:11:44.130 ⇒ 00:11:52.819 Patrick Trainer: Where all of our different inputs from our calling workflow our calling workflow is this main workflow here.
140 00:11:53.210 ⇒ 00:11:57.130 Patrick Trainer: And this is like the kind of like, the
141 00:11:57.140 ⇒ 00:12:05.297 Patrick Trainer: the highest level of where you’re actually like inputting like data, right? Instead of
142 00:12:06.320 ⇒ 00:12:08.460 Patrick Trainer: like, this is like config.
143 00:12:08.510 ⇒ 00:12:10.692 Patrick Trainer: So in here,
144 00:12:11.840 ⇒ 00:12:18.420 Patrick Trainer: and in in here is where you’re actually defining like your inputs for the run
145 00:12:18.975 ⇒ 00:12:22.230 Patrick Trainer: where your environment. Variables are being defined.
146 00:12:22.370 ⇒ 00:12:25.250 Patrick Trainer: And then, like the one job
147 00:12:25.410 ⇒ 00:12:28.069 Patrick Trainer: that you’re that you’re calling from here.
148 00:12:28.070 ⇒ 00:12:28.500 Bryce Codell: Yep.
149 00:12:28.981 ⇒ 00:12:35.719 Patrick Trainer: Is that reusable workflow? So the idea is like, if you if we needed
150 00:12:35.730 ⇒ 00:12:36.750 Patrick Trainer: of
151 00:12:37.310 ⇒ 00:12:47.010 Patrick Trainer: to run like things on a different Cron, or to a different target, or something using a different profile.
152 00:12:47.010 ⇒ 00:12:47.490 Bryce Codell: Yeah.
153 00:12:47.490 ⇒ 00:12:49.769 Patrick Trainer: The idea is that we can just copy
154 00:12:49.890 ⇒ 00:12:50.820 Patrick Trainer: this.
155 00:12:51.110 ⇒ 00:12:53.410 Patrick Trainer: and it’s going to
156 00:12:53.950 ⇒ 00:12:57.659 Patrick Trainer: push everything down to this reusable workflow.
157 00:12:57.660 ⇒ 00:12:58.070 Bryce Codell: Yep.
158 00:12:58.070 ⇒ 00:13:05.789 Patrick Trainer: And then everything else is just going to run kind of like Dbt in that in those normal steps. So this is kind of like the the
159 00:13:06.090 ⇒ 00:13:10.160 Patrick Trainer: like, I said, kind of like the config of like where your data is going to go.
160 00:13:10.400 ⇒ 00:13:10.980 Bryce Codell: Yet.
161 00:13:12.890 ⇒ 00:13:14.390 Patrick Trainer: a.
162 00:13:14.400 ⇒ 00:13:15.640 Patrick Trainer: And
163 00:13:17.050 ⇒ 00:13:20.340 Patrick Trainer: and let’s say, workflow dots.
164 00:13:21.080 ⇒ 00:13:22.469 Patrick Trainer: This is a month.
165 00:13:24.550 ⇒ 00:13:34.420 Bryce Codell: So I get that, like the manifest gets emitted when Dbt run is invoked. But where’s the step? Where you’re actually like putting it somewhere.
166 00:13:35.080 ⇒ 00:13:36.350 Patrick Trainer: So
167 00:13:36.690 ⇒ 00:13:38.110 Patrick Trainer: we have.
168 00:13:38.110 ⇒ 00:13:41.630 Bryce Codell: Or fetching it for use.
169 00:13:42.370 ⇒ 00:13:43.020 Patrick Trainer: Right.
170 00:13:46.830 ⇒ 00:13:48.319 Patrick Trainer: Dt burn with artifact.
171 00:13:49.190 ⇒ 00:13:52.249 Patrick Trainer: So this is here with the artifact.
172 00:13:52.460 ⇒ 00:13:59.859 Patrick Trainer: So the way we’re actually calling this and grabbing that artifact is using the Github
173 00:14:00.150 ⇒ 00:14:01.300 Patrick Trainer: command line.
174 00:14:01.310 ⇒ 00:14:01.860 Bryce Codell: -
175 00:14:01.860 ⇒ 00:14:04.890 Patrick Trainer: And so you’re able to
176 00:14:05.170 ⇒ 00:14:11.949 Patrick Trainer: authenticate in the runner. Has the environment variables already set up for. That’s there.
177 00:14:12.100 ⇒ 00:14:15.370 Patrick Trainer: And then what this is doing. It’s just calling
178 00:14:15.620 ⇒ 00:14:19.240 Patrick Trainer: the Github Api with our repo
179 00:14:20.380 ⇒ 00:14:23.300 Patrick Trainer: hitting that artifacts endpoint
180 00:14:24.208 ⇒ 00:14:26.450 Patrick Trainer: and then it’s we’re just
181 00:14:26.570 ⇒ 00:14:29.450 Patrick Trainer: grabbing the stuff from Jq
182 00:14:30.061 ⇒ 00:14:33.510 Patrick Trainer: of the artifact on the most recent branch.
183 00:14:34.407 ⇒ 00:14:45.570 Patrick Trainer: For the most recent run on the main branch. It’s grabbing the manifest from that run. And then that’s what’s being diffed
184 00:14:46.274 ⇒ 00:14:50.140 Patrick Trainer: to know? Like, what’s what’s changed
185 00:14:51.670 ⇒ 00:14:54.770 Patrick Trainer: most recently, right? And so.
186 00:14:56.370 ⇒ 00:14:57.550 Patrick Trainer: And here
187 00:14:57.800 ⇒ 00:15:01.929 Patrick Trainer: we have like, this is where we’re getting the
188 00:15:01.950 ⇒ 00:15:06.969 Patrick Trainer: run like the workflow id that that posted the manifest.
189 00:15:07.220 ⇒ 00:15:09.010 Patrick Trainer: And then we’re using
190 00:15:09.830 ⇒ 00:15:11.650 Patrick Trainer: that artifact id
191 00:15:12.190 ⇒ 00:15:15.140 Patrick Trainer: down here. And so this is where
192 00:15:15.280 ⇒ 00:15:16.780 Patrick Trainer: we’re then
193 00:15:17.700 ⇒ 00:15:19.960 Patrick Trainer: actually downloading the artifact
194 00:15:20.380 ⇒ 00:15:21.540 Patrick Trainer: using that
195 00:15:21.770 ⇒ 00:15:23.280 Patrick Trainer: workflow id.
196 00:15:23.430 ⇒ 00:15:23.960 Bryce Codell: -
197 00:15:24.356 ⇒ 00:15:25.150 Patrick Trainer: And then
198 00:15:26.080 ⇒ 00:15:29.519 Patrick Trainer: down here is where we are
199 00:15:31.020 ⇒ 00:15:33.429 Patrick Trainer: actually running dbt.
200 00:15:33.920 ⇒ 00:15:34.820 Patrick Trainer: with.
201 00:15:35.470 ⇒ 00:15:37.320 Patrick Trainer: like those state
202 00:15:37.480 ⇒ 00:15:38.480 Patrick Trainer: flags.
203 00:15:40.150 ⇒ 00:15:44.249 Bryce Codell: Okay, wait. So it’s getting stored in the repo itself.
204 00:15:44.510 ⇒ 00:15:46.320 Patrick Trainer: So the way.
205 00:15:46.810 ⇒ 00:15:50.059 Patrick Trainer: Yes. So the way Github
206 00:15:50.360 ⇒ 00:15:54.740 Patrick Trainer: stores artifacts is an artifact is
207 00:15:55.950 ⇒ 00:15:59.510 Patrick Trainer: attributed to a specific
208 00:15:59.630 ⇒ 00:16:01.520 Patrick Trainer: workflow. Id
209 00:16:01.760 ⇒ 00:16:03.820 Patrick Trainer: right? And so
210 00:16:03.850 ⇒ 00:16:05.490 Patrick Trainer: each workflow
211 00:16:06.130 ⇒ 00:16:08.619 Patrick Trainer: has an opportunity to
212 00:16:09.100 ⇒ 00:16:12.799 Patrick Trainer: save load an artifact
213 00:16:14.030 ⇒ 00:16:15.160 Patrick Trainer: to that.
214 00:16:15.460 ⇒ 00:16:18.979 Patrick Trainer: or like PIN to that id right.
215 00:16:18.980 ⇒ 00:16:19.390 Bryce Codell: Okay.
216 00:16:19.390 ⇒ 00:16:24.359 Patrick Trainer: And so to grab that artifact. You have to have
217 00:16:24.560 ⇒ 00:16:29.830 Patrick Trainer: the that workflow id. And in order to to know like
218 00:16:30.830 ⇒ 00:16:36.240 Patrick Trainer: what it is, it’s like you’re you’re gonna have to hit that, either like know it from
219 00:16:37.180 ⇒ 00:16:38.090 Patrick Trainer: like
220 00:16:38.930 ⇒ 00:16:40.400 Patrick Trainer: in actions here.
221 00:16:41.370 ⇒ 00:16:42.320 Bryce Codell: And okay, and.
222 00:16:42.320 ⇒ 00:16:44.129 Patrick Trainer: And and find it there like.
223 00:16:44.370 ⇒ 00:16:50.139 Bryce Codell: Alright. So that’s why, like, when I pull down the repo, I there’s no reference to the artifact cause I’m pulling down.
224 00:16:50.140 ⇒ 00:16:53.959 Patrick Trainer: Right. So let me actually I should have
225 00:17:02.150 ⇒ 00:17:03.960 Patrick Trainer: so workflow!
226 00:17:06.780 ⇒ 00:17:07.730 Patrick Trainer: Let me.
227 00:17:09.730 ⇒ 00:17:17.249 Bryce Codell: So when you all push a change to the main branch, do you? Only you only run the models that have been modified.
228 00:17:17.480 ⇒ 00:17:18.190 Patrick Trainer: Correct.
229 00:17:18.339 ⇒ 00:17:23.879 Bryce Codell: And then you run, but then you run the whole project like once a day on that loop on that loop in the area.
230 00:17:23.880 ⇒ 00:17:28.460 Patrick Trainer: Right, right? Right? Right? And so when
231 00:17:29.120 ⇒ 00:17:32.579 Patrick Trainer: that actually brings me back to
232 00:17:34.240 ⇒ 00:17:35.169 Patrick Trainer: what I was
233 00:17:35.310 ⇒ 00:17:36.230 Patrick Trainer: burst.
234 00:17:36.880 ⇒ 00:17:38.390 Patrick Trainer: showing you
235 00:17:40.360 ⇒ 00:17:41.490 Patrick Trainer: what’s happened.
236 00:17:54.380 ⇒ 00:17:55.930 Patrick Trainer: So here.
237 00:17:55.970 ⇒ 00:18:01.700 Patrick Trainer: like, we have this workflow runner. And this is basically running on a loop. So every few hours
238 00:18:02.900 ⇒ 00:18:06.219 Patrick Trainer: the workflow runner is kicking off this job
239 00:18:06.320 ⇒ 00:18:09.879 Patrick Trainer: that’s running the full project. Yeah. And then
240 00:18:09.900 ⇒ 00:18:12.089 Patrick Trainer: that main branch is
241 00:18:12.200 ⇒ 00:18:24.920 Patrick Trainer: posting that manifest to the storage from that workflow. And that’s just happening. That’s just kind of like ongoing, and then, whenever, like, we open a Pr. And that workflow gets triggered.
242 00:18:25.560 ⇒ 00:18:26.850 Patrick Trainer: We’re grabbing
243 00:18:27.910 ⇒ 00:18:29.500 Patrick Trainer: the most recent
244 00:18:29.990 ⇒ 00:18:39.159 Patrick Trainer: art or manifest from this loop wherever whenever that’s happened. And so that manifest gets pulled down. And then that’s what’s
245 00:18:39.570 ⇒ 00:18:42.259 Patrick Trainer: I’ve been, or it gets rung.
246 00:18:42.400 ⇒ 00:18:45.020 Patrick Trainer: And then when that gets merged.
247 00:18:45.460 ⇒ 00:18:55.510 Patrick Trainer: kind of like happens all over again. And then this every few hours is always kind of just ongoing. And so that’s how we’re able to stay
248 00:18:55.680 ⇒ 00:18:56.950 Patrick Trainer: like current
249 00:18:57.300 ⇒ 00:19:02.039 Patrick Trainer: in in like, everything’s up to date and then everything like
250 00:19:02.990 ⇒ 00:19:03.930 Patrick Trainer: host.
251 00:19:04.300 ⇒ 00:19:06.730 Patrick Trainer: This main workflow run like
252 00:19:07.070 ⇒ 00:19:12.519 Patrick Trainer: has to be changed, or it it is going to be changed from
253 00:19:13.770 ⇒ 00:19:19.960 Patrick Trainer: like it’s it’s it’s in the future. So like it’s it’s going to be different, like, there’s not yeah.
254 00:19:22.880 ⇒ 00:19:29.400 Bryce Codell: So then a question for you from a developer perspective, if I wanted to.
255 00:19:30.841 ⇒ 00:19:34.499 Bryce Codell: Like, add a new model.
256 00:19:37.900 ⇒ 00:19:42.969 Bryce Codell: and I was like, and but I also had to modify an existing model in order.
257 00:19:43.630 ⇒ 00:19:44.960 Bryce Codell: Add the new model.
258 00:19:45.100 ⇒ 00:19:50.860 Bryce Codell: And I wanted to do some local development, and just like run those 2 models to make sure that they work.
259 00:19:51.309 ⇒ 00:19:57.570 Bryce Codell: So the tool that I would typically or the feature I would typically reach for is Dbts defer functionality.
260 00:19:58.370 ⇒ 00:20:05.739 Bryce Codell: So what’s the best for me? But I, in order to do that, I need a production version of the manifest.
261 00:20:06.570 ⇒ 00:20:09.290 Patrick Trainer: Right. So
262 00:20:10.390 ⇒ 00:20:15.769 Patrick Trainer: we haven’t used the defer flag yet.
263 00:20:16.830 ⇒ 00:20:17.480 Patrick Trainer: But
264 00:20:18.530 ⇒ 00:20:20.769 Patrick Trainer: one way that we
265 00:20:21.000 ⇒ 00:20:22.540 Patrick Trainer: can grab
266 00:20:22.690 ⇒ 00:20:26.550 Patrick Trainer: that artifact from the main branch
267 00:20:26.650 ⇒ 00:20:30.639 Patrick Trainer: is by hitting that, like Github
268 00:20:32.800 ⇒ 00:20:33.820 Patrick Trainer: api.
269 00:20:34.020 ⇒ 00:20:34.490 Bryce Codell: -
270 00:20:34.490 ⇒ 00:20:39.949 Patrick Trainer: And so like, we have our like pool, parts, actions, artifact, endpoint.
271 00:20:39.950 ⇒ 00:20:40.450 Bryce Codell: -
272 00:20:40.450 ⇒ 00:20:44.290 Patrick Trainer: And let’s actually just like pipe this into less.
273 00:20:45.220 ⇒ 00:20:48.199 Patrick Trainer: That is terrible. Okay.
274 00:20:49.840 ⇒ 00:20:50.540 Patrick Trainer: Nope.
275 00:20:50.910 ⇒ 00:20:51.690 Patrick Trainer: sorry.
276 00:20:55.410 ⇒ 00:20:59.559 Bryce Codell: Fyi. We still see the brain forward, jackions.
277 00:20:59.560 ⇒ 00:21:01.689 Patrick Trainer: Oh, really. Okay, hold on.
278 00:21:01.690 ⇒ 00:21:02.340 Bryce Codell: Yeah.
279 00:21:02.600 ⇒ 00:21:03.950 Bryce Codell: and your screen share.
280 00:21:04.370 ⇒ 00:21:06.530 Bryce Codell: Yeah, it was like, something is happening.
281 00:21:06.530 ⇒ 00:21:10.839 Patrick Trainer: Yeah, I that makes a lot of sense. Why.
282 00:21:12.570 ⇒ 00:21:14.520 Patrick Trainer: you weren’t seeing all the things.
283 00:21:14.850 ⇒ 00:21:15.549 Bryce Codell: Yes. Okay.
284 00:21:16.043 ⇒ 00:21:17.030 Patrick Trainer: Very glad.
285 00:21:17.070 ⇒ 00:21:19.720 Patrick Trainer: So let’s
286 00:21:20.410 ⇒ 00:21:23.259 Patrick Trainer: yeah the zoom boxes
287 00:21:25.120 ⇒ 00:21:25.940 Patrick Trainer: popping up.
288 00:21:26.730 ⇒ 00:21:28.110 Patrick Trainer: So when we hit that
289 00:21:28.690 ⇒ 00:21:32.268 Patrick Trainer: that Api, we have all of these
290 00:21:35.108 ⇒ 00:21:44.510 Patrick Trainer: like different runs that the manifest has been posted on. So for each like action run. See, we have, like from the
291 00:21:44.840 ⇒ 00:21:47.200 Patrick Trainer: 20 second. Most recent is going.
292 00:21:47.200 ⇒ 00:21:47.960 Bryce Codell: Be up with it.
293 00:21:47.990 ⇒ 00:21:49.250 Patrick Trainer: At the top here.
294 00:21:50.620 ⇒ 00:21:51.960 Patrick Trainer: believe are
295 00:21:52.050 ⇒ 00:21:53.970 Patrick Trainer: like we. We store
296 00:21:54.880 ⇒ 00:21:57.199 Patrick Trainer: these artifacts. They’re valid for
297 00:21:57.800 ⇒ 00:21:59.529 Patrick Trainer: the 14 days.
298 00:21:59.690 ⇒ 00:22:00.410 Bryce Codell: Yeah, thank, you.
299 00:22:00.850 ⇒ 00:22:10.787 Patrick Trainer: And so basically what you’re able to to get from here. And this is kind of how the that one action that’s grabbing
300 00:22:12.740 ⇒ 00:22:15.140 Patrick Trainer: the most recent from the main branch.
301 00:22:15.900 ⇒ 00:22:18.770 Patrick Trainer: that this is how this is working. And so
302 00:22:19.250 ⇒ 00:22:22.978 Patrick Trainer: we have our artifact id here.
303 00:22:24.553 ⇒ 00:22:30.909 Patrick Trainer: We’ve got an archive of a download. So this is where we’re actually able to to hit
304 00:22:31.270 ⇒ 00:22:34.540 Patrick Trainer: and and like you could pull down that manifest locally
305 00:22:35.198 ⇒ 00:22:39.269 Patrick Trainer: and then we also are showing like there’s
306 00:22:39.860 ⇒ 00:22:48.090 Patrick Trainer: we’ve got our workflow run id, and then which branch that, it’s running on. And so you could see like here
307 00:22:48.945 ⇒ 00:22:58.820 Patrick Trainer: like this was this ran, so, okay, so this is a good example. So this was run on the branch for this 660 issue.
308 00:23:00.294 ⇒ 00:23:01.870 Patrick Trainer: Around 9, 54
309 00:23:02.607 ⇒ 00:23:09.590 Patrick Trainer: and that ran, this was a like a Ci workflow. So let’s
310 00:23:09.780 ⇒ 00:23:12.330 Patrick Trainer: actually what we can do.
311 00:23:19.340 ⇒ 00:23:20.040 Patrick Trainer: Nope.
312 00:23:25.800 ⇒ 00:23:27.850 Patrick Trainer: sorry. Never mind. Live. Demos.
313 00:23:31.390 ⇒ 00:23:35.279 Patrick Trainer: this is essentially this is our ci running.
314 00:23:36.350 ⇒ 00:23:37.200 Patrick Trainer: Posting
315 00:23:37.820 ⇒ 00:23:40.520 Patrick Trainer: it, it produces its own manifest.
316 00:23:40.570 ⇒ 00:23:43.359 Patrick Trainer: And then, after this, it’s been pushed.
317 00:23:43.760 ⇒ 00:23:45.930 Patrick Trainer: And this is like that main
318 00:23:46.040 ⇒ 00:23:47.310 Patrick Trainer: manifest.
319 00:23:47.766 ⇒ 00:23:48.300 Bryce Codell: For it!
320 00:23:52.150 ⇒ 00:23:56.119 Patrick Trainer: that’s another example there and then. Here’s like the
321 00:23:56.270 ⇒ 00:23:58.290 Patrick Trainer: the previous main run.
322 00:23:58.290 ⇒ 00:23:59.090 Bryce Codell: Yeah.
323 00:23:59.300 ⇒ 00:23:59.850 Bryce Codell: Dinner.
324 00:24:02.610 ⇒ 00:24:03.290 Bryce Codell: Yeah.
325 00:24:05.060 ⇒ 00:24:11.310 Patrick Trainer: Trying to think of like. So you’re the you’re the 1st out outside of like when this was built
326 00:24:11.640 ⇒ 00:24:22.357 Patrick Trainer: of trying to understand it so that it’s kind of like, why, this is rocky and explaining it. so please, yeah, do do forgive me.
327 00:24:22.740 ⇒ 00:24:31.150 Bryce Codell: So like this whole workflow makes sense like one I’m just thinking about like tech from a technical standpoint, like one like the way that I have done things that Ashby is.
328 00:24:32.690 ⇒ 00:24:35.160 Bryce Codell: I just create a
329 00:24:35.440 ⇒ 00:24:36.180 Bryce Codell: yeah
330 00:24:36.840 ⇒ 00:24:42.249 Bryce Codell: up like a production profile that has my own credentials in it. And then I like.
331 00:24:42.670 ⇒ 00:24:43.770 Bryce Codell: Whoa!
332 00:24:44.290 ⇒ 00:24:48.130 Bryce Codell: You in like could run dbt, compile
333 00:24:49.560 ⇒ 00:24:51.490 Bryce Codell: with that like
334 00:24:51.590 ⇒ 00:24:53.859 Bryce Codell: with that like, pseudo prod.
335 00:24:57.430 ⇒ 00:24:59.899 Bryce Codell: yeah, what’s it called? Yeah, let’s do that.
336 00:24:59.900 ⇒ 00:25:01.420 Patrick Trainer: Like the target. Yeah.
337 00:25:01.834 ⇒ 00:25:04.319 Bryce Codell: Target. Yeah, target profile. And then
338 00:25:04.420 ⇒ 00:25:08.380 Bryce Codell: and then it spits out the manifest id. And then I just move that that manifest in.
339 00:25:08.380 ⇒ 00:25:08.970 Patrick Trainer: Right.
340 00:25:08.970 ⇒ 00:25:12.649 Bryce Codell: Like into a different directory, and then I have, like a
341 00:25:12.780 ⇒ 00:25:25.280 Bryce Codell: an alias saved in my shell that allows me to run. Do a Dbt defer, run and while referencing that version of the manifest, and it works.
342 00:25:25.280 ⇒ 00:25:26.020 Patrick Trainer: Got it.
343 00:25:26.020 ⇒ 00:25:27.639 Bryce Codell: And outside little Manhattan.
344 00:25:27.640 ⇒ 00:25:28.160 Patrick Trainer: Actually.
345 00:25:28.516 ⇒ 00:25:29.229 Bryce Codell: Karen! But.
346 00:25:29.710 ⇒ 00:25:34.000 Patrick Trainer: That. Yeah, that that makes sense. I should actually have.
347 00:25:38.360 ⇒ 00:25:41.130 Patrick Trainer: these, what are all these?
348 00:25:46.170 ⇒ 00:25:48.569 Patrick Trainer: Okay, yeah, I know I had a branch
349 00:25:48.650 ⇒ 00:25:49.730 Patrick Trainer: here somewhere.
350 00:25:56.600 ⇒ 00:25:57.720 Patrick Trainer: a
351 00:25:57.960 ⇒ 00:25:59.819 Patrick Trainer: okay. So in here.
352 00:26:00.180 ⇒ 00:26:03.789 Patrick Trainer: like we can see that there was like this change.
353 00:26:04.250 ⇒ 00:26:04.720 Bryce Codell: Yet.
354 00:26:04.720 ⇒ 00:26:05.420 Patrick Trainer: This
355 00:26:05.630 ⇒ 00:26:07.150 Patrick Trainer: foobar dot SQL.
356 00:26:07.230 ⇒ 00:26:08.480 Patrick Trainer: Was changed.
357 00:26:08.700 ⇒ 00:26:11.570 Patrick Trainer: and then that triggered
358 00:26:14.340 ⇒ 00:26:15.730 Patrick Trainer: these 2
359 00:26:17.080 ⇒ 00:26:18.200 Patrick Trainer: workflows
360 00:26:18.350 ⇒ 00:26:20.660 Patrick Trainer: which we’re going to
361 00:26:21.080 ⇒ 00:26:22.180 Patrick Trainer: authenticate.
362 00:26:22.390 ⇒ 00:26:23.270 Patrick Trainer: That’s
363 00:26:23.720 ⇒ 00:26:29.850 Patrick Trainer: not very fancy. This is what this is doing. This is going to actually run
364 00:26:32.260 ⇒ 00:26:33.400 Patrick Trainer: the Github
365 00:26:34.070 ⇒ 00:26:38.090 Patrick Trainer: Cli commands. And so you can see that here
366 00:26:39.124 ⇒ 00:26:42.569 Patrick Trainer: and then we’ve got a Jq, query.
367 00:26:42.620 ⇒ 00:26:44.700 Patrick Trainer: that we are
368 00:26:45.420 ⇒ 00:26:48.229 Patrick Trainer: essentially like piping it into
369 00:26:49.028 ⇒ 00:26:51.720 Patrick Trainer: and so we can actually
370 00:26:52.370 ⇒ 00:26:53.949 Patrick Trainer: emulate that
371 00:26:55.040 ⇒ 00:26:55.960 Patrick Trainer: math
372 00:26:57.380 ⇒ 00:26:59.580 Patrick Trainer: here. So let’s
373 00:27:00.830 ⇒ 00:27:01.610 Patrick Trainer: that
374 00:27:02.160 ⇒ 00:27:03.190 Patrick Trainer: like that.
375 00:27:15.390 ⇒ 00:27:16.390 Patrick Trainer: What am I doing.
376 00:27:25.530 ⇒ 00:27:27.000 Patrick Trainer: this is
377 00:27:41.030 ⇒ 00:27:42.350 Patrick Trainer: live Demos. Again.
378 00:27:42.750 ⇒ 00:27:46.320 Bryce Codell: It wouldn’t be on the main branch, would it? Since these are all getting committed.
379 00:27:46.320 ⇒ 00:27:47.890 Patrick Trainer: Yeah, we’re.
380 00:27:47.890 ⇒ 00:27:50.240 Bryce Codell: So branch instances.
381 00:27:50.390 ⇒ 00:27:53.199 Patrick Trainer: Right right right right right right
382 00:27:53.730 ⇒ 00:27:55.180 Patrick Trainer: a
383 00:27:56.390 ⇒ 00:27:57.140 Patrick Trainer: both.
384 00:27:57.610 ⇒ 00:28:01.931 Patrick Trainer: So I’ll I’ll figure this out like 5 min after we get off this call.
385 00:28:02.250 ⇒ 00:28:02.960 Bryce Codell: But yeah.
386 00:28:02.960 ⇒ 00:28:05.659 Patrick Trainer: Of course. But so
387 00:28:07.440 ⇒ 00:28:08.589 Patrick Trainer: this comes.
388 00:28:08.690 ⇒ 00:28:15.489 Patrick Trainer: And then we’re setting an output of the artifact Id. That the Jq. Query is
389 00:28:16.370 ⇒ 00:28:17.640 Patrick Trainer: looking for.
390 00:28:17.640 ⇒ 00:28:18.070 Bryce Codell: Yeah.
391 00:28:18.935 ⇒ 00:28:22.979 Patrick Trainer: And we’ve got this most recent artifact. Id.
392 00:28:23.775 ⇒ 00:28:24.260 Patrick Trainer: Here.
393 00:28:25.010 ⇒ 00:28:28.720 Patrick Trainer: And then in this, this is where we’re actually downloading
394 00:28:29.810 ⇒ 00:28:31.549 Patrick Trainer: the artifact.
395 00:28:31.550 ⇒ 00:28:31.970 Bryce Codell: Yeah.
396 00:28:31.970 ⇒ 00:28:34.730 Patrick Trainer: We’ve so we have the the workflow id
397 00:28:34.960 ⇒ 00:28:37.060 Patrick Trainer: the repo that it’s doing
398 00:28:37.881 ⇒ 00:28:43.448 Patrick Trainer: and then this is like the name of the file that we’ve loaded from before.
399 00:28:43.940 ⇒ 00:28:47.249 Patrick Trainer: And then it gives us this like.
400 00:28:47.610 ⇒ 00:28:49.879 Patrick Trainer: yeah, URL, which it downloads
401 00:28:50.360 ⇒ 00:28:51.700 Patrick Trainer: and then
402 00:28:52.320 ⇒ 00:29:01.129 Patrick Trainer: Dbt installs. So we have this artifact downloaded, and that becomes kind of like available in the in the runner context.
403 00:29:02.300 ⇒ 00:29:05.699 Patrick Trainer: the run dbt, and the use artifact id
404 00:29:06.188 ⇒ 00:29:12.079 Patrick Trainer: run. Dvt is dependent on this. And so you’re able to pass that artifact to it.
405 00:29:12.500 ⇒ 00:29:13.290 Bryce Codell: Yet.
406 00:29:13.762 ⇒ 00:29:16.027 Patrick Trainer: And so this run dbt.
407 00:29:17.019 ⇒ 00:29:21.240 Patrick Trainer: just going to move over into that test, Slim.
408 00:29:22.665 ⇒ 00:29:23.270 Patrick Trainer: And
409 00:29:23.460 ⇒ 00:29:25.275 Patrick Trainer: then it’s going to
410 00:29:27.420 ⇒ 00:29:30.209 Patrick Trainer: run the things that have changed.
411 00:29:30.510 ⇒ 00:29:31.270 Bryce Codell: Yeah. Stream.
412 00:29:31.390 ⇒ 00:29:36.069 Bryce Codell: Why was it unable to do partial parsing? Because couldn’t you fetch the manifest
413 00:29:37.290 ⇒ 00:29:38.769 Bryce Codell: in line 24.
414 00:29:39.010 ⇒ 00:29:40.842 Patrick Trainer: Yeah, I see that.
415 00:29:45.200 ⇒ 00:29:49.209 Patrick Trainer: I don’t have a good answer for that. That might be the that might be
416 00:29:49.700 ⇒ 00:29:51.260 Patrick Trainer: the defer
417 00:29:51.450 ⇒ 00:29:52.409 Patrick Trainer: flag.
418 00:29:53.390 ⇒ 00:29:55.010 Patrick Trainer: because, like the
419 00:29:55.500 ⇒ 00:30:00.190 Patrick Trainer: right now, or how this is working, it’s like, we’re looking at
420 00:30:00.830 ⇒ 00:30:02.900 Patrick Trainer: modified state.
421 00:30:04.315 ⇒ 00:30:09.850 Patrick Trainer: Yeah, it’s it’s we’re using like the state flag. And then like selecting
422 00:30:10.080 ⇒ 00:30:12.690 Patrick Trainer: the modified models.
423 00:30:12.810 ⇒ 00:30:17.249 Patrick Trainer: And so we’re essentially using yeah, the or the manifest as state
424 00:30:18.165 ⇒ 00:30:20.790 Patrick Trainer: or like, ask that base. And then
425 00:30:22.380 ⇒ 00:30:23.720 Patrick Trainer: selecting
426 00:30:23.810 ⇒ 00:30:27.719 Patrick Trainer: the files that we have modified on our most recent Dbt. Run
427 00:30:27.850 ⇒ 00:30:30.559 Patrick Trainer: and and gone from there. Gotcha.
428 00:30:30.560 ⇒ 00:30:31.600 Bryce Codell: Definitely tense.
429 00:30:32.310 ⇒ 00:30:36.540 Bryce Codell: Okay, I have a security question, do.
430 00:30:37.530 ⇒ 00:30:38.696 Bryce Codell: are there
431 00:30:39.460 ⇒ 00:30:42.780 Bryce Codell: like controls in place? As to who can write
432 00:30:43.000 ⇒ 00:30:46.670 Bryce Codell: like which users which roles can write to
433 00:30:47.510 ⇒ 00:30:50.380 Bryce Codell: like, whatever the production schema is that
434 00:30:50.915 ⇒ 00:30:54.320 Bryce Codell: like these, like github action, run and stuff.
435 00:30:54.320 ⇒ 00:30:56.869 Patrick Trainer: So so the
436 00:30:57.830 ⇒ 00:31:00.600 Patrick Trainer: actions, yes, we we have
437 00:31:00.980 ⇒ 00:31:09.430 Patrick Trainer: are like we have separate roles and users, for, like our Ci environment as well as like our prod, runs.
438 00:31:09.430 ⇒ 00:31:22.659 Bryce Codell: For individual users like here, let me give you an example. So like let’s say I, accidentally, I may. I mean, I’m like doing local development, and I intend to execute a Dbt compile target prod
439 00:31:22.970 ⇒ 00:31:27.890 Bryce Codell: command, and I absolutely minded we execute dbt, run, target prod.
440 00:31:28.050 ⇒ 00:31:36.580 Bryce Codell: Am I going to? And but my local prod target is configured just with the same credential, like, just with my personal credentials.
441 00:31:36.580 ⇒ 00:31:37.170 Patrick Trainer: Right.
442 00:31:38.710 ⇒ 00:31:40.139 Bryce Codell: Am I gonna get? Yeah, you.
443 00:31:40.140 ⇒ 00:31:40.960 Patrick Trainer: Yeah, able to.
444 00:31:41.220 ⇒ 00:31:43.100 Bryce Codell: Oh, I’d be able to overwrite everything.
445 00:31:43.100 ⇒ 00:31:45.939 Patrick Trainer: Right? Yeah, you you’d you’d be able to force, push.
446 00:31:46.350 ⇒ 00:31:54.419 Bryce Codell: Okay, I would be able to. So I would be able to overwrite everything in the production schema databases. Okay, alright. Just have to be careful. There, that is.
447 00:31:54.420 ⇒ 00:31:56.549 Patrick Trainer: Yeah, I mean, so like
448 00:31:56.780 ⇒ 00:32:00.800 Patrick Trainer: to. And to be fair with that, too. It’s like, if that does happen.
449 00:32:01.750 ⇒ 00:32:02.890 Patrick Trainer: because
450 00:32:03.040 ⇒ 00:32:06.009 Patrick Trainer: the nature of Dbt being like indefinite like
451 00:32:06.040 ⇒ 00:32:07.520 Patrick Trainer: you just run it again
452 00:32:08.413 ⇒ 00:32:12.010 Patrick Trainer: it would rebuild the entire project and take 15 min. But
453 00:32:13.010 ⇒ 00:32:15.210 Patrick Trainer: I think that’s why I’ve always felt pretty
454 00:32:15.870 ⇒ 00:32:17.499 Patrick Trainer: like safe, if
455 00:32:17.640 ⇒ 00:32:19.110 Patrick Trainer: like. If if
456 00:32:20.320 ⇒ 00:32:24.500 Patrick Trainer: if our like say our marts or goal player, whatever
457 00:32:24.630 ⇒ 00:32:25.530 Patrick Trainer: was
458 00:32:25.920 ⇒ 00:32:27.739 Patrick Trainer: dropped entirely.
459 00:32:27.900 ⇒ 00:32:34.410 Patrick Trainer: then you can just kind of like rerun the project, I guess snapshots would be fucked. But that’s that’s
460 00:32:34.950 ⇒ 00:32:42.619 Patrick Trainer: not the the end of the world there. But but yes, the the the so the pool parts.
461 00:32:43.030 ⇒ 00:32:47.300 Patrick Trainer: Snowflake instance itself. The
462 00:32:47.810 ⇒ 00:32:51.289 Patrick Trainer: role hierarchy is an absolute mess.
463 00:32:52.630 ⇒ 00:32:58.849 Patrick Trainer: and that’s just been on like back burner for me. I’m not gonna
464 00:32:58.910 ⇒ 00:33:03.470 Patrick Trainer: name names, but it was like that before I got here, and just haven’t touched it.
465 00:33:04.680 ⇒ 00:33:14.020 Patrick Trainer: Just to not break anything else the Stella. And there’s another client. All those like those are set up
466 00:33:14.110 ⇒ 00:33:15.679 Patrick Trainer: really well in
467 00:33:16.970 ⇒ 00:33:19.070 Patrick Trainer: like a right hierarchy.
468 00:33:21.461 ⇒ 00:33:22.615 Patrick Trainer: and then
469 00:33:25.830 ⇒ 00:33:27.180 Patrick Trainer: What am I thinking?
470 00:33:27.675 ⇒ 00:33:29.239 Patrick Trainer: I think. Where was he going with that?
471 00:33:30.216 ⇒ 00:33:31.550 Patrick Trainer: Oh! And
472 00:33:32.110 ⇒ 00:33:36.830 Patrick Trainer: as you poke through the Dbt project. You’ll also
473 00:33:37.030 ⇒ 00:33:39.919 Patrick Trainer: see that the lineage is
474 00:33:41.400 ⇒ 00:33:46.530 Patrick Trainer: crazy as well. So like we’ll have like source models that
475 00:33:46.650 ⇒ 00:34:02.250 Patrick Trainer: will be called in intermediate models, and then you’ll also have those same source. Models be called in the like post intermediate models, like production models. And so it’s like trying to trace that lineage is really difficult
476 00:34:02.615 ⇒ 00:34:07.499 Patrick Trainer: and it’s also just kinda like all over the place. So that’s 1 of the little
477 00:34:07.540 ⇒ 00:34:12.600 Patrick Trainer: quirks. And another thing in in backlog to that needs to be cleaned.
478 00:34:13.639 ⇒ 00:34:14.880 Patrick Trainer: Yeah.
479 00:34:14.880 ⇒ 00:34:16.455 Bryce Codell: No, that makes sense.
480 00:34:17.179 ⇒ 00:34:30.759 Bryce Codell: like with the like. The stuff that I’m doing is pretty experimental, like, I just have, like a bunch of analysis tools that I wanna hack on in my spare time. A new town was kind enough to say like, Go for it, and like.
481 00:34:30.870 ⇒ 00:34:33.240 Bryce Codell: we won’t pay you until you figure out how to.
482 00:34:33.480 ⇒ 00:34:34.050 Patrick Trainer: Yeah. He’s.
483 00:34:34.050 ⇒ 00:34:40.850 Bryce Codell: That’s sellable, but that works great for me. So my my intent, candidly, is to
484 00:34:40.980 ⇒ 00:34:45.680 Bryce Codell: kinda start from scratch. I’m just trying to kind of wrap my arms around the
485 00:34:47.469 ⇒ 00:34:54.770 Bryce Codell: like the sense of like what already exists. What can I like? What is available that I can leverage or like copy pasta, and then, like
486 00:34:54.850 ⇒ 00:35:00.840 Bryce Codell: what? What are the like? Dev workflow best practices that the team has in place.
487 00:35:01.240 ⇒ 00:35:02.040 Patrick Trainer: Right.
488 00:35:02.630 ⇒ 00:35:09.599 Bryce Codell: So like, like, I’ll give you an example like yesterday. So like, I’m like vehemently opposed to like Pip
489 00:35:10.006 ⇒ 00:35:18.283 Bryce Codell: like global Pip. And so when I saw the requirements file, I just like Googled like a stack overflow solution to
490 00:35:18.950 ⇒ 00:35:19.730 Bryce Codell: like
491 00:35:19.990 ⇒ 00:35:37.789 Bryce Codell: pipe requirements. File into like auto, creating a like poetry, invite a new poetry based environment. So like I have poetry set up in my pool in my local pool parts. Repo. I’m like a personal brand. John, like a local debt branch and.
492 00:35:37.790 ⇒ 00:35:38.200 Patrick Trainer: Ice.
493 00:35:38.200 ⇒ 00:35:41.150 Bryce Codell: Dot com doing environment management. And then
494 00:35:41.230 ⇒ 00:35:46.679 Bryce Codell: I like added, like I did the like schema and database override
495 00:35:46.730 ⇒ 00:35:55.304 Bryce Codell: play from Dbt, so that I’m just like always writing to like a Dbt underscore. Brice.
496 00:35:56.000 ⇒ 00:36:05.219 Bryce Codell: Yeah, I saw that you could put like I saw that you had some sort of setup available for data like for data and files. So I’d like installed dear end. And then that.
497 00:36:05.220 ⇒ 00:36:05.880 Patrick Trainer: Right.
498 00:36:05.880 ⇒ 00:36:14.453 Bryce Codell: And that’s how I’m doing environment variable management. But like, and some models with Dbt yesterday, work just fine
499 00:36:15.270 ⇒ 00:36:19.690 Patrick Trainer: Yeah, that’s that’s how my like local is set up as well.
500 00:36:20.780 ⇒ 00:36:25.500 Patrick Trainer: With with this, what is nice is we are caching
501 00:36:25.770 ⇒ 00:36:28.279 Patrick Trainer: the the like pip dependencies.
502 00:36:28.280 ⇒ 00:36:28.600 Bryce Codell: Yeah.
503 00:36:28.600 ⇒ 00:36:30.960 Patrick Trainer: And so if we
504 00:36:32.200 ⇒ 00:36:34.180 Patrick Trainer: fucking find there we go.
505 00:36:36.180 ⇒ 00:36:37.810 Patrick Trainer: It’s actually come into here.
506 00:36:39.120 ⇒ 00:36:40.810 Patrick Trainer: Let’s find a summary.
507 00:36:45.410 ⇒ 00:36:49.669 Patrick Trainer: Okay, so this is just a checkout job where we’re
508 00:36:49.680 ⇒ 00:36:51.960 Patrick Trainer: checking out the the repo.
509 00:36:53.433 ⇒ 00:36:54.500 Patrick Trainer: In here.
510 00:36:55.280 ⇒ 00:36:58.440 Patrick Trainer: We’ve got our setup where we’re
511 00:36:59.078 ⇒ 00:37:05.180 Patrick Trainer: actually running that one action we’re passing in the project python version.
512 00:37:05.230 ⇒ 00:37:08.110 Patrick Trainer: Dbt, version, if we wanted to do that.
513 00:37:10.400 ⇒ 00:37:12.739 Patrick Trainer: we are caching
514 00:37:13.310 ⇒ 00:37:17.380 Patrick Trainer: Pip, and we have these like this, like Pip Key
515 00:37:17.670 ⇒ 00:37:22.010 Patrick Trainer: of, or the the the cache key that we’re doing
516 00:37:22.974 ⇒ 00:37:27.200 Patrick Trainer: and what’s nice is like, we run that like Pip install.
517 00:37:27.680 ⇒ 00:37:28.580 Patrick Trainer: I’ve
518 00:37:32.790 ⇒ 00:37:34.130 Patrick Trainer: looking on there
519 00:37:34.768 ⇒ 00:37:43.979 Patrick Trainer: and what’s nice is like we do have these like like the the cash hits. And so all of these
520 00:37:46.800 ⇒ 00:37:51.340 Patrick Trainer: dependencies are already cached. So I think I think this.
521 00:37:52.180 ⇒ 00:37:54.399 Patrick Trainer: I mean, yeah, this entire step.
522 00:37:54.880 ⇒ 00:37:56.120 Patrick Trainer: Yeah, Runton.
523 00:37:56.290 ⇒ 00:37:59.359 Patrick Trainer: not much or not much time.
524 00:37:59.360 ⇒ 00:38:05.239 Bryce Codell: Yeah, this is awesome, like, this is a really slick setup for Cicd, and.
525 00:38:07.170 ⇒ 00:38:08.190 Patrick Trainer: heads, and then.
526 00:38:08.190 ⇒ 00:38:10.280 Bryce Codell: Running low on battery. My friend.
527 00:38:10.830 ⇒ 00:38:12.789 Patrick Trainer: Who I am. You’ll call.
528 00:38:15.010 ⇒ 00:38:16.710 Patrick Trainer: Hold on! I’ve got that here
529 00:38:19.980 ⇒ 00:38:20.820 Patrick Trainer: boom!
530 00:38:23.370 ⇒ 00:38:27.079 Patrick Trainer: And we’re also caching the Dbt packages.
531 00:38:27.080 ⇒ 00:38:27.670 Bryce Codell: Swell.
532 00:38:27.670 ⇒ 00:38:28.710 Patrick Trainer: Which is nice.
533 00:38:31.110 ⇒ 00:38:32.200 Patrick Trainer: this is
534 00:38:32.270 ⇒ 00:38:33.519 Patrick Trainer: just some like.
535 00:38:33.660 ⇒ 00:38:36.939 Patrick Trainer: basically debugging stuff. This is showing like the
536 00:38:37.000 ⇒ 00:38:39.090 Patrick Trainer: we’re listing the
537 00:38:39.110 ⇒ 00:38:40.620 Patrick Trainer: runner environment.
538 00:38:40.620 ⇒ 00:38:49.642 Bryce Codell: Yeah, yeah, it was pretty fucking insane when I ran Dbt depths yesterday, and it just like it felt like it was downloading the entire Internet.
539 00:38:49.990 ⇒ 00:38:57.488 Patrick Trainer: Yeah, yeah. The so one of the packages that they’re using, or that that’s being used is
540 00:38:59.010 ⇒ 00:39:03.450 Patrick Trainer: like, it’s from the Dbt package hub. It’s like the
541 00:39:05.470 ⇒ 00:39:07.429 Patrick Trainer: it’s like an ad sales
542 00:39:07.480 ⇒ 00:39:10.230 Patrick Trainer: thing or like, it adds
543 00:39:10.520 ⇒ 00:39:14.964 Patrick Trainer: like a add conglomerate package. Basically that packages
544 00:39:17.020 ⇒ 00:39:19.949 Patrick Trainer: like Google analytics
545 00:39:20.160 ⇒ 00:39:24.850 Patrick Trainer: top spot, shopify this and that, and it’s it’s
546 00:39:25.040 ⇒ 00:39:29.019 Patrick Trainer: that package has all these sub packages that
547 00:39:29.290 ⇒ 00:39:31.179 Patrick Trainer: aren’t even being used.
548 00:39:31.952 ⇒ 00:39:35.489 Patrick Trainer: But because that one package is using it.
549 00:39:35.810 ⇒ 00:39:43.650 Patrick Trainer: They all like kind of like cascade, and I’ll find the conversation we had in slack about it, because, like that
550 00:39:43.780 ⇒ 00:39:47.099 Patrick Trainer: load is is like absolutely
551 00:39:48.280 ⇒ 00:39:50.429 Patrick Trainer: insane. There’s way too much.
552 00:39:56.350 ⇒ 00:39:57.509 Patrick Trainer: right? Now, I’m just gonna
553 00:39:57.550 ⇒ 00:39:58.590 Patrick Trainer: get out of that.
554 00:40:02.010 ⇒ 00:40:03.350 Patrick Trainer: Okay, yeah.
555 00:40:03.690 ⇒ 00:40:07.050 Patrick Trainer: So we can see here, we’re basically
556 00:40:07.750 ⇒ 00:40:10.150 Patrick Trainer: yeah, we have our like, Ci target.
557 00:40:11.520 ⇒ 00:40:12.530 Patrick Trainer: Everything
558 00:40:12.600 ⇒ 00:40:14.760 Patrick Trainer: that’s set in our environment.
559 00:40:15.610 ⇒ 00:40:17.540 Patrick Trainer: Then we’re going to hit
560 00:40:17.580 ⇒ 00:40:18.979 Patrick Trainer: our base
561 00:40:19.030 ⇒ 00:40:20.150 Patrick Trainer: action.
562 00:40:20.810 ⇒ 00:40:23.890 Patrick Trainer: It’s going to pass in our python version.
563 00:40:23.890 ⇒ 00:40:24.760 Bryce Codell: On, that.
564 00:40:26.030 ⇒ 00:40:26.890 Patrick Trainer: Sets up
565 00:40:29.190 ⇒ 00:40:30.380 Patrick Trainer: updates. Pip.
566 00:40:30.500 ⇒ 00:40:33.190 Patrick Trainer: Of course, everything’s already cached.
567 00:40:35.080 ⇒ 00:40:36.109 Patrick Trainer: We have some
568 00:40:36.160 ⇒ 00:40:37.639 Patrick Trainer: unused keys
569 00:40:38.393 ⇒ 00:40:39.340 Patrick Trainer: and then
570 00:40:41.500 ⇒ 00:40:43.249 Patrick Trainer: the models that have changed.
571 00:40:43.560 ⇒ 00:40:43.890 Bryce Codell: Or what.
572 00:40:43.890 ⇒ 00:40:46.940 Patrick Trainer: Runs, and so, like our run or
573 00:40:46.990 ⇒ 00:40:54.960 Patrick Trainer: full project, like Dbt run, actually only runs in 32 seconds. Before we did this it was taken like 15
574 00:40:55.010 ⇒ 00:40:56.870 Patrick Trainer: minutes or so.
575 00:40:57.010 ⇒ 00:40:57.420 Bryce Codell: Yeah.
576 00:40:58.490 ⇒ 00:40:59.699 Patrick Trainer: And then this
577 00:40:59.870 ⇒ 00:41:05.389 Patrick Trainer: last one is just the the test, and it goes through like the same loop.
578 00:41:09.640 ⇒ 00:41:10.380 Patrick Trainer: A.
579 00:41:10.380 ⇒ 00:41:11.050 Bryce Codell: Ice.
580 00:41:11.370 ⇒ 00:41:14.930 Bryce Codell: Does the does anyone in the team use 0 copy cloning.
581 00:41:16.480 ⇒ 00:41:18.480 Patrick Trainer: No.
582 00:41:18.550 ⇒ 00:41:20.109 Patrick Trainer: not that I know of.
583 00:41:21.720 ⇒ 00:41:26.100 Patrick Trainer: There’s like. So I would say, apart from
584 00:41:27.840 ⇒ 00:41:31.080 Patrick Trainer: like things are very, I guess, like bare bones.
585 00:41:31.080 ⇒ 00:41:31.550 Bryce Codell: Hmm.
586 00:41:31.550 ⇒ 00:41:33.719 Patrick Trainer: In the sense that it’s kind of like.
587 00:41:35.320 ⇒ 00:41:42.390 Patrick Trainer: Everything’s being rebuilt almost fully apart from like this notion of like slim ci
588 00:41:43.800 ⇒ 00:41:44.600 Patrick Trainer: each
589 00:41:44.840 ⇒ 00:41:48.630 Patrick Trainer: time, and I’m sure that there’s
590 00:41:49.680 ⇒ 00:42:02.118 Patrick Trainer: some developers like when they’re changing their model. They just dbt run and run the entire thing. Wait 10 min. Do that instead of like just selecting their the models that they’re working on
591 00:42:03.400 ⇒ 00:42:06.909 Patrick Trainer: and so like that is going on
592 00:42:09.520 ⇒ 00:42:10.840 Patrick Trainer: And so
593 00:42:12.140 ⇒ 00:42:13.599 Patrick Trainer: as far as like
594 00:42:13.630 ⇒ 00:42:17.350 Patrick Trainer: are things streamlined? Not
595 00:42:17.470 ⇒ 00:42:21.479 Patrick Trainer: really. There’s a lot of room for improvement, for sure.
596 00:42:22.939 ⇒ 00:42:28.329 Patrick Trainer: The setting up been like tearing down of like dev environments
597 00:42:28.480 ⇒ 00:42:29.310 Patrick Trainer: a
598 00:42:29.750 ⇒ 00:42:30.950 Patrick Trainer: that meets
599 00:42:31.130 ⇒ 00:42:32.349 Patrick Trainer: a lot of work.
600 00:42:32.370 ⇒ 00:42:34.890 Patrick Trainer: There’s a lot of kind of like Croft
601 00:42:35.170 ⇒ 00:42:36.360 Patrick Trainer: out there.
602 00:42:36.490 ⇒ 00:42:43.579 Patrick Trainer: One thing that you did mention already is that, like that dpt like schema override macro.
603 00:42:44.890 ⇒ 00:42:51.270 Patrick Trainer: I think that would be huge in getting in there like, right now, when everybody else is
604 00:42:51.790 ⇒ 00:42:55.559 Patrick Trainer: running it, and it it kind of like overwrites everything.
605 00:42:55.560 ⇒ 00:42:56.150 Bryce Codell: Yeah.
606 00:42:56.150 ⇒ 00:42:57.400 Patrick Trainer: Yeah, yeah, and so.
607 00:42:57.400 ⇒ 00:43:00.620 Bryce Codell: I I can push that pr like.
608 00:43:01.080 ⇒ 00:43:01.580 Patrick Trainer: Yeah.
609 00:43:01.580 ⇒ 00:43:02.250 Bryce Codell: Okay. Yeah.
610 00:43:02.250 ⇒ 00:43:04.510 Patrick Trainer: Yeah, it’s yeah. Go for that. Go for it.
611 00:43:05.503 ⇒ 00:43:10.589 Bryce Codell: It’s got poetry in there, too, in case that is of interest as well.
612 00:43:11.258 ⇒ 00:43:15.025 Bryce Codell: But I can also just get. I can also just exclude those
613 00:43:16.950 ⇒ 00:43:18.460 Bryce Codell: if that’s preferable.
614 00:43:20.350 ⇒ 00:43:22.729 Patrick Trainer: Yeah, no, I mean that all
615 00:43:23.010 ⇒ 00:43:24.310 Patrick Trainer: sounds
616 00:43:24.660 ⇒ 00:43:25.930 Patrick Trainer: really great
617 00:43:26.360 ⇒ 00:43:27.060 Patrick Trainer: bad.
618 00:43:28.440 ⇒ 00:43:30.829 Patrick Trainer: and then there’s a whole lot of
619 00:43:31.990 ⇒ 00:43:34.430 Patrick Trainer: like a bunch of other stuff that
620 00:43:34.780 ⇒ 00:43:35.900 Patrick Trainer: kind of.
621 00:43:35.900 ⇒ 00:43:36.870 Bryce Codell: Yeah.
622 00:43:37.210 ⇒ 00:43:44.529 Bryce Codell: yeah, we can skip over some of the like. The light desk stuff, depend, etc. But the one other question that’s
623 00:43:44.710 ⇒ 00:43:52.570 Bryce Codell: top of mind for me is so one of our analyses that I’m I’m exploring is related to
624 00:43:53.906 ⇒ 00:43:56.249 Bryce Codell: weather data. And so
625 00:43:56.550 ⇒ 00:44:10.670 Bryce Codell: there is a potential scenario where we are able to attain weather data from stuff like data share, and that just like click on the marketplace. So then, that just works fine. We don’t have to like worry about Elt, or anything like that
626 00:44:10.740 ⇒ 00:44:19.860 Bryce Codell: or the El stuff. But in the event that the best path forward is to consume data from an Api
627 00:44:19.870 ⇒ 00:44:41.570 Bryce Codell: with like a pipe like with a python script schedule like, What’s the do you have any recommendations for like the best path there. Would it be like, write a python app and then define a Github action to execute it on a cron schedule? Or is there a different flavor.
628 00:44:41.570 ⇒ 00:44:42.680 Patrick Trainer: That’s
629 00:44:44.960 ⇒ 00:44:46.160 Patrick Trainer: let me
630 00:44:47.950 ⇒ 00:44:50.330 Patrick Trainer: let me think about it a little bit. I
631 00:44:51.840 ⇒ 00:44:56.440 Patrick Trainer: I like. So I’ve had really good success with like
632 00:44:57.040 ⇒ 00:44:58.040 Patrick Trainer: running
633 00:44:58.230 ⇒ 00:45:06.590 Patrick Trainer: scripts, workflows like basically substituting what you would run in like airflow or dax, or whatever any orchestration tool
634 00:45:06.890 ⇒ 00:45:12.169 Patrick Trainer: with gitlab or github ci, and so like, that’s
635 00:45:12.680 ⇒ 00:45:14.370 Patrick Trainer: like super nice
636 00:45:15.066 ⇒ 00:45:17.009 Patrick Trainer: and so I would
637 00:45:18.120 ⇒ 00:45:23.730 Patrick Trainer: do something like that the way I might approach it would be
638 00:45:27.330 ⇒ 00:45:28.380 Patrick Trainer: like if.
639 00:45:30.090 ⇒ 00:45:32.290 Patrick Trainer: say, you hit an Api.
640 00:45:33.310 ⇒ 00:45:35.479 Patrick Trainer: pull that down, pull down your data.
641 00:45:37.590 ⇒ 00:45:39.229 Patrick Trainer: and then dump that
642 00:45:39.280 ⇒ 00:45:41.610 Patrick Trainer: into like an S. 3 bucket
643 00:45:41.990 ⇒ 00:45:45.219 Patrick Trainer: as like a parquet file. And then we
644 00:45:46.470 ⇒ 00:45:50.099 Patrick Trainer: just create like an external stage and then
645 00:45:51.020 ⇒ 00:45:54.730 Patrick Trainer: have a tab, or either use like a task or
646 00:45:55.220 ⇒ 00:46:02.810 Patrick Trainer: that like snow pipe trigger that will load it down and and load that into like our raw
647 00:46:02.820 ⇒ 00:46:03.904 Patrick Trainer: schemas.
648 00:46:08.550 ⇒ 00:46:10.430 Patrick Trainer: That’s what makes
649 00:46:10.590 ⇒ 00:46:11.939 Patrick Trainer: sense to me.
650 00:46:12.120 ⇒ 00:46:12.540 Bryce Codell: Yeah.
651 00:46:16.070 ⇒ 00:46:22.619 Bryce Codell: So, okay, so what’s the database schema name for Spot?
652 00:46:22.620 ⇒ 00:46:24.060 Patrick Trainer: Yeah, let’s actually go.
653 00:46:24.060 ⇒ 00:46:31.390 Bryce Codell: And then both for both for raw data. But then also for like a dev raw environment, like, if I’m gonna
654 00:46:32.420 ⇒ 00:46:36.680 Bryce Codell: run like a test where I just like insert, like a couple of records that
655 00:46:36.890 ⇒ 00:46:39.679 Bryce Codell: make sure that things are working correctly.
656 00:46:39.910 ⇒ 00:46:41.330 Patrick Trainer: By all means
657 00:46:42.670 ⇒ 00:46:44.050 Patrick Trainer: create your
658 00:46:44.240 ⇒ 00:46:47.131 Patrick Trainer: stuff as you see fit.
659 00:46:48.140 ⇒ 00:46:49.169 Patrick Trainer: we’re kind of.
660 00:46:50.160 ⇒ 00:46:52.430 Patrick Trainer: We’re shooting from the hip very much.
661 00:46:52.430 ⇒ 00:46:53.210 Bryce Codell: Very nice.
662 00:46:53.618 ⇒ 00:46:55.660 Patrick Trainer: That’s that’s that’s for sure.
663 00:46:56.903 ⇒ 00:46:58.509 Patrick Trainer: And so.
664 00:47:02.540 ⇒ 00:47:06.114 Bryce Codell: So there’s brain 4. Gtl, there’s.
665 00:47:06.710 ⇒ 00:47:14.700 Patrick Trainer: Yeah. So this like, dbt, database is like the main kind of like that that transformed
666 00:47:15.378 ⇒ 00:47:16.872 Patrick Trainer: and then the
667 00:47:17.690 ⇒ 00:47:20.709 Patrick Trainer: aptly name, 5 trend. Database is
668 00:47:21.200 ⇒ 00:47:22.240 Patrick Trainer: the raw.
669 00:47:22.690 ⇒ 00:47:23.110 Bryce Codell: S-.
670 00:47:24.980 ⇒ 00:47:29.540 Patrick Trainer: like 5 trans dumping all into here. But this is
671 00:47:30.220 ⇒ 00:47:31.050 Patrick Trainer: what
672 00:47:32.080 ⇒ 00:47:35.549 Patrick Trainer: yeah, this is. This is where I would suggest dumping that
673 00:47:35.670 ⇒ 00:47:36.170 Patrick Trainer: I.
674 00:47:36.550 ⇒ 00:47:37.310 Patrick Trainer: Putting all of that.
675 00:47:37.310 ⇒ 00:47:44.209 Bryce Codell: So, even if it’s like a custom script, and it’s not, and it’s in like generated like a custom, python script. It’s not 5 trend
676 00:47:44.230 ⇒ 00:47:47.079 Bryce Codell: generated. It’s we should still dump it there.
677 00:47:47.260 ⇒ 00:47:47.705 Patrick Trainer: I
678 00:47:48.950 ⇒ 00:47:50.940 Patrick Trainer: by all means like, don’t
679 00:47:51.560 ⇒ 00:47:57.969 Patrick Trainer: feel constrained in that sense to do it. There, I think, like there is a better way.
680 00:47:58.565 ⇒ 00:47:59.570 Patrick Trainer: To do it.
681 00:48:01.490 ⇒ 00:48:04.379 Patrick Trainer: like just to keep things contained.
682 00:48:06.510 ⇒ 00:48:08.010 Patrick Trainer: But as far as like.
683 00:48:08.790 ⇒ 00:48:11.219 Patrick Trainer: do we have a structured
684 00:48:11.500 ⇒ 00:48:13.790 Patrick Trainer: process of like this is where
685 00:48:13.860 ⇒ 00:48:18.070 Patrick Trainer: things need to go. Yeah, that process
686 00:48:19.730 ⇒ 00:48:25.490 Patrick Trainer: has been like, it’s like one of those like, Oh, yeah, we should definitely do that. And then it’s like.
687 00:48:26.080 ⇒ 00:48:33.069 Patrick Trainer: okay. But so like, we’ll get to that. Yeah. And so like, we’re kind of getting like, I guess, yeah, we’re at that point now
688 00:48:33.604 ⇒ 00:48:38.459 Patrick Trainer: where that actually needs to happen, and like I’ll show you
689 00:48:38.640 ⇒ 00:48:40.537 Patrick Trainer: like these roles.
690 00:48:41.430 ⇒ 00:48:47.359 Patrick Trainer: you know. We’ll even go into there like this is where the graph go. There we go
691 00:48:48.030 ⇒ 00:48:49.690 Patrick Trainer: like it’s all
692 00:48:49.840 ⇒ 00:48:52.230 Patrick Trainer: a. It’s all a mess like.
693 00:48:53.900 ⇒ 00:48:56.820 Patrick Trainer: Sysadmin is inherited by Dbt.
694 00:48:59.390 ⇒ 00:49:01.240 Patrick Trainer: This guy’s got his own role.
695 00:49:02.300 ⇒ 00:49:04.870 Patrick Trainer: 5. Trans. Inherited from Sysadmin.
696 00:49:05.200 ⇒ 00:49:06.389 Bryce Codell: That seems correct.
697 00:49:06.610 ⇒ 00:49:09.890 Patrick Trainer: It’s yeah. But it should be.
698 00:49:09.890 ⇒ 00:49:11.739 Bryce Codell: Sys Admin is above dbt.
699 00:49:12.110 ⇒ 00:49:16.599 Patrick Trainer: Yeah, and it’s it’s all. It’s all kinda a mess.
700 00:49:16.600 ⇒ 00:49:18.120 Bryce Codell: Divide, but no like, yeah.
701 00:49:18.120 ⇒ 00:49:18.470 Patrick Trainer: Is, a.
702 00:49:18.470 ⇒ 00:49:29.660 Bryce Codell: Totally fine. It’ll like those things get cleaned up over time. It’s just those are the difficult things to clean up, because they will cause pipelines to fail unintentionally well, like.
703 00:49:30.010 ⇒ 00:49:30.560 Patrick Trainer: Yep.
704 00:49:30.560 ⇒ 00:49:31.970 Bryce Codell: Yeah, so, yeah.
705 00:49:32.440 ⇒ 00:49:33.580 Patrick Trainer: Is it.
706 00:49:34.550 ⇒ 00:49:36.110 Bryce Codell: So okay.
707 00:49:36.840 ⇒ 00:49:38.140 Bryce Codell: So
708 00:49:40.100 ⇒ 00:49:41.880 Bryce Codell: if I want to go, if you’re.
709 00:49:41.880 ⇒ 00:49:43.480 Patrick Trainer: For like.
710 00:49:43.540 ⇒ 00:49:49.669 Patrick Trainer: So this Dbt role is like what we what you would think of as like
711 00:49:50.150 ⇒ 00:49:54.039 Patrick Trainer: the canonical like, transformer role, where you can
712 00:49:54.860 ⇒ 00:50:00.250 Patrick Trainer: basically like where you can read from raw and then write to everywhere else.
713 00:50:02.180 ⇒ 00:50:04.329 Patrick Trainer: so this is, you’d
714 00:50:04.380 ⇒ 00:50:11.619 Patrick Trainer: be great to to use this and kind of have like free range. There. So like
715 00:50:12.810 ⇒ 00:50:14.720 Patrick Trainer: again, by all means like.
716 00:50:16.200 ⇒ 00:50:17.000 Patrick Trainer: create
717 00:50:17.890 ⇒ 00:50:19.849 Patrick Trainer: database schema away
718 00:50:20.010 ⇒ 00:50:21.989 Patrick Trainer: as you need, and then
719 00:50:22.350 ⇒ 00:50:23.470 Patrick Trainer: we can
720 00:50:24.070 ⇒ 00:50:25.260 Patrick Trainer: figure out
721 00:50:26.100 ⇒ 00:50:27.590 Patrick Trainer: how to weave that in.
722 00:50:27.870 ⇒ 00:50:28.250 Bryce Codell: Yeah.
723 00:50:28.250 ⇒ 00:50:29.050 Patrick Trainer: It’s like.
724 00:50:29.450 ⇒ 00:50:39.119 Patrick Trainer: I really like this because it’s like forcing us now to like address this because it’s been needed to be like this is some like super serious tech deck
725 00:50:40.012 ⇒ 00:50:43.150 Patrick Trainer: and like, I’m I’m very glad that we’re
726 00:50:43.530 ⇒ 00:50:54.759 Patrick Trainer: having this conversation now to to make things like actually make sense, because, as I’m even trying to explain, kind of like the the nuance to it.
727 00:50:54.900 ⇒ 00:50:57.589 Patrick Trainer: it’s like it. It’s it’s pretty
728 00:50:57.780 ⇒ 00:50:58.810 Patrick Trainer: out there.
729 00:50:58.810 ⇒ 00:51:02.720 Bryce Codell: Yeah, no. I mean, all this stuff makes sense like, this is like.
730 00:51:02.957 ⇒ 00:51:03.670 Patrick Trainer: Yeah, you’re right.
731 00:51:03.670 ⇒ 00:51:13.010 Bryce Codell: Setup is honestly, totally reasonable. I’m asking purely from the standpoint of like trying to tease out what I like, how I should
732 00:51:13.100 ⇒ 00:51:18.470 Bryce Codell: shape my own dev workflow as I’m going through things so that I’m
733 00:51:18.720 ⇒ 00:51:23.660 Bryce Codell: limiting the amount of toe stepping I’m doing in the process. Yeah.
734 00:51:23.955 ⇒ 00:51:26.020 Patrick Trainer: Like, I don’t think you’re gonna be
735 00:51:26.850 ⇒ 00:51:29.720 Patrick Trainer: really stepping on any toes.
736 00:51:30.653 ⇒ 00:51:32.079 Patrick Trainer: So like.
737 00:51:35.660 ⇒ 00:51:37.480 Patrick Trainer: like, if I were to
738 00:51:44.530 ⇒ 00:51:54.009 Patrick Trainer: like basically like, the entire workflow is just of all like issues and and prs, like kind of like that standard like Get flow
739 00:51:55.100 ⇒ 00:51:56.720 Patrick Trainer: but like working
740 00:51:56.920 ⇒ 00:51:57.990 Patrick Trainer: in
741 00:51:58.150 ⇒ 00:52:00.350 Patrick Trainer: like Dbt project here.
742 00:52:00.350 ⇒ 00:52:01.330 Bryce Codell: Like you’re.
743 00:52:01.910 ⇒ 00:52:05.737 Patrick Trainer: Not gonna be stepping on any toes of like
744 00:52:06.650 ⇒ 00:52:09.089 Patrick Trainer: come to models. I have like a dummy
745 00:52:10.230 ⇒ 00:52:12.500 Patrick Trainer: test. SQL, here.
746 00:52:12.500 ⇒ 00:52:12.880 Bryce Codell: Yeah.
747 00:52:13.295 ⇒ 00:52:13.710 Patrick Trainer: So
748 00:52:15.070 ⇒ 00:52:19.900 Patrick Trainer: we’ll just change that to a hundred. And
749 00:52:20.260 ⇒ 00:52:22.870 Patrick Trainer: well, actually, I need to.
750 00:52:23.920 ⇒ 00:52:27.409 Patrick Trainer: You’re there, and we’ll pull that.
751 00:52:29.950 ⇒ 00:52:32.059 Patrick Trainer: I hate these little zoom windows.
752 00:52:33.630 ⇒ 00:52:35.660 Patrick Trainer: block everything.
753 00:52:37.590 ⇒ 00:52:39.400 Bryce Codell: Yeah, they’re a little annoying.
754 00:52:40.003 ⇒ 00:52:41.210 Patrick Trainer: Trade branch.
755 00:52:43.650 ⇒ 00:52:44.760 Patrick Trainer: do that.
756 00:52:49.440 ⇒ 00:52:50.640 Patrick Trainer: I’ll just like
757 00:52:52.200 ⇒ 00:52:53.020 Patrick Trainer: test
758 00:52:54.760 ⇒ 00:52:55.620 Patrick Trainer: change.
759 00:52:59.120 ⇒ 00:53:00.580 Patrick Trainer: Make 100
760 00:53:02.350 ⇒ 00:53:03.290 Patrick Trainer: that
761 00:53:03.650 ⇒ 00:53:05.060 Patrick Trainer: we can push all this up.
762 00:53:12.240 ⇒ 00:53:13.719 Patrick Trainer: We’ll create that Pr.
763 00:53:19.200 ⇒ 00:53:20.759 Patrick Trainer: what this is.
764 00:53:24.750 ⇒ 00:53:25.600 Patrick Trainer: Pr.
765 00:53:27.330 ⇒ 00:53:30.190 Patrick Trainer: and then what we can do is actually come into here.
766 00:53:30.980 ⇒ 00:53:33.010 Patrick Trainer: So we’ll see.
767 00:53:45.150 ⇒ 00:53:46.210 Patrick Trainer: So here.
768 00:53:46.670 ⇒ 00:53:50.150 Patrick Trainer: this getting that artifact workflow, id, and.
769 00:53:50.150 ⇒ 00:53:50.530 Bryce Codell: In, the.
770 00:53:50.530 ⇒ 00:53:51.720 Patrick Trainer: And pass this
771 00:53:52.350 ⇒ 00:53:56.270 Patrick Trainer: pass that Id. To get the art to download the artifact
772 00:53:57.090 ⇒ 00:53:59.639 Patrick Trainer: from here which it has
773 00:54:01.230 ⇒ 00:54:02.290 Patrick Trainer: aspired.
774 00:54:07.870 ⇒ 00:54:09.750 Patrick Trainer: and when that fails.
775 00:54:10.740 ⇒ 00:54:12.190 Bryce Codell: Then it just runs. If if.
776 00:54:12.190 ⇒ 00:54:15.030 Patrick Trainer: Falls back and runs it normally.
777 00:54:15.030 ⇒ 00:54:15.820 Bryce Codell: Nice.
778 00:54:16.100 ⇒ 00:54:17.630 Patrick Trainer: Yeah.
779 00:54:20.803 ⇒ 00:54:25.460 Patrick Trainer: and and so, okay, and so and then, so what this is. Once this is done running
780 00:54:28.330 ⇒ 00:54:31.830 Patrick Trainer: it will upload an artifact again.
781 00:54:32.588 ⇒ 00:54:35.540 Patrick Trainer: And then we’ll be able to kind of continue.
782 00:54:35.780 ⇒ 00:54:36.440 Bryce Codell: Yen.
783 00:54:37.180 ⇒ 00:54:38.430 Bryce Codell: Yeah, this is, I think.
784 00:54:38.430 ⇒ 00:54:39.560 Patrick Trainer: This is what, yeah.
785 00:54:39.850 ⇒ 00:54:44.749 Patrick Trainer: Google ads, Microsoft, ads, pinterest, we’re not using pinterest reddit. Nope.
786 00:54:44.750 ⇒ 00:54:47.689 Bryce Codell: I was like. Damn! I don’t think I saw these.
787 00:54:47.690 ⇒ 00:54:50.400 Patrick Trainer: Yeah, not not using any of that.
788 00:54:50.530 ⇒ 00:54:51.200 Bryce Codell: Yen.
789 00:54:52.420 ⇒ 00:54:55.740 Bryce Codell: What was the name of the package that auto installs? All of these.
790 00:54:56.410 ⇒ 00:54:57.080 Patrick Trainer: It’s.
791 00:54:59.040 ⇒ 00:54:59.810 Bryce Codell: Add reporting.
792 00:54:59.810 ⇒ 00:55:03.170 Patrick Trainer: Yeah, it’s this 5 Chan add reporting
793 00:55:05.620 ⇒ 00:55:07.060 Bryce Codell: Oh, 5 trend.
794 00:55:07.550 ⇒ 00:55:08.260 Patrick Trainer: Right.
795 00:55:08.840 ⇒ 00:55:09.185 Bryce Codell: Yeah.
796 00:55:10.690 ⇒ 00:55:12.490 Bryce Codell: Have you tried out? Moodle?
797 00:55:13.250 ⇒ 00:55:14.070 Patrick Trainer: No.
798 00:55:14.960 ⇒ 00:55:16.420 Bryce Codell: Do you know what modal is?
799 00:55:16.865 ⇒ 00:55:17.309 Patrick Trainer: Don’t!
800 00:55:17.580 ⇒ 00:55:19.309 Bryce Codell: Modal.com.
801 00:55:19.900 ⇒ 00:55:20.490 Patrick Trainer: That’s.
802 00:55:21.720 ⇒ 00:55:23.610 Bryce Codell: That’s a that’s a porn website.
803 00:55:28.170 ⇒ 00:55:29.800 Patrick Trainer: They are teams from China.
804 00:55:29.800 ⇒ 00:55:31.170 Bryce Codell: And then.
805 00:55:32.206 ⇒ 00:55:32.900 Patrick Trainer: This cool.
806 00:55:32.900 ⇒ 00:55:34.710 Bryce Codell: It’s like lambdas
807 00:55:34.750 ⇒ 00:55:43.119 Bryce Codell: on demand. But any but the underlying. But there’s like no cold start issue, and the underlying machine can be a tiny like
808 00:55:43.300 ⇒ 00:55:48.800 Bryce Codell: order, virtual CPU to like the biggest Nvidia Gpu enabled machine. Yeah.
809 00:55:48.800 ⇒ 00:55:49.630 Patrick Trainer: Oh, wow!
810 00:55:49.630 ⇒ 00:55:52.550 Bryce Codell: Cloud and so
811 00:55:52.700 ⇒ 00:55:54.759 Bryce Codell: like, I thought a lot about this
812 00:55:55.050 ⇒ 00:55:56.634 Bryce Codell: product for or like.
813 00:55:57.030 ⇒ 00:55:57.400 Patrick Trainer: Yeah.
814 00:55:57.400 ⇒ 00:56:03.049 Bryce Codell: Cool for a lot of different things like web hook based data syncs and stuff for one. But like.
815 00:56:03.430 ⇒ 00:56:10.403 Bryce Codell: The other one that honestly could work really nicely is like it provides storage and would work great as
816 00:56:10.890 ⇒ 00:56:13.297 Bryce Codell: artifact storage for
817 00:56:14.230 ⇒ 00:56:15.880 Bryce Codell: something like Dbt.
818 00:56:15.910 ⇒ 00:56:21.909 Bryce Codell: Where, instead of having to like spin up an S. 3 bucket, and like to deal with all the.
819 00:56:21.910 ⇒ 00:56:22.700 Patrick Trainer: Right.
820 00:56:22.700 ⇒ 00:56:27.549 Bryce Codell: Crap. It’s just like, write a python like a small python script. And then.
821 00:56:28.200 ⇒ 00:56:30.649 Bryce Codell: like, Yeah, hmm.
822 00:56:31.060 ⇒ 00:56:34.429 Bryce Codell: push and pull out of your like. Manifest as you need them.
823 00:56:34.430 ⇒ 00:56:35.390 Patrick Trainer: Right.
824 00:56:36.230 ⇒ 00:56:43.460 Patrick Trainer: That’s that makes a lot of sense. That’s so. That’s 1 thing about github. That
825 00:56:43.800 ⇒ 00:56:51.370 Patrick Trainer: gitlab is like eating their lunch at is gitlab stores their artifacts at the repo level.
826 00:56:52.020 ⇒ 00:56:53.630 Patrick Trainer: And so like.
827 00:56:53.760 ⇒ 00:57:00.889 Patrick Trainer: everything’s kind of like, always available to all jobs which are nice. Github has that.
828 00:57:01.080 ⇒ 00:57:08.549 Patrick Trainer: It’s that it’s pinned to a particular workflow like I mean, I guess I get it, but
829 00:57:08.980 ⇒ 00:57:12.909 Patrick Trainer: I feel like there should be a level above that where
830 00:57:13.650 ⇒ 00:57:17.499 Patrick Trainer: it shouldn’t be such a pain to to grab that.
831 00:57:17.500 ⇒ 00:57:18.150 Bryce Codell: Yeah.
832 00:57:18.690 ⇒ 00:57:20.179 Patrick Trainer: Programmatically. You know.
833 00:57:20.180 ⇒ 00:57:27.310 Bryce Codell: Yeah, that makes sense. But, generally speaking, I should not just like start throwing a clients like
834 00:57:27.510 ⇒ 00:57:34.739 Bryce Codell: production content into some other 3rd party tool without getting some sort of approval. Right?
835 00:57:36.136 ⇒ 00:57:37.330 Patrick Trainer: I mean.
836 00:57:38.370 ⇒ 00:57:39.940 Patrick Trainer: maybe
837 00:57:43.700 ⇒ 00:57:46.749 Bryce Codell: Yeah, I think for now, what I’ll do is I’ll
838 00:57:47.475 ⇒ 00:57:51.474 Bryce Codell: so the 2 things I need to figure out are like, how am I gonna manage like
839 00:57:51.800 ⇒ 00:57:56.628 Bryce Codell: like building dbt, like building dbt, models that I wanna that I need to build.
840 00:57:57.540 ⇒ 00:58:03.150 Bryce Codell: And while like navigating the world of the of like
841 00:58:03.290 ⇒ 00:58:09.640 Bryce Codell: dev schemas and production and like production schemas and tables. And then how?
842 00:58:09.810 ⇒ 00:58:12.390 Bryce Codell: And so the way I think the way that I’ll do that is by like
843 00:58:12.490 ⇒ 00:58:24.737 Bryce Codell: creating a mock prod environment target with my own credentials, and then just like being cautious and not and making sure I don’t actually accidentally run like Dbt. Run on that.
844 00:58:25.755 ⇒ 00:58:28.410 Bryce Codell: So that I can leverage the defer functionality. Give you.
845 00:58:28.410 ⇒ 00:58:28.840 Patrick Trainer: Right.
846 00:58:28.840 ⇒ 00:58:32.840 Bryce Codell: Functionality and then on the like new data source side.
847 00:58:32.860 ⇒ 00:58:48.450 Bryce Codell: I’ll try to bias towards just like finding some weather data on the Snowflake marketplace. But if I can’t, or if there’s like, there’s an Api that’s really compelling, I’ll write a python script, and then I’ll like just ping you, and we can have another chat about.
848 00:58:48.450 ⇒ 00:58:49.110 Patrick Trainer: Yeah.
849 00:58:49.470 ⇒ 00:58:51.430 Bryce Codell: Best path forward for for.
850 00:58:51.430 ⇒ 00:58:51.960 Patrick Trainer: Yeah.
851 00:58:51.960 ⇒ 00:58:52.590 Bryce Codell: Him.
852 00:58:52.760 ⇒ 00:58:53.719 Patrick Trainer: Yeah, that.
853 00:58:55.480 ⇒ 00:58:57.090 Patrick Trainer: Yeah, that sounds really good.
854 00:58:57.220 ⇒ 00:58:58.360 Bryce Codell: Okay. Cool. Cool.
855 00:58:59.750 ⇒ 00:59:08.680 Patrick Trainer: and I do have like docs and diagrams of all of the stuff that I just showed you. But I am blanking on
856 00:59:09.220 ⇒ 00:59:13.099 Patrick Trainer: where that actually is. So that’s like another
857 00:59:14.070 ⇒ 00:59:18.004 Patrick Trainer: like tech. That thing is getting everything in the right place.
858 00:59:19.680 ⇒ 00:59:22.480 Patrick Trainer: but I will come back to
859 00:59:22.670 ⇒ 00:59:24.090 Patrick Trainer: get that
860 00:59:25.600 ⇒ 00:59:28.699 Patrick Trainer: But here is like an example of
861 00:59:28.970 ⇒ 00:59:32.569 Patrick Trainer: like. After this, ran we uploaded that artifact.
862 00:59:32.690 ⇒ 00:59:33.610 Patrick Trainer: Then.
863 00:59:33.760 ⇒ 00:59:36.029 Patrick Trainer: like you like, click, that link.
864 00:59:36.030 ⇒ 00:59:37.790 Bryce Codell: Yep, and now you’ve got the manifest.
865 00:59:37.790 ⇒ 00:59:40.700 Patrick Trainer: Yeah, now we got the manifest. But we’ve also got like
866 00:59:40.790 ⇒ 00:59:42.670 Patrick Trainer: the graph summary and.
867 00:59:43.870 ⇒ 00:59:44.500 Patrick Trainer: That.
868 00:59:45.150 ⇒ 00:59:51.739 Bryce Codell: I have. I’ve never looked into any of the other artifacts that Dbt. Publishes. Besides, the manifest.
869 00:59:51.740 ⇒ 00:59:56.270 Patrick Trainer: Yeah, I’ve never looked at semantic or or graph, like, those are pretty new.
870 00:59:56.420 ⇒ 00:59:58.399 Patrick Trainer: I’ve used run results
871 00:59:58.630 ⇒ 01:00:01.210 Patrick Trainer: before, but like, not
872 01:00:02.740 ⇒ 01:00:04.186 Patrick Trainer: not much.
873 01:00:07.790 ⇒ 01:00:10.970 Patrick Trainer: But yeah, the manifest is is kind of where it is.
874 01:00:10.970 ⇒ 01:00:12.240 Bryce Codell: Hmm! But.
875 01:00:12.240 ⇒ 01:00:23.759 Patrick Trainer: Yeah. And spanifest is absolutely huge, too, because of yeah, almost 10 Megs. Because of all of those like bullshit.
876 01:00:24.370 ⇒ 01:00:31.249 Patrick Trainer: pinterest and Snapchat packages that are being loaded into there, and like all of the
877 01:00:31.340 ⇒ 01:00:36.430 Patrick Trainer: stuff being associated there. So it’s it’s it’s polluting the manifest like
878 01:00:37.080 ⇒ 01:00:39.219 Patrick Trainer: to an insane degree.
879 01:00:39.220 ⇒ 01:00:41.459 Bryce Codell: That is very irritating.
880 01:00:41.600 ⇒ 01:00:44.006 Bryce Codell: But so, okay, so just to confirm.
881 01:00:44.350 ⇒ 01:00:47.129 Patrick Trainer: And you can see the manifest down here, too.
882 01:00:47.130 ⇒ 01:00:50.019 Bryce Codell: So then one last thing. So
883 01:00:51.090 ⇒ 01:00:54.960 Bryce Codell: it’s all 5 tran for pool parts. No more. Nexela.
884 01:00:57.000 ⇒ 01:00:58.819 Patrick Trainer: Correct. Yes.
885 01:00:58.820 ⇒ 01:00:59.540 Bryce Codell: Okay.
886 01:00:59.800 ⇒ 01:01:09.499 Bryce Codell: alright. We’re catching up with the the Aes in a few hours, so I’ll we can ask them again, just to confirm. But if you’re not totally sure but.
887 01:01:09.500 ⇒ 01:01:12.919 Patrick Trainer: Yeah, I’m not a hundred percent. But I’m
888 01:01:12.950 ⇒ 01:01:16.909 Patrick Trainer: 95%. Yeah, on the
889 01:01:18.800 ⇒ 01:01:20.070 Patrick Trainer: just 5 train thing.
890 01:01:20.070 ⇒ 01:01:30.874 Bryce Codell: Yeah, okay, cool. We can always double check with your time as well. But say, this is awesome, very impressive setup the whole componentized workflow and action stuff is super cool.
891 01:01:31.550 ⇒ 01:01:33.909 Patrick Trainer: Yeah, it’ll it’ll I. I’m
892 01:01:34.240 ⇒ 01:01:41.259 Patrick Trainer: like bummed that I don’t have this diagram at my fingertips. I’ve run into this like twice. Now, this is the the second time. But
893 01:01:41.780 ⇒ 01:01:43.249 Patrick Trainer: I will. I will get that
894 01:01:43.550 ⇒ 01:01:46.939 Patrick Trainer: to you, and it’ll it’ll make a lot more sense.
895 01:01:47.760 ⇒ 01:01:57.554 Bryce Codell: Yeah, sounds good. Nico. I totally domineered this conversation. I’m very sorry about that. Did you have questions about the any of the stuff that we were going through.
896 01:01:58.440 ⇒ 01:02:12.020 Nicolas Sucari: No, I think everything was all pretty clear from the explanation from Patrick. It’s kind of difficult for me to understand everything, but it was great to have that walkthrough. What I was gonna ask Patrick, is probably to
897 01:02:12.590 ⇒ 01:02:19.290 Nicolas Sucari: jump something into like that new data source like right now we are doing some like manual work
898 01:02:19.330 ⇒ 01:02:25.479 Nicolas Sucari: integrating like the emails that we’re receiving for pull parts and that kind of stuff. I know you were trying to automate that.
899 01:02:25.560 ⇒ 01:02:35.630 Nicolas Sucari: But probably we. I don’t know if price is doing the same, or we’ll need to do kind of the same of what Brian is doing, adding that information right?
900 01:02:36.110 ⇒ 01:02:38.380 Nicolas Sucari: Probably need to understand that workflow. Yeah.
901 01:02:38.380 ⇒ 01:02:44.930 Patrick Trainer: Yeah. So so that’s actually, that’s actually happening now. And I can like I, I
902 01:02:45.060 ⇒ 01:02:46.459 Patrick Trainer: have all of that.
903 01:02:47.400 ⇒ 01:02:55.000 Nicolas Sucari: Because when you, when when you, Brice, we’re talking about like new data source, or where to add that we are doing like some manual work and.
904 01:02:55.000 ⇒ 01:02:56.750 Patrick Trainer: It is similar, so.
905 01:02:56.750 ⇒ 01:02:57.100 Nicolas Sucari: Yeah.
906 01:02:57.552 ⇒ 01:03:03.880 Patrick Trainer: For contacts price the every day or every couple of days. There’s like a
907 01:03:04.300 ⇒ 01:03:14.070 Patrick Trainer: static, basically like a seed file that gets emailed to a slack channel like you know how slack has like dedicated emails for.
908 01:03:14.070 ⇒ 01:03:14.410 Bryce Codell: Yep.
909 01:03:14.984 ⇒ 01:03:16.134 Patrick Trainer: A channel
910 01:03:16.800 ⇒ 01:03:22.380 Patrick Trainer: and what it is, or the way the process. It’s so they email, a Csv, right?
911 01:03:22.720 ⇒ 01:03:30.399 Patrick Trainer: And right now, the kind of like. The process is Adam’s going in manually copy, pasting it into a Google sheet
912 01:03:31.100 ⇒ 01:03:32.079 Patrick Trainer: and then
913 01:03:32.620 ⇒ 01:03:38.659 Patrick Trainer: 5 trans, actually picking up that Google sheet and loading it into snowflake and so
914 01:03:40.470 ⇒ 01:03:41.460 Patrick Trainer: The
915 01:03:41.600 ⇒ 01:03:53.079 Patrick Trainer: need here is to basically not send it into slack and not use 5 trend to to get those small files. And so what I have
916 01:03:53.210 ⇒ 01:03:54.950 Patrick Trainer: made is this.
917 01:03:55.160 ⇒ 01:03:56.690 Patrick Trainer: it’s called G mad.
918 01:03:57.486 ⇒ 01:04:00.109 Patrick Trainer: Says Gmail. Attachment downloader.
919 01:04:00.110 ⇒ 01:04:02.273 Bryce Codell: I saw this repo I was wondering what it was.
920 01:04:02.490 ⇒ 01:04:10.400 Patrick Trainer: And and so what this does is this parses
921 01:04:11.600 ⇒ 01:04:15.259 Patrick Trainer: like a Gmail inbox for
922 01:04:17.543 ⇒ 01:04:21.806 Patrick Trainer: Emails with attachments, and then grabs those attachments.
923 01:04:22.678 ⇒ 01:04:25.860 Patrick Trainer: And so I can. Actually, I can show you that.
924 01:04:28.110 ⇒ 01:04:30.540 Nicolas Sucari: Yeah, I thought this was like worth
925 01:04:30.610 ⇒ 01:04:32.459 Nicolas Sucari: to share with Bryce. Probably.
926 01:04:32.810 ⇒ 01:04:33.320 Patrick Trainer: Whoops.
927 01:04:33.320 ⇒ 01:04:40.659 Nicolas Sucari: But we create. It’s it’s a new email account that we created. I think it’s brain forge receiving right or something like that.
928 01:04:40.660 ⇒ 01:04:41.500 Patrick Trainer: Yeah.
929 01:04:54.060 ⇒ 01:04:55.230 Patrick Trainer: So
930 01:04:56.680 ⇒ 01:04:58.110 Patrick Trainer: what we have
931 01:04:59.680 ⇒ 01:05:00.750 Patrick Trainer: is
932 01:05:02.240 ⇒ 01:05:02.930 Patrick Trainer: Wooke.
933 01:05:05.100 ⇒ 01:05:06.619 Patrick Trainer: I don’t know how that got there.
934 01:05:10.330 ⇒ 01:05:12.080 Patrick Trainer: What we have
935 01:05:13.760 ⇒ 01:05:16.370 Patrick Trainer: is, and I can oops.
936 01:05:17.390 ⇒ 01:05:19.690 Patrick Trainer: Oh, pop that out there.
937 01:05:20.732 ⇒ 01:05:29.990 Patrick Trainer: So we’re wanting to download attachments from Gmail based on search criteria. And so what this is doing is it allows you to
938 01:05:30.589 ⇒ 01:05:35.419 Patrick Trainer: this is like a service account that is able to impersonate
939 01:05:36.169 ⇒ 01:05:51.180 Patrick Trainer: other accounts in the brain forge domain. And query their inbox for specific files. So just like in Gmail and the little query box you have these like query operators
940 01:05:51.611 ⇒ 01:05:54.660 Patrick Trainer: where you can just like in slack like
941 01:05:54.710 ⇒ 01:06:07.019 Patrick Trainer: define where things are from as well as like looking for specific files. These file names Pdfs to kind of like, limit your your
942 01:06:07.760 ⇒ 01:06:11.240 Patrick Trainer: idea or to limit your your search results.
943 01:06:12.216 ⇒ 01:06:13.509 Patrick Trainer: And you
944 01:06:14.410 ⇒ 01:06:17.139 Patrick Trainer: this, it’ll spit out
945 01:06:18.620 ⇒ 01:06:21.819 Patrick Trainer: the like. Those downloaded files.
946 01:06:21.820 ⇒ 01:06:22.170 Bryce Codell: Yeah.
947 01:06:22.170 ⇒ 01:06:24.619 Patrick Trainer: And so the way this is working
948 01:06:25.020 ⇒ 01:06:26.400 Patrick Trainer: is
949 01:06:26.660 ⇒ 01:06:28.860 Patrick Trainer: the best way to
950 01:06:29.190 ⇒ 01:06:29.920 Patrick Trainer: it’s on it.
951 01:06:32.430 ⇒ 01:06:38.200 Patrick Trainer: And so we have. I can walk through the classes later. But.
952 01:06:40.600 ⇒ 01:06:43.620 Patrick Trainer: Come through here. Let me actually, I will
953 01:06:46.890 ⇒ 01:06:48.300 Patrick Trainer: doing all those.
954 01:06:49.890 ⇒ 01:06:50.740 Patrick Trainer: So
955 01:06:50.930 ⇒ 01:06:51.790 Patrick Trainer: this
956 01:06:51.980 ⇒ 01:06:53.049 Patrick Trainer: just like a
957 01:07:02.040 ⇒ 01:07:02.760 Patrick Trainer: woof
958 01:07:03.900 ⇒ 01:07:06.063 Patrick Trainer: I’ll get. Let me get back to you on this
959 01:07:06.850 ⇒ 01:07:08.210 Patrick Trainer: again. Live Demos.
960 01:07:08.210 ⇒ 01:07:09.010 Bryce Codell: So long.
961 01:07:09.010 ⇒ 01:07:13.919 Patrick Trainer: But so, so essentially like, what we’re, what we’re doing is like, we
962 01:07:14.020 ⇒ 01:07:15.200 Patrick Trainer: have this
963 01:07:16.010 ⇒ 01:07:17.670 Patrick Trainer: Gmail service right.
964 01:07:18.020 ⇒ 01:07:19.930 Patrick Trainer: And so the Gmail service
965 01:07:21.435 ⇒ 01:07:24.659 Patrick Trainer: gets passed to
966 01:07:25.125 ⇒ 01:07:27.960 Patrick Trainer: like, we have this like downloader class.
967 01:07:28.450 ⇒ 01:07:31.280 Patrick Trainer: where this is where, like the actual
968 01:07:31.410 ⇒ 01:07:33.460 Patrick Trainer: initializations happening.
969 01:07:33.919 ⇒ 01:07:38.309 Patrick Trainer: So we have the the Gmail service a message fetcher
970 01:07:38.530 ⇒ 01:07:48.219 Patrick Trainer: an attachment processor. And then we have these different like sub classes here for different things that we’re wanting to download.
971 01:07:48.220 ⇒ 01:07:48.930 Bryce Codell: Death.
972 01:07:48.930 ⇒ 01:07:49.960 Patrick Trainer: And so
973 01:07:50.560 ⇒ 01:07:51.620 Patrick Trainer: from
974 01:07:51.750 ⇒ 01:07:55.650 Patrick Trainer: this downloader class, like we can go. We already saw the
975 01:07:56.040 ⇒ 01:08:01.529 Patrick Trainer: the service there, and that does this handles all of the authentication
976 01:08:02.390 ⇒ 01:08:03.000 Patrick Trainer: and
977 01:08:04.550 ⇒ 01:08:07.310 Patrick Trainer: based on environment variables
978 01:08:07.340 ⇒ 01:08:09.139 Patrick Trainer: determines like
979 01:08:09.880 ⇒ 01:08:10.980 Patrick Trainer: what you’re
980 01:08:12.360 ⇒ 01:08:23.239 Patrick Trainer: basically authenticating ass. If, like as a user with local scopes, oh, that’s why it didn’t run a user with local credentials or
981 01:08:23.270 ⇒ 01:08:24.610 Patrick Trainer: a Github
982 01:08:24.859 ⇒ 01:08:28.510 Patrick Trainer: service account that has all of their stuff in.
983 01:08:29.006 ⇒ 01:08:30.000 Bryce Codell: That that.
984 01:08:30.000 ⇒ 01:08:32.699 Patrick Trainer: The context of a a github runner
985 01:08:33.149 ⇒ 01:08:34.300 Patrick Trainer: urb.
986 01:08:35.180 ⇒ 01:08:38.830 Patrick Trainer: And so we have this message feature class.
987 01:08:38.840 ⇒ 01:08:43.618 Patrick Trainer: which this is where it’s actually like parsing through
988 01:08:44.290 ⇒ 01:08:47.469 Patrick Trainer: in querying the the Gmail Api.
989 01:08:47.630 ⇒ 01:08:50.669 Patrick Trainer: And so this gets past that service
990 01:08:50.819 ⇒ 01:08:52.350 Patrick Trainer: and the the
991 01:08:52.630 ⇒ 01:08:55.319 Patrick Trainer: the service is like the that it
992 01:08:55.350 ⇒ 01:08:58.060 Patrick Trainer: the service is what’s authenticating to
993 01:08:58.420 ⇒ 01:09:07.679 Patrick Trainer: Google and Gmail, and then provides the like, the Api that we’re using to fetch these messages. And so for here
994 01:09:08.910 ⇒ 01:09:14.620 Patrick Trainer: we’re able to get our service users. We look through a list of messages
995 01:09:15.108 ⇒ 01:09:22.869 Patrick Trainer: we can pass in a like a user user id to impersonate and then we get our Max results.
996 01:09:23.069 ⇒ 01:09:24.299 Patrick Trainer: and then
997 01:09:24.640 ⇒ 01:09:26.380 Patrick Trainer: this gets executed.
998 01:09:27.299 ⇒ 01:09:28.439 Patrick Trainer: Which then.
999 01:09:28.979 ⇒ 01:09:30.319 Patrick Trainer: if we go back
1000 01:09:30.580 ⇒ 01:09:31.769 Patrick Trainer: to the loader.
1001 01:09:33.779 ⇒ 01:09:35.169 Patrick Trainer: comes in here.
1002 01:09:36.229 ⇒ 01:09:40.789 Patrick Trainer: It’s looking for messages, goes fetches the message.
1003 01:09:40.830 ⇒ 01:09:47.060 Patrick Trainer: gets the payload, gets part the parts. This is all kind of like part looping through
1004 01:09:47.390 ⇒ 01:09:49.899 Patrick Trainer: just the Api structure of
1005 01:09:50.770 ⇒ 01:09:52.729 Patrick Trainer: the Gmail inbox
1006 01:09:53.621 ⇒ 01:09:58.148 Patrick Trainer: and then we’re looking at this. We have this attachment processor.
1007 01:10:00.963 ⇒ 01:10:02.430 Patrick Trainer: Stop here
1008 01:10:03.132 ⇒ 01:10:09.320 Patrick Trainer: where we are going, and we have this fetch attachment.
1009 01:10:09.846 ⇒ 01:10:15.680 Patrick Trainer: Which is where we’re actually able to grab that attachment and then save it
1010 01:10:15.690 ⇒ 01:10:17.100 Patrick Trainer: into our
1011 01:10:17.190 ⇒ 01:10:19.460 Patrick Trainer: like local directory.
1012 01:10:19.670 ⇒ 01:10:20.830 Patrick Trainer: which is
1013 01:10:21.120 ⇒ 01:10:22.420 Patrick Trainer: very nice.
1014 01:10:22.610 ⇒ 01:10:23.440 Bryce Codell: Down on me.
1015 01:10:23.700 ⇒ 01:10:25.130 Patrick Trainer: Let’s see
1016 01:10:29.910 ⇒ 01:10:30.580 Patrick Trainer: a.
1017 01:10:32.640 ⇒ 01:10:34.020 Patrick Trainer: And so
1018 01:10:34.040 ⇒ 01:10:35.260 Patrick Trainer: example.
1019 01:10:36.100 ⇒ 01:10:38.800 Patrick Trainer: we should be able to
1020 01:10:43.030 ⇒ 01:10:44.320 Patrick Trainer: let’s run this.
1021 01:10:55.690 ⇒ 01:10:56.609 Patrick Trainer: Here we go.
1022 01:10:56.840 ⇒ 01:10:57.980 Patrick Trainer: And so
1023 01:10:58.160 ⇒ 01:11:00.539 Patrick Trainer: here are the Pdf files
1024 01:11:02.260 ⇒ 01:11:03.090 Patrick Trainer: that
1025 01:11:03.730 ⇒ 01:11:08.650 Patrick Trainer: are currently in my Gmail inbox oops.
1026 01:11:08.650 ⇒ 01:11:09.650 Bryce Codell: And it was slick.
1027 01:11:10.420 ⇒ 01:11:11.120 Bryce Codell: chrome.
1028 01:11:12.020 ⇒ 01:11:13.630 Patrick Trainer: Gmail.
1029 01:11:14.810 ⇒ 01:11:19.629 Patrick Trainer: and so like what this is called anomalo overview.
1030 01:11:23.180 ⇒ 01:11:24.220 Patrick Trainer: Actually.
1031 01:11:25.190 ⇒ 01:11:27.781 Patrick Trainer: this will be a better
1032 01:11:28.810 ⇒ 01:11:31.829 Patrick Trainer: So we go into this fetcher wait. Now
1033 01:11:33.100 ⇒ 01:11:34.980 Patrick Trainer: let’s skip that my example.
1034 01:11:35.090 ⇒ 01:11:36.470 Patrick Trainer: So here
1035 01:11:37.210 ⇒ 01:11:40.100 Patrick Trainer: we’ve got the file name
1036 01:11:40.750 ⇒ 01:11:44.279 Patrick Trainer: at a station there, so we can check here.
1037 01:11:44.560 ⇒ 01:11:44.940 Bryce Codell: So here.
1038 01:11:44.940 ⇒ 01:11:45.540 Patrick Trainer: Here.
1039 01:11:45.930 ⇒ 01:11:48.650 Patrick Trainer: These are all the Pdf files that
1040 01:11:49.220 ⇒ 01:11:50.260 Patrick Trainer: have in
1041 01:11:51.030 ⇒ 01:11:52.650 Patrick Trainer: my email, right?
1042 01:11:53.310 ⇒ 01:11:56.520 Patrick Trainer: And so what this is doing? This is essentially passing
1043 01:11:56.830 ⇒ 01:11:58.059 Patrick Trainer: this query
1044 01:11:59.530 ⇒ 01:12:00.640 Patrick Trainer: into here.
1045 01:12:00.980 ⇒ 01:12:02.139 Patrick Trainer: looping through
1046 01:12:02.370 ⇒ 01:12:04.090 Patrick Trainer: finding the message
1047 01:12:04.540 ⇒ 01:12:06.490 Patrick Trainer: message ids. And then
1048 01:12:06.550 ⇒ 01:12:08.060 Patrick Trainer: in the
1049 01:12:08.290 ⇒ 01:12:10.150 Patrick Trainer: metadata here, there’s
1050 01:12:10.800 ⇒ 01:12:12.860 Patrick Trainer: like this has its own
1051 01:12:13.470 ⇒ 01:12:17.669 Patrick Trainer: message with its own part. Id. And that’s what
1052 01:12:17.980 ⇒ 01:12:19.120 Patrick Trainer: is
1053 01:12:20.500 ⇒ 01:12:21.740 Patrick Trainer: happening
1054 01:12:24.750 ⇒ 01:12:27.010 Patrick Trainer: somewhere in here with the part.
1055 01:12:28.105 ⇒ 01:12:29.119 Patrick Trainer: And
1056 01:12:29.320 ⇒ 01:12:31.130 Patrick Trainer: oh, yeah, it’s processing that.
1057 01:12:32.440 ⇒ 01:12:33.350 Patrick Trainer: And
1058 01:12:34.630 ⇒ 01:12:37.809 Patrick Trainer: then it spits it out into
1059 01:12:39.190 ⇒ 01:12:40.600 Patrick Trainer: our downloads here.
1060 01:12:42.290 ⇒ 01:12:46.540 Patrick Trainer: So we’re able to then grab. And so the idea
1061 01:12:46.770 ⇒ 01:12:49.919 Patrick Trainer: is like we would be able to.
1062 01:12:51.880 ⇒ 01:12:54.120 Patrick Trainer: like, say, we have a
1063 01:12:54.530 ⇒ 01:12:55.540 Patrick Trainer: Csv
1064 01:12:55.950 ⇒ 01:12:56.670 Patrick Trainer: file
1065 01:12:58.410 ⇒ 01:12:59.850 Patrick Trainer: like Csv.
1066 01:13:02.500 ⇒ 01:13:05.160 Patrick Trainer: I don’t know if I have any Csv’s.
1067 01:13:05.480 ⇒ 01:13:06.770 Patrick Trainer: but we can then
1068 01:13:10.400 ⇒ 01:13:11.549 Patrick Trainer: oh, I do.
1069 01:13:12.380 ⇒ 01:13:13.350 Patrick Trainer: and boom
1070 01:13:13.490 ⇒ 01:13:16.610 Patrick Trainer: downloaded that Csv. That was in my inbox.
1071 01:13:22.300 ⇒ 01:13:22.990 Patrick Trainer: Yep.
1072 01:13:25.480 ⇒ 01:13:26.469 Patrick Trainer: there we go.
1073 01:13:26.860 ⇒ 01:13:27.490 Bryce Codell: Damn
1074 01:13:27.950 ⇒ 01:13:29.280 Bryce Codell: very impressive.
1075 01:13:29.280 ⇒ 01:13:36.230 Patrick Trainer: And so so the next idea, because pool parts is sending us Csvs.
1076 01:13:36.590 ⇒ 01:13:44.699 Patrick Trainer: the problem is like, how do we get that? Csv, so this solves that problem of getting the Csv, so what’s next? The
1077 01:13:45.170 ⇒ 01:13:50.249 Patrick Trainer: because these are like, it’s a static file, and they send like
1078 01:13:51.160 ⇒ 01:13:55.060 Patrick Trainer: this entire file with just like new.
1079 01:13:55.340 ⇒ 01:13:57.409 Patrick Trainer: like rows added to it.
1080 01:13:57.650 ⇒ 01:14:00.449 Patrick Trainer: like they just had more values right.
1081 01:14:00.450 ⇒ 01:14:01.500 Bryce Codell: Just have like.
1082 01:14:01.980 ⇒ 01:14:04.409 Patrick Trainer: More more rows being added.
1083 01:14:04.550 ⇒ 01:14:05.870 Patrick Trainer: And so
1084 01:14:06.150 ⇒ 01:14:08.100 Patrick Trainer: the idea is.
1085 01:14:08.490 ⇒ 01:14:10.690 Patrick Trainer: once we have this.
1086 01:14:11.380 ⇒ 01:14:12.810 Patrick Trainer: we can
1087 01:14:13.280 ⇒ 01:14:18.570 Patrick Trainer: all like this package in Github actions, and then recommit
1088 01:14:19.401 ⇒ 01:14:25.349 Patrick Trainer: like once this loads, it’s in that Github runner context. We can then commit that to
1089 01:14:25.560 ⇒ 01:14:29.789 Patrick Trainer: the repo into that Dbd project in into the seed
1090 01:14:30.090 ⇒ 01:14:31.799 Patrick Trainer: Directory, and then
1091 01:14:32.230 ⇒ 01:14:36.640 Patrick Trainer: use dbt. To seed that into the into the database, or into stuff like.
1092 01:14:36.980 ⇒ 01:14:37.630 Bryce Codell: Dim
1093 01:14:38.130 ⇒ 01:14:39.760 Bryce Codell: quite the daisy chain, but.
1094 01:14:40.230 ⇒ 01:14:44.631 Patrick Trainer: It’s yeah, it’s it’s a it’s a daisy chain. It’s kind of hacky.
1095 01:14:45.320 ⇒ 01:14:48.519 Patrick Trainer: Maybe it’s not hacky. I don’t know but it it’s
1096 01:14:49.150 ⇒ 01:14:56.450 Patrick Trainer: it it, it works. The the other idea was so like the limitation that we’re
1097 01:14:56.918 ⇒ 01:15:01.820 Patrick Trainer: like working with is that it like it had to be emailed
1098 01:15:02.464 ⇒ 01:15:04.919 Patrick Trainer: and so it’s like we couldn’t have it.
1099 01:15:06.390 ⇒ 01:15:12.119 Patrick Trainer: It’s like it. It had to be emailed. And so it was either going to like.
1100 01:15:12.160 ⇒ 01:15:17.939 Patrick Trainer: involve some like app script stuff, or had to figure out another way around it.
1101 01:15:19.210 ⇒ 01:15:19.930 Patrick Trainer: But
1102 01:15:20.230 ⇒ 01:15:21.120 Patrick Trainer: anyway.
1103 01:15:21.890 ⇒ 01:15:22.510 Bryce Codell: Yep.
1104 01:15:24.400 ⇒ 01:15:26.219 Patrick Trainer: A a.
1105 01:15:30.770 ⇒ 01:15:33.169 Patrick Trainer: I know I was kind of going long there, but
1106 01:15:37.430 ⇒ 01:15:38.389 Patrick Trainer: That’s that.
1107 01:15:38.940 ⇒ 01:15:42.981 Bryce Codell: That’s sick. I don’t think that that’ll work for the weather data, but.
1108 01:15:43.270 ⇒ 01:15:44.210 Patrick Trainer: So either. Yeah.
1109 01:15:44.210 ⇒ 01:15:45.250 Bryce Codell: And just.
1110 01:15:45.410 ⇒ 01:15:45.930 Patrick Trainer: Because of.
1111 01:15:45.930 ⇒ 01:15:53.248 Bryce Codell: That’s good to know. Yeah, that there’s a resource like that available in the event that there’s other data that needs to get pulled in Via Csv.
1112 01:15:53.650 ⇒ 01:15:55.389 Bryce Codell: Reasonable path is.
1113 01:15:55.920 ⇒ 01:15:59.869 Bryce Codell: instead of just jacking up the 5 trend bill using using.
1114 01:15:59.870 ⇒ 01:16:00.240 Patrick Trainer: Right.
1115 01:16:00.240 ⇒ 01:16:00.820 Bryce Codell: It’s already.
1116 01:16:00.820 ⇒ 01:16:02.369 Patrick Trainer: Right? Yeah. Right?
1117 01:16:04.860 ⇒ 01:16:06.040 Bryce Codell: So very slick.
1118 01:16:07.030 ⇒ 01:16:08.540 Patrick Trainer: That is
1119 01:16:11.310 ⇒ 01:16:12.780 Patrick Trainer: yeah about it from me.
1120 01:16:13.900 ⇒ 01:16:17.759 Patrick Trainer: And any and all questions like, please direct them
1121 01:16:17.960 ⇒ 01:16:21.219 Patrick Trainer: my way, because this does need to.
1122 01:16:21.310 ⇒ 01:16:22.370 Patrick Trainer: I need to
1123 01:16:22.450 ⇒ 01:16:24.300 Patrick Trainer: tighten. Tighten this up a bit.
1124 01:16:26.385 ⇒ 01:16:28.869 Bryce Codell: No, this is awesome dude. I will
1125 01:16:29.140 ⇒ 01:16:30.790 Bryce Codell: definitely take you up on
1126 01:16:31.250 ⇒ 01:16:37.819 Bryce Codell: and follow up questions, and I’ll just ping you as I like come across things. I’ll let you know what I’m thinking in terms of like
1127 01:16:38.213 ⇒ 01:16:40.079 Bryce Codell: platform for the weather data.
1128 01:16:40.260 ⇒ 01:16:48.169 Patrick Trainer: Yeah, yeah, like, wh, what’s nice? And like at the point that we’re at to is like, really, everything’s like, it’s super green field.
1129 01:16:48.724 ⇒ 01:16:50.209 Patrick Trainer: In that, like
1130 01:16:52.010 ⇒ 01:16:54.409 Patrick Trainer: you have good, like good ideas or good ideas.
1131 01:16:54.410 ⇒ 01:16:55.639 Bryce Codell: Yeah, right.
1132 01:16:55.820 ⇒ 01:16:57.569 Patrick Trainer: And like, we’ve
1133 01:16:58.690 ⇒ 01:17:00.989 Patrick Trainer: like the like, what didn’t like so
1134 01:17:01.130 ⇒ 01:17:02.089 Patrick Trainer: kind of like
1135 01:17:02.350 ⇒ 01:17:04.600 Patrick Trainer: stressing on, or
1136 01:17:04.980 ⇒ 01:17:07.671 Patrick Trainer: what we’re trying to. Leverage is
1137 01:17:09.680 ⇒ 01:17:10.370 Patrick Trainer: like
1138 01:17:11.980 ⇒ 01:17:24.049 Patrick Trainer: biasing towards new tools, and kind of like throwing all the spaghetti at the wall and seeing what sticks. And it’s like we’ve tried a couple of different like anomaly detection
1139 01:17:24.641 ⇒ 01:17:32.949 Patrick Trainer: like packages like we’ve running elementary. We had anomalo before, and there’s something else you’re using. Dbt tests. They’re all kind of like
1140 01:17:33.070 ⇒ 01:17:34.269 Patrick Trainer: they’re all kind of shit.
1141 01:17:35.926 ⇒ 01:17:36.679 Patrick Trainer: But
1142 01:17:38.389 ⇒ 01:17:39.150 Patrick Trainer: that.
1143 01:17:39.220 ⇒ 01:17:41.579 Patrick Trainer: And then we’ve like used light dash.
1144 01:17:42.660 ⇒ 01:17:50.280 Patrick Trainer: Then we use like evidence still using evidence. But then we’ve gotten on to reals. I really like real. Real’s awesome.
1145 01:17:52.210 ⇒ 01:17:56.860 Patrick Trainer: but it’s like just keeping that in mind that like.
1146 01:17:57.930 ⇒ 01:18:01.420 Patrick Trainer: good ideas are good ideas. Yeah. And so like
1147 01:18:01.910 ⇒ 01:18:05.900 Patrick Trainer: if there’s a better way or a novel way to do something like
1148 01:18:06.760 ⇒ 01:18:07.810 Patrick Trainer: let’s give it a shot.
1149 01:18:08.140 ⇒ 01:18:18.592 Bryce Codell: Yeah, sweet sounds good on that note. I will open up a Pr at some point with the like. The schema database name overrides and.
1150 01:18:18.980 ⇒ 01:18:19.490 Patrick Trainer: Yeah.
1151 01:18:19.490 ⇒ 01:18:24.079 Bryce Codell: Yeah, cool. I have no idea where to put the documentation, but I’ll
1152 01:18:24.140 ⇒ 01:18:25.519 Bryce Codell: open up Pr in the weekend.
1153 01:18:26.210 ⇒ 01:18:28.279 Patrick Trainer: Yeah, like, yeah.
1154 01:18:29.470 ⇒ 01:18:31.080 Patrick Trainer: so the
1155 01:18:32.370 ⇒ 01:18:41.989 Patrick Trainer: it’s it the again poorly name. I do have a ticket to rename the repo. But the brain forge actions. Repo has kind of what’s been our like it.
1156 01:18:42.340 ⇒ 01:18:54.700 Patrick Trainer: It will be renamed to like brain forge internal. And that’s where a lot of the documentation except for the documentation that I was looking for today. Lives and so like
1157 01:18:55.060 ⇒ 01:18:58.096 Patrick Trainer: that’s a good spot to put it.
1158 01:18:59.360 ⇒ 01:19:02.790 Patrick Trainer: anything, I think, like client specific.
1159 01:19:05.840 ⇒ 01:19:08.810 Patrick Trainer: make sense to go into like its own
1160 01:19:09.160 ⇒ 01:19:16.330 Patrick Trainer: client, specific repo. But the the the brain for to actions repo is a good spot to
1161 01:19:16.730 ⇒ 01:19:17.470 Patrick Trainer: start.
1162 01:19:17.740 ⇒ 01:19:19.409 Bryce Codell: Sweet sounds good to me
1163 01:19:19.680 ⇒ 01:19:20.370 Bryce Codell: yet.
1164 01:19:21.950 ⇒ 01:19:22.970 Bryce Codell: Alrighty
1165 01:19:23.000 ⇒ 01:19:29.044 Bryce Codell: good stuff. I have the information I need to keep forging ahead to keep brain forging ahead.
1166 01:19:30.575 ⇒ 01:19:36.090 Bryce Codell: just staring me in the face. Alright! Good talk, everybody, Nico. I’ll see you in a couple of hours, Patrick.
1167 01:19:36.090 ⇒ 01:19:36.799 Nicolas Sucari: Yeah, take.
1168 01:19:36.800 ⇒ 01:19:38.424 Bryce Codell: You’re not on the interwebs.
1169 01:19:38.750 ⇒ 01:19:39.709 Patrick Trainer: Take care! Aye.
1170 01:19:39.710 ⇒ 01:19:40.130 Bryce Codell: Hi! Everyone.
1171 01:19:40.130 ⇒ 01:19:41.339 Nicolas Sucari: Thank you. Bye, bye.