Meeting Title: Brainforge x Stackblitz Weekly Check Date: 2025-01-31 Meeting participants: Luke Daque, Mitchell Wright, Uttam Kumaran
WEBVTT
1 00:00:16.239 ⇒ 00:00:16.930 Uttam Kumaran: Let’s see.
2 00:00:21.290 ⇒ 00:00:24.419 Uttam Kumaran: Oof, okay, some water.
3 00:01:39.570 ⇒ 00:01:41.650 Luke Daque: I don’t see the models on stage.
4 00:01:42.400 ⇒ 00:01:44.050 Luke Daque: Looks like it created it.
5 00:01:44.670 ⇒ 00:01:50.710 Luke Daque: The actions the workflow created, the models in Dbt instead of stage marts.
6 00:01:53.140 ⇒ 00:01:56.900 Luke Daque: So I guess it’s still created in analytics.
7 00:01:57.850 ⇒ 00:01:59.420 Uttam Kumaran: Oh, yeah, it did. Why.
8 00:01:59.420 ⇒ 00:02:00.340 Luke Daque: I don’t know.
9 00:02:02.227 ⇒ 00:02:10.519 Luke Daque: Yeah. Maybe it did run the actions on from the main.
10 00:02:22.490 ⇒ 00:02:24.429 Uttam Kumaran: Let me check the actions. Then.
11 00:02:27.680 ⇒ 00:02:33.840 Luke Daque: So it created the tables under.
12 00:02:34.800 ⇒ 00:02:38.260 Luke Daque: yeah, it’s it’s doing target Cicd, but it’s in.
13 00:02:47.920 ⇒ 00:02:49.200 Mitchell Wright: What’s up?
14 00:02:49.980 ⇒ 00:02:50.780 Uttam Kumaran: Hey!
15 00:02:51.300 ⇒ 00:02:52.289 Mitchell Wright: How we doing.
16 00:02:53.350 ⇒ 00:02:54.499 Uttam Kumaran: Good Dean, how are you?
17 00:02:56.260 ⇒ 00:02:57.110 Mitchell Wright: Pretty good.
18 00:02:58.420 ⇒ 00:02:59.580 Uttam Kumaran: What’s the latest.
19 00:03:05.980 ⇒ 00:03:07.160 Mitchell Wright: I don’t know just
20 00:03:08.615 ⇒ 00:03:09.480 Uttam Kumaran: Alright!
21 00:03:10.330 ⇒ 00:03:11.509 Mitchell Wright: I mean, if there’s like.
22 00:03:11.510 ⇒ 00:03:12.039 Uttam Kumaran: You’re like.
23 00:03:12.040 ⇒ 00:03:16.930 Mitchell Wright: 1 million things. Dude hiring is the worst. Like.
24 00:03:18.019 ⇒ 00:03:25.800 Mitchell Wright: yeah, we need. Yeah, there’s just just a lot. But overall things are good, you know. Despite everything.
25 00:03:26.330 ⇒ 00:03:29.589 Mitchell Wright: Mrr is still increasing. So that’s the the best.
26 00:03:29.590 ⇒ 00:03:34.200 Uttam Kumaran: Yeah, if it’s going off, then you’re like, what data? Who needs data.
27 00:03:34.200 ⇒ 00:03:35.600 Mitchell Wright: Yeah, yeah, who needs it?
28 00:03:36.310 ⇒ 00:03:40.320 Mitchell Wright: We’ll just keep shooting blind. And it’s going. Okay. I guess so.
29 00:03:40.320 ⇒ 00:03:42.339 Uttam Kumaran: Yeah, awesome.
30 00:03:42.820 ⇒ 00:03:57.039 Uttam Kumaran: Well, I think we made some good progress. I kinda wanted to take today and a couple of things I want to walk you through our Dbt structure. I also want to walk you through Snowflake and make sure
31 00:03:57.240 ⇒ 00:04:00.439 Uttam Kumaran: you’re like, okay, with sort of how we have things set up.
32 00:04:01.052 ⇒ 00:04:04.217 Uttam Kumaran: I also do want to talk about
33 00:04:05.530 ⇒ 00:04:20.001 Uttam Kumaran: we do have a stripe. Pr, that’s that’s we just push through that has a couple of models. But it also tests sort of our Github actions. And then I want to talk about hubspot and segment. So a couple of things, maybe we can just try to get to those
34 00:04:21.570 ⇒ 00:04:23.839 Uttam Kumaran: anything else we want to try to get to.
35 00:04:25.098 ⇒ 00:04:35.119 Mitchell Wright: So I I think that I connected Snowflake and Hubspot. I’m just looking
36 00:04:37.110 ⇒ 00:04:44.320 Mitchell Wright: yeah, I mean, it says it’s connected but that it could take a little time for the data to come through, so I don’t know.
37 00:04:44.320 ⇒ 00:04:45.060 Uttam Kumaran: Okay.
38 00:04:45.060 ⇒ 00:04:49.380 Mitchell Wright: There’s anything we need to do there. But that is
39 00:04:50.710 ⇒ 00:04:55.250 Mitchell Wright: yeah. It should be coming through now. The only thing
40 00:04:55.410 ⇒ 00:05:00.829 Mitchell Wright: I’m not sure in snowflake. If there’s something we needed to do.
41 00:05:03.420 ⇒ 00:05:11.090 Mitchell Wright: Yeah, if there was like something we need to do there. But yeah, I went through that that process. So maybe that’s something we could look at and just double check.
42 00:05:12.230 ⇒ 00:05:13.789 Mitchell Wright: Okay, as well for a sec.
43 00:05:14.880 ⇒ 00:05:16.560 Uttam Kumaran: But yeah, okay, cool.
44 00:05:18.160 ⇒ 00:05:20.449 Uttam Kumaran: So let me this up.
45 00:05:31.820 ⇒ 00:05:39.169 Uttam Kumaran: Great so I guess I’ll kind of walk you through how we are thinking about our snowflake setup.
46 00:05:39.613 ⇒ 00:05:43.949 Uttam Kumaran: It’s kind of evolved as we’ve done a bunch of these. But
47 00:05:44.210 ⇒ 00:05:54.370 Uttam Kumaran: it’s sort of in the theme of trying to keep things pretty clear, but also saving you down the line from probably what some other smart data engineer would come and be like.
48 00:05:54.520 ⇒ 00:05:58.769 Uttam Kumaran: We need to do a bunch of change, a bunch of grants, and it’ll cause a huge mess.
49 00:05:59.266 ⇒ 00:06:24.529 Uttam Kumaran: Basically. We are. Now, we basically have the concept of dev staging and prod, that you can also think about it as like just for developers, locally, staging is sort of for a Cicd, and then production is basically all the bi tools, and any analysts can so can sort of use that the nice thing is local development. You can run queries or also run full
50 00:06:24.840 ⇒ 00:06:30.609 Uttam Kumaran: full pipelines. Like, if you’re doing local development, you have a place where that can hit and actually
51 00:06:30.900 ⇒ 00:06:35.150 Uttam Kumaran: like run the entire thing without impacting production.
52 00:06:36.540 ⇒ 00:06:39.219 Uttam Kumaran: Second, you can also basically prevent
53 00:06:39.420 ⇒ 00:06:53.565 Uttam Kumaran: developers from ever being able to even write to production. That way. There’s not like a Oh, like Oh, should I actually dropped like one of the production tables. I totally forgot to change like a prefix or something.
54 00:06:54.080 ⇒ 00:07:15.719 Uttam Kumaran: so that’s sort of the structure. And then dbt, when you push a pr, and I’ll show you that process. The Github action actually runs the code in the branch against staging, so that when when someone goes and says, Okay, I want to review this Pr typically in data world, you’re just like looking at the sequel. You’re like, cool. This is good. But
55 00:07:15.870 ⇒ 00:07:20.260 Uttam Kumaran: it’s not often that you actually get to see like, does this run, which is like
56 00:07:20.410 ⇒ 00:07:35.790 Uttam Kumaran: that’s like a huge component of this whole thing. So one is like the Dbt staging, it’ll get run in staging intermediate and staging marts. And additionally, you can then go and actually see and query those tables as if, like you just had another environment.
57 00:07:36.198 ⇒ 00:07:48.590 Uttam Kumaran: Kind of like. The only difference between this and sort of what you probably will see on like the like on a on a software team is typically they have like a staging branch, and then they do like a cut over
58 00:07:48.750 ⇒ 00:07:50.780 Uttam Kumaran: for us like that would require 4.
59 00:07:50.780 ⇒ 00:07:53.280 Mitchell Wright: What? What is it? Blue, green, or whatever.
60 00:07:53.280 ⇒ 00:08:08.319 Uttam Kumaran: Yeah, they have like that, or they’ll have like, pre, they’ll have like, pre prod prod. That’s basically like, if you were, I don’t know. I don’t think in data, you need to go that Od, but basically it’s like you would just have a bunch of changes and you can cut them over. I think this is like
61 00:08:08.460 ⇒ 00:08:09.500 Uttam Kumaran: good enough.
62 00:08:09.931 ⇒ 00:08:20.048 Uttam Kumaran: The the kind of the way we did this, and I can. I’m happy to give you all of these grants so that you have, because in the case you want to go change something
63 00:08:20.940 ⇒ 00:08:46.830 Uttam Kumaran: this has, like everything we’ve run so you can, you could see what we’ve done. And then also, in case you need to change things, you could do that. Basically. What what we’ve done is, we’ve created reader and writer roles for production, staging, and development. What that allows you to do is basically, then assign those roles to higher roles, right? And and what are the higher roles that I’m talking about? We have role transform role developer and role ingest
64 00:08:46.830 ⇒ 00:08:57.060 Uttam Kumaran: polytomic uses role ingest Dbt runners and developers Dbt runners like in in in
65 00:08:57.280 ⇒ 00:09:07.429 Uttam Kumaran: Dvt runners. And yeah, they use role transform. And the role developer basically can read from prod and then also writes to development.
66 00:09:07.866 ⇒ 00:09:32.213 Uttam Kumaran: And so what what basically we do is you create reader and writer roles for each environment, and then you can assign those as needed. What this allows you to do is you can say, anyone who’s a developer. You don’t have the you don’t by inheriting the just the roles for the development environment. And just the reader role for production. There’s no way they can basically go write anything to production or change that
67 00:09:32.670 ⇒ 00:09:35.170 Uttam Kumaran: it’s a little bit again. It’s like a little bit
68 00:09:35.350 ⇒ 00:09:41.700 Uttam Kumaran: more in depth than if you were to just say, like Grant, select or Grant all to
69 00:09:42.050 ⇒ 00:09:43.979 Uttam Kumaran: the user roles
70 00:09:44.340 ⇒ 00:09:57.160 Uttam Kumaran: and doing everything through roles is great. Because when you bring on people, you can just slot them to the right place and they inherit everything. Basically, it takes the notion of having to grant to individual users, which is where a lot of like issues stem from down the line.
71 00:09:57.730 ⇒ 00:10:02.700 Mitchell Wright: Yeah, I. So so at Vercel we used whatever the gitlab.
72 00:10:03.920 ⇒ 00:10:07.639 Uttam Kumaran: Yes, this is very similar to that. Yeah. Permafrost, this is very. It’s like.
73 00:10:07.640 ⇒ 00:10:08.580 Mitchell Wright: Cost, yeah, yeah, exactly.
74 00:10:08.580 ⇒ 00:10:19.190 Uttam Kumaran: It’s basically like the same. It’s sort of it’s basically like a similar structure to what they say. I think they have a couple they have, maybe, like one more layer but
75 00:10:19.740 ⇒ 00:10:21.349 Uttam Kumaran: very, very similar.
76 00:10:22.190 ⇒ 00:10:26.709 Mitchell Wright: Cool. Yeah, yeah, yeah, that makes sense. Yeah. My big beef with permafrost is that it’s just
77 00:10:26.820 ⇒ 00:10:33.699 Mitchell Wright: incredibly slow to to run whenever we had to update permissions. It’s like a 30 min.
78 00:10:33.700 ⇒ 00:10:36.340 Uttam Kumaran: Hold on one second I think I may have lost you.
79 00:10:38.630 ⇒ 00:10:39.470 Uttam Kumaran: Wait!
80 00:10:39.570 ⇒ 00:10:41.859 Uttam Kumaran: Can you hear, Mitch? Maybe it’s me.
81 00:10:41.860 ⇒ 00:10:43.660 Luke Daque: Yeah, yeah, I can. I can hear, Mitch.
82 00:10:43.660 ⇒ 00:10:48.329 Mitchell Wright: Yeah, I think like your, it sounded like your speaker or your microphone changed.
83 00:10:48.730 ⇒ 00:10:52.159 Uttam Kumaran: Okay, how about you? Okay, I can hear you now. Sorry. Maybe my headphone system.
84 00:10:52.610 ⇒ 00:10:53.880 Uttam Kumaran: Sorry. Can you say that one more time.
85 00:10:53.880 ⇒ 00:10:57.799 Mitchell Wright: But yeah, permafrost, just so slow. It was really frustrating every time we had
86 00:10:58.580 ⇒ 00:11:03.870 Mitchell Wright: it would take like 2030 min for that to actually run. So yeah.
87 00:11:03.870 ⇒ 00:11:09.069 Uttam Kumaran: Yeah, I’m I’m hoping I can give you a nice visual to pair with this, where you can see
88 00:11:09.270 ⇒ 00:11:19.339 Uttam Kumaran: like, it’s basically like, kind of composable roles by environment. And, for example, like you then have, like ingest roles for any of your Etl tools. You have
89 00:11:19.762 ⇒ 00:11:22.259 Uttam Kumaran: report roles for any of your bi tools.
90 00:11:22.410 ⇒ 00:11:26.222 Uttam Kumaran: And then you have a developer role for any of your developers.
91 00:11:26.540 ⇒ 00:11:40.900 Mitchell Wright: So the one thing that we did that I actually kind of liked was having specific roles for individual tools that we that we’re interacting with snowflake, just so that we could track costs of
92 00:11:41.360 ⇒ 00:11:47.710 Mitchell Wright: individual tools and how they’re working with snowflake is that thing or.
93 00:11:47.710 ⇒ 00:11:48.220 Uttam Kumaran: Yeah.
94 00:11:48.380 ⇒ 00:11:50.130 Mitchell Wright: Manage that a little differently.
95 00:11:50.350 ⇒ 00:12:01.100 Uttam Kumaran: Yeah, so to talk about users, we created 3 users, there’s a user for ingest user for transform user for report. We can expand this to basically create
96 00:12:01.320 ⇒ 00:12:12.090 Uttam Kumaran: service users for each tool. At the moment. Basically, all Etl goes, all Etl will use the user ingest role. All
97 00:12:12.540 ⇒ 00:12:19.209 Uttam Kumaran: transformation will use this. All report will use this. We can change this to create users for each tool. If you’d like.
98 00:12:19.560 ⇒ 00:12:22.309 Mitchell Wright: Yeah, for now that’s probably fine, because polytomic is.
99 00:12:22.700 ⇒ 00:12:26.950 Uttam Kumaran: It’s just polyatomic. It just just really.
100 00:12:26.950 ⇒ 00:12:28.680 Mitchell Wright: Reporting, yeah, yeah. And.
101 00:12:28.680 ⇒ 00:12:29.500 Uttam Kumaran: Yeah, like.
102 00:12:29.500 ⇒ 00:12:30.170 Mitchell Wright: No problem.
103 00:12:30.630 ⇒ 00:12:41.980 Uttam Kumaran: And I. So the way Hubspot works is Hubspot will share us a shared database. It won’t write to this, so I don’t think.
104 00:12:41.980 ⇒ 00:12:44.039 Mitchell Wright: What we do there will just be transform.
105 00:12:44.570 ⇒ 00:12:45.230 Uttam Kumaran: Yeah.
106 00:12:46.010 ⇒ 00:12:46.700 Mitchell Wright: Okay.
107 00:12:46.700 ⇒ 00:12:50.280 Uttam Kumaran: Like I. I don’t think I don’t think it’ll actually, it’s basically we’ll just create a
108 00:12:51.810 ⇒ 00:12:59.650 Uttam Kumaran: yeah, like a shared dB with us segment. We’ll definitely need the ingest role. So we can create.
109 00:12:59.890 ⇒ 00:13:02.140 Uttam Kumaran: I can create a service user for segment.
110 00:13:02.960 ⇒ 00:13:10.720 Mitchell Wright: Yeah, that’d be great, just so that we can track costs and and see, you know what which tools are actually costing us. What
111 00:13:10.950 ⇒ 00:13:14.010 Mitchell Wright: I just. I like the little bit more granularity. There.
112 00:13:14.870 ⇒ 00:13:18.100 Uttam Kumaran: Okay, cool.
113 00:13:18.860 ⇒ 00:13:22.590 Uttam Kumaran: And the other thing is like we’re using key pair off
114 00:13:22.860 ⇒ 00:13:27.539 Uttam Kumaran: for the most part. This is a process that’s like a little bit.
115 00:13:27.640 ⇒ 00:13:33.269 Uttam Kumaran: It’s not that complicated. But basically, when you add new users, we typically recommend people use.
116 00:13:33.440 ⇒ 00:13:47.619 Uttam Kumaran: keep air off. We. We have keep arrow set up for the service users. We don’t have it set up for, like our users, like, in terms of like our user accounts. But it’s that’s like the most secure. Basically, you create a public and a private key.
117 00:13:47.770 ⇒ 00:13:49.409 Uttam Kumaran: And then that’s how you
118 00:13:50.620 ⇒ 00:13:53.650 Uttam Kumaran: log in and interact with snowflake.
119 00:13:55.930 ⇒ 00:13:58.679 Uttam Kumaran: I don’t know. Just try to get the security right early. So.
120 00:13:59.990 ⇒ 00:14:03.939 Uttam Kumaran: but I can walk you through that process. It’s it’s not like super complicated.
121 00:14:03.940 ⇒ 00:14:07.230 Mitchell Wright: Okay, yeah, I’m not. Yeah. That sounds great. Not worried about that.
122 00:14:07.560 ⇒ 00:14:14.010 Uttam Kumaran: Okay? Any other questions on Snowflake. Like, I think we’re
123 00:14:14.960 ⇒ 00:14:23.789 Uttam Kumaran: basically in a good place on everything. Snowflake. I mean, I think once we get down the line I can help show you show like, how to get
124 00:14:24.130 ⇒ 00:14:36.499 Uttam Kumaran: the best like cost and and usage reports. I don’t. I think we still have some time left in our in our trial. I need to actually double check what the date is, but we have 30 days, so I think we probably have another 15 days.
125 00:14:38.540 ⇒ 00:14:41.869 Uttam Kumaran: But this is this will sort of give you enough
126 00:14:42.410 ⇒ 00:14:44.681 Uttam Kumaran: to like. Run for a while.
127 00:14:45.060 ⇒ 00:14:53.610 Mitchell Wright: Yeah, no, that’s great. And then, yeah, I can go in and set up billing for that.
128 00:14:54.090 ⇒ 00:15:01.928 Mitchell Wright: So yeah, I think that that looks great. We’ve also got a data candidate who I think
129 00:15:02.810 ⇒ 00:15:10.460 Mitchell Wright: is is working through the interview loop now, and I’d really like to hire him and so I think
130 00:15:10.780 ⇒ 00:15:13.850 Mitchell Wright: I’d love to have him like work with you guys and
131 00:15:14.130 ⇒ 00:15:14.770 Uttam Kumaran: Cool.
132 00:15:15.170 ⇒ 00:15:17.770 Mitchell Wright: Yeah, like. So I mean, that’s still probably like
133 00:15:18.530 ⇒ 00:15:25.349 Mitchell Wright: at the earliest, would be like 3 to 4 weeks out. So probably not even till beginning of March. But
134 00:15:26.080 ⇒ 00:15:32.239 Mitchell Wright: yeah, just to kind of give you guys an idea. And then and then, yeah, I think I think that there’s probably
135 00:15:32.790 ⇒ 00:15:42.564 Mitchell Wright: enough work that it would make sense to still keep you guys on. But then he would sort of become the main point of contact and work with you guys to
136 00:15:43.470 ⇒ 00:15:54.351 Mitchell Wright: to kind of manage the modeling and stuff like that? Because I think he’s he’s more data engineering and would do more on pipelines and platform management.
137 00:15:55.210 ⇒ 00:15:56.310 Mitchell Wright: At this point.
138 00:15:56.690 ⇒ 00:16:03.259 Uttam Kumaran: Nice. Okay, perfect. Yeah. I mean, I think you’ll I think you’ll be happy with the setup. I wish I had this setup. I joined.
139 00:16:03.710 ⇒ 00:16:07.659 Mitchell Wright: Yeah, that’s how I sort of think about these things is like.
140 00:16:07.940 ⇒ 00:16:16.179 Uttam Kumaran: What would I have wanted as a developer? And we’re getting better better every client we go to. So the other thing I’ll show you that, I think is really nice.
141 00:16:17.150 ⇒ 00:16:41.809 Uttam Kumaran: you know. I’m sure you know, like, one of the the really tough parts about data is like, there’s not really great testing. Cicd isn’t like really baked into people’s brains. But this will save anybody who’s working on this anybody who’s downstream of this like a ton of headache. Basically. We use dbt, core via actions.
142 00:16:42.327 ⇒ 00:16:48.869 Uttam Kumaran: And where you can go if you go into the dock, Github, and go into workflows. You could actually see
143 00:16:49.150 ⇒ 00:16:54.889 Uttam Kumaran: the 2 actions that we run. Not sure if you’re familiar with github actions. If you’re not.
144 00:16:55.140 ⇒ 00:16:55.710 Uttam Kumaran: I wouldn’t.
145 00:16:55.710 ⇒ 00:16:58.230 Mitchell Wright: A little bit, but I haven’t done a ton with them yet.
146 00:16:58.230 ⇒ 00:17:00.502 Uttam Kumaran: Okay, it’s like, horrible.
147 00:17:01.320 ⇒ 00:17:04.540 Mitchell Wright: Yes, that’s what I’ve heard, and debugging.
148 00:17:04.540 ⇒ 00:17:08.260 Uttam Kumaran: It’s like this took us a long time to get
149 00:17:08.440 ⇒ 00:17:15.163 Uttam Kumaran: and it like, now, we’re basically like, don’t touch it. But we change some variables. But basically we
150 00:17:16.466 ⇒ 00:17:36.040 Uttam Kumaran: the way this basically works is like the branch gets checked out, we install python and all the dependencies. And then we run dbt within the the Github machine. The nice thing is for Github. You get a bunch of free minutes like 30,000 free minutes or something. So this doesn’t come up at much cost unless
151 00:17:36.300 ⇒ 00:18:01.190 Uttam Kumaran: like this is going crazy. But basically we run dbt, source fresh this dbt test, and then we run dbt, so we install dependencies and then run. The nice thing is like, if we want to layer on other Dbt calls or run certain projects like, we want to run certain models or certain tags. It’s very flexible, and that’s also free.
152 00:18:01.736 ⇒ 00:18:04.850 Uttam Kumaran: So we run this. This runs on a Cron.
153 00:18:05.440 ⇒ 00:18:06.860 Uttam Kumaran: Right now. We just have it
154 00:18:07.380 ⇒ 00:18:14.469 Uttam Kumaran: run a couple of times a day. We can change this or not. It’s right now. It’s whatever
155 00:18:14.610 ⇒ 00:18:17.739 Uttam Kumaran: as we get bigger. We can probably think about if we want to have
156 00:18:18.080 ⇒ 00:18:23.170 Uttam Kumaran: weekly versus daily versus intraday, typically like what we suggest. And then
157 00:18:23.330 ⇒ 00:18:26.679 Uttam Kumaran: you sort of just have 3 buckets that you can toss every model into.
158 00:18:28.270 ⇒ 00:18:36.060 Uttam Kumaran: and then we also have a run Dvt. Staging. This is the same thing, except it runs on pull requests to the marine branch.
159 00:18:36.762 ⇒ 00:18:42.197 Uttam Kumaran: So if, for example, if you are to go to one of our pull requests
160 00:18:42.700 ⇒ 00:19:08.509 Uttam Kumaran: this is literally the one I created in order to to do this. You’ll see that. There’s a little checkmark here, and so what happens is when you push a Pr, as part of the Cicd step in order to get the Pr merge. It’ll run through running Dvt within the staging environment. So it’ll create intermediate models, and it’ll create the March models. And then basically, like everything, RAM.
161 00:19:09.400 ⇒ 00:19:11.150 Mitchell Wright: In the staging, so like staging.
162 00:19:11.150 ⇒ 00:19:35.200 Uttam Kumaran: In staging. So that so the the what that resolves is that you’re not running. You’re not running it on your production like, oh, you’re not overriding your production. So you’re just running it, basically like you’re running in another environment, you’re but you’re running the entire pipeline. And then basically, once this gets merged it automatically goes into main and then gets picked up by the primary job.
163 00:19:37.280 ⇒ 00:19:41.649 Uttam Kumaran: And so we can run as many of these. But these are pretty good.
164 00:19:41.950 ⇒ 00:19:50.979 Uttam Kumaran: The other point about pull requests and stuff. And we’re working on some of these, you know, across like brain forged clients, is like adding things like linting on. Commit
165 00:19:51.817 ⇒ 00:19:56.350 Uttam Kumaran: and like more style more style, based linting and things like that.
166 00:19:57.034 ⇒ 00:19:58.339 Uttam Kumaran: Like sequel, Fluff.
167 00:19:58.500 ⇒ 00:20:05.859 Uttam Kumaran: and like pilots and stuff like that, or or I think it’s called black. So we’ll we’ll. We’re like basically getting our like.
168 00:20:06.330 ⇒ 00:20:21.560 Uttam Kumaran: We’re now that we sort of have all we’re that’s like the last templated thing we’re sort of working on. So we’ll just as soon as that’s ready we’ll apply it. And then, in case you have any particular like style guides, or whatever we can implement. But we have a pretty good idea about how to
169 00:20:22.010 ⇒ 00:20:25.039 Uttam Kumaran: how to do those, and that again, the nice thing about having that
170 00:20:25.330 ⇒ 00:20:41.419 Uttam Kumaran: as part of your build process and as part of a as like an on commit hook is that people don’t. You don’t have to worry about bad code entering. And then also, as people just make changes, the commit hooks will just kind of like format their stuff really nicely.
171 00:20:42.530 ⇒ 00:20:45.210 Uttam Kumaran: And so it just keeps the repo pretty clean.
172 00:20:45.960 ⇒ 00:20:46.820 Mitchell Wright: Perfect.
173 00:20:48.308 ⇒ 00:20:55.709 Uttam Kumaran: Yeah. And I think the only thing you know, the big thing that we were working on this week was, yeah, just getting the
174 00:20:56.465 ⇒ 00:21:01.630 Uttam Kumaran: stuff from polytomic in which is.
175 00:21:02.180 ⇒ 00:21:07.330 Uttam Kumaran: I think it’s somewhere good shot.
176 00:21:09.040 ⇒ 00:21:14.090 Uttam Kumaran: I’ve been mainly looking at stripe stuff. So let me confirm I don’t know. Did that land yet?
177 00:21:29.820 ⇒ 00:21:32.930 Uttam Kumaran: Okay, we disabled it.
178 00:21:33.930 ⇒ 00:21:41.169 Uttam Kumaran: Should I enable it? I don’t know. What do we? I don’t wanna touch anything now.
179 00:21:41.170 ⇒ 00:21:42.969 Mitchell Wright: Wait. What did we disable? Sorry.
180 00:21:42.970 ⇒ 00:21:45.649 Uttam Kumaran: The stack Blitz, the production to snowflakes.
181 00:21:45.650 ⇒ 00:21:54.749 Mitchell Wright: So yeah, don’t touch that yet. I need to just talk to Pete and figure out what next steps are there for them? Because
182 00:21:54.900 ⇒ 00:22:02.240 Mitchell Wright: whatever they did it, it broke something for us where users couldn’t delete their accounts.
183 00:22:04.290 ⇒ 00:22:09.280 Mitchell Wright: Which, in the grand scheme of things that could break. That’s like very low on my list of.
184 00:22:09.280 ⇒ 00:22:11.029 Uttam Kumaran: But it could have been something else. Okay, I won’t.
185 00:22:11.030 ⇒ 00:22:31.597 Mitchell Wright: Yeah, yeah, like, people could create a case or you know. So let me follow up with Pete and Pi, and just see what they are thinking as far as next steps to get that in. The other piece is so so we have 2 databases. There’s the postgres, and then there, there’s the I’m sorry. There’s the stack, Blitz, and then there’s the bolt
186 00:22:32.110 ⇒ 00:22:36.409 Mitchell Wright: the bolt. There were some issues on settings as far as
187 00:22:38.420 ⇒ 00:22:54.419 Mitchell Wright: certain settings that have to be set up to make the Cdc work, and in order to set them we have to restart the server. We already have a scheduled downtime for Sunday to do an upgrade. So he’s gonna just turn it on then, so that should be
188 00:22:55.260 ⇒ 00:23:12.699 Mitchell Wright: possible for Monday. But I don’t know if they’re gonna do it on Monday, just due to, we need to figure out, yeah, what we can do to make it work. And then I don’t know if they’re gonna have to just do a migration and add primary keys to those tables that didn’t or what they’re gonna do. But
189 00:23:13.090 ⇒ 00:23:14.020 Mitchell Wright: we’ll see.
190 00:23:15.130 ⇒ 00:23:23.880 Uttam Kumaran: Okay. Well, I mean, we’ll just see. I mean, there’s there’s alternatives if they want to give us even just like the data, manually or something, and we can start modeling. But.
191 00:23:24.210 ⇒ 00:23:24.530 Mitchell Wright: Okay.
192 00:23:24.530 ⇒ 00:23:29.939 Uttam Kumaran: Again for us. We really just need a shape to just start getting some models out.
193 00:23:31.070 ⇒ 00:23:36.090 Uttam Kumaran: But again, maybe if it gets past like Wednesday or Thursday. I can be like, Okay, what? Maybe.
194 00:23:36.090 ⇒ 00:23:38.599 Mitchell Wright: Yeah, then we can, we can do something different.
195 00:23:38.910 ⇒ 00:23:39.530 Uttam Kumaran: Yeah.
196 00:23:39.720 ⇒ 00:23:46.340 Mitchell Wright: And then, yeah, we yes, what he has done in the past is he’s just cut like a
197 00:23:47.210 ⇒ 00:23:55.529 Mitchell Wright: a replica of the database that doesn’t update. And we could just do that and just pull that in, not via Cdc.
198 00:23:56.230 ⇒ 00:24:02.390 Mitchell Wright: okay, maybe maybe I’ll tell him we just do that to move forward while they figure out what they need to do to get the Cdc.
199 00:24:02.640 ⇒ 00:24:07.080 Uttam Kumaran: Yeah, it’s gonna take us a, it’s gonna take us a little bit to finish all the modeling.
200 00:24:08.350 ⇒ 00:24:09.470 Uttam Kumaran: And so like.
201 00:24:09.860 ⇒ 00:24:21.019 Uttam Kumaran: yeah, I really just need like the data. So we can start building the models for. And we can. We’ll be able to cut over super easily. We’ll just change the references on our side.
202 00:24:22.360 ⇒ 00:24:23.700 Uttam Kumaran: if if that’s like.
203 00:24:23.880 ⇒ 00:24:29.730 Uttam Kumaran: yeah, no, I don’t wanna like I don’t wanna have. There’s a lot of dependencies to get that stuff right? So
204 00:24:30.580 ⇒ 00:24:31.569 Uttam Kumaran: we can, we can.
205 00:24:31.570 ⇒ 00:24:32.299 Mitchell Wright: Let me, ask him.
206 00:24:32.300 ⇒ 00:24:32.720 Uttam Kumaran: Yeah, I can.
207 00:24:32.720 ⇒ 00:24:34.179 Mitchell Wright: Do there as far as
208 00:24:34.380 ⇒ 00:24:40.899 Mitchell Wright: doing that and getting it connected. And just so that we have something to model with. And then it takes a little bit of
209 00:24:42.040 ⇒ 00:24:45.040 Mitchell Wright: pressure off them on timeframe.
210 00:24:45.420 ⇒ 00:24:55.650 Uttam Kumaran: Yeah, the other thing is like, if they if they want to give us that as a Csv or like a parquet, and like, we can load that in manually, or if they want to drop it into an S. 3 bucket, we could load that in.
211 00:24:56.150 ⇒ 00:24:56.850 Mitchell Wright: Okay.
212 00:24:57.070 ⇒ 00:24:58.310 Uttam Kumaran: We can even go as like.
213 00:24:58.380 ⇒ 00:24:58.960 Mitchell Wright: To connect.
214 00:24:59.080 ⇒ 00:25:00.830 Uttam Kumaran: As bare bones as that. Yeah.
215 00:25:01.160 ⇒ 00:25:06.799 Mitchell Wright: Yeah. Okay, cool. I’ll make a note.
216 00:25:11.270 ⇒ 00:25:16.129 Uttam Kumaran: Yeah, cause even like creating a replica. And like all that, it may just be overkill, like.
217 00:25:16.320 ⇒ 00:25:25.007 Uttam Kumaran: we just need like 5 tables as a Csv like, very, very simply. And I just need to know that the shape and the data types are like, gonna stay the same
218 00:25:25.870 ⇒ 00:25:30.730 Uttam Kumaran: if they can give it to us in like a parquet or or yeah, that’s even better.
219 00:25:31.270 ⇒ 00:25:34.739 Mitchell Wright: Cool. Okay, I’ll I’ll ask them about that then
220 00:25:36.240 ⇒ 00:25:42.519 Mitchell Wright: and and see if we could get that. I mean, yeah, that should be pretty straightforward, and they could probably even get us that Monday.
221 00:25:43.690 ⇒ 00:25:44.610 Uttam Kumaran: Okay, cool.
222 00:25:45.260 ⇒ 00:25:49.510 Uttam Kumaran: And then, yeah, we just worked on a couple of things for
223 00:25:50.120 ⇒ 00:25:52.890 Uttam Kumaran: stripe. And we’re sort of continuing on
224 00:25:53.804 ⇒ 00:26:00.199 Uttam Kumaran: like working on subscriptions. And then, yeah, we couple of things, I think in order to do that, we need from
225 00:26:02.320 ⇒ 00:26:05.229 Uttam Kumaran: we need from users and orgs and things like that.
226 00:26:06.550 ⇒ 00:26:09.279 Uttam Kumaran: So that’s like, I think, the core blocker here. But
227 00:26:10.240 ⇒ 00:26:17.379 Uttam Kumaran: I don’t. I don’t think we have like much other blockers. I think the next steps on stripe are pretty clear. We’re just working through trying to get everything we need there.
228 00:26:18.232 ⇒ 00:26:21.780 Uttam Kumaran: And then we just confirm that everything’s sort of running on schedule. Now.
229 00:26:24.520 ⇒ 00:26:29.239 Uttam Kumaran: I also think that we’re not paying for polytomic yet, so I think I would. I would rather just wait until
230 00:26:29.390 ⇒ 00:26:32.409 Uttam Kumaran: everything sort of hooked up before getting the quote from him.
231 00:26:32.910 ⇒ 00:26:33.400 Mitchell Wright: Yeah.
232 00:26:34.420 ⇒ 00:26:35.060 Uttam Kumaran: So.
233 00:26:35.060 ⇒ 00:26:39.379 Mitchell Wright: Yeah, cause, he said, he said, it’ll be best if we just can kind of see.
234 00:26:39.590 ⇒ 00:26:44.370 Mitchell Wright: yeah, what that’s gonna look like. With getting actual data flowing through. So
235 00:26:45.233 ⇒ 00:26:51.589 Mitchell Wright: and I think he’s open to it, taking longer now, cause they kind of fucked up.
236 00:26:52.050 ⇒ 00:26:56.679 Uttam Kumaran: Yeah, which is good. I mean, I’m glad they’re helpful. It’s a.
237 00:26:57.400 ⇒ 00:26:58.160 Mitchell Wright: Yeah.
238 00:26:58.290 ⇒ 00:27:03.650 Mitchell Wright: no, I mean, that’s 1 of the things I like about them is they’re very quick to respond on support stuff. And.
239 00:27:03.650 ⇒ 00:27:04.330 Uttam Kumaran: Yeah.
240 00:27:04.330 ⇒ 00:27:05.260 Mitchell Wright: Bookings.
241 00:27:05.470 ⇒ 00:27:06.260 Mitchell Wright: So
242 00:27:08.410 ⇒ 00:27:25.856 Mitchell Wright: yeah, and and some of it. So Ryan, like, for example, your questions, yeah, some of this stuff I haven’t actually gone through and cleaned all of the models up. So if there’s something yeah, exactly, just market with Red, and I’ll come back through, and I’ll be able to identify where?
243 00:27:27.810 ⇒ 00:27:32.659 Mitchell Wright: if something is like, yeah, just ignore it for now, or if we actually do need it.
244 00:27:33.120 ⇒ 00:27:40.319 Luke Daque: Yeah, like some of these we might already have. But we just need some sort of calculation or logic in them.
245 00:27:40.320 ⇒ 00:27:41.609 Mitchell Wright: Yeah, yeah, give me a better.
246 00:27:41.610 ⇒ 00:27:41.980 Luke Daque: Big ass.
247 00:27:41.980 ⇒ 00:27:43.120 Mitchell Wright: Requirements.
248 00:27:43.310 ⇒ 00:27:53.810 Luke Daque: Yeah, that would be great like. And like you mentioned, some of these may be coming from postgres. So we’ll have to wait until we have that data in snowflake and stuff like that. So, yeah.
249 00:27:55.840 ⇒ 00:28:02.229 Uttam Kumaran: But I think probably by early next week we’ll have a couple of things that we’re just gonna we’ll just deploy real and you can
250 00:28:02.400 ⇒ 00:28:11.560 Uttam Kumaran: run it locally, or we can even deploy Cloud. And I think they’ll be okay for a little bit if we want to just test it out that way you can start. You can just start seeing the data.
251 00:28:12.090 ⇒ 00:28:15.899 Uttam Kumaran: And also you can check out how you like rail.
252 00:28:16.640 ⇒ 00:28:17.270 Mitchell Wright: Yeah.
253 00:28:18.330 ⇒ 00:28:21.289 Uttam Kumaran: So let’s do that, Ryan. I think we’ll plan for next week to get
254 00:28:21.710 ⇒ 00:28:24.909 Uttam Kumaran: some of the stripe stuff in real by early next week.
255 00:28:25.830 ⇒ 00:28:27.870 Uttam Kumaran: Yeah, I kind of want to just attack
256 00:28:28.040 ⇒ 00:28:34.749 Uttam Kumaran: the entity other entities as soon as we get them. But I guess maybe we can. We’ll see like. If if Hubspot.
257 00:28:34.910 ⇒ 00:28:40.150 Uttam Kumaran: I’ll just try to see check again this weekend and see if I can confirm spot lands.
258 00:28:40.890 ⇒ 00:28:43.300 Uttam Kumaran: And then what do you think about for segment?
259 00:28:45.319 ⇒ 00:28:49.731 Mitchell Wright: Yeah, I need to just look at what that what that’s gonna take.
260 00:28:50.900 ⇒ 00:28:57.690 Mitchell Wright: let’s see segment. Let me just look real quick. Right now we have a couple of minutes segment, snowflay integration.
261 00:29:03.560 ⇒ 00:29:10.030 Mitchell Wright: Okay, here, let me just share my screen and let’s walk through this real together.
262 00:29:12.450 ⇒ 00:29:13.640 Mitchell Wright: Gay.
263 00:29:14.170 ⇒ 00:29:20.530 Mitchell Wright: So getting started? Oh, yeah, yeah, that’s right. Okay, so did you have a chance to look at this stuff and do any of this.
264 00:29:20.530 ⇒ 00:29:22.619 Uttam Kumaran: Oh, no! I did not.
265 00:29:22.620 ⇒ 00:29:26.389 Mitchell Wright: Okay, okay, no worries, no worries. That’s right. I remember, I did look at this. And I was like, Okay, great.
266 00:29:26.390 ⇒ 00:29:27.140 Uttam Kumaran: Right. You’re right.
267 00:29:28.410 ⇒ 00:29:30.520 Mitchell Wright: Steps, one through 5, I think.
268 00:29:31.100 ⇒ 00:29:32.610 Uttam Kumaran: Yes, you’re totally right.
269 00:29:32.610 ⇒ 00:29:35.780 Mitchell Wright: For the database, the role user.
270 00:29:36.860 ⇒ 00:29:41.490 Uttam Kumaran: Yeah, let me create. Let me create this, the segment user, and I’ll get you all that in like a.
271 00:29:42.340 ⇒ 00:29:43.210 Mitchell Wright: Yeah.
272 00:29:43.210 ⇒ 00:29:43.870 Uttam Kumaran: Perfect.
273 00:29:47.190 ⇒ 00:29:49.219 Mitchell Wright: Oops, did I? I told that wrong.
274 00:29:49.470 ⇒ 00:29:50.250 Mitchell Wright: There we go.
275 00:30:11.500 ⇒ 00:30:20.430 Uttam Kumaran: Yeah, okay, I will get you all of these as soon as I create the service user.
276 00:30:20.940 ⇒ 00:30:24.179 Uttam Kumaran: I’ll just send it as like a have you used like one time secret before.
277 00:30:25.947 ⇒ 00:30:29.619 Mitchell Wright: I don’t think so, but that works for me.
278 00:30:30.150 ⇒ 00:30:33.210 Uttam Kumaran: Kind of like this. It’s just this, you just can paste in
279 00:30:33.810 ⇒ 00:30:41.320 Uttam Kumaran: mean secrets, and then it’ll just delete the yeah. URL, after convenient thing. Yeah.
280 00:30:41.320 ⇒ 00:30:44.407 Mitchell Wright: So, yeah, so just I think all this stuff.
281 00:30:45.020 ⇒ 00:30:46.630 Uttam Kumaran: Sure. Yeah, I can get all this.
282 00:30:47.290 ⇒ 00:30:50.990 Mitchell Wright: And then and then I can go in and add that as a destination.
283 00:30:51.500 ⇒ 00:30:53.420 Mitchell Wright: And
284 00:30:56.000 ⇒ 00:31:05.130 Mitchell Wright: I don’t know if you create a network policy with Snowflake and are located in the Us. Oh, I don’t know if we need to allow list, Ips, but.
285 00:31:05.170 ⇒ 00:31:08.690 Uttam Kumaran: Oh, yeah, we’re we’re not. We don’t have an allow list right now.
286 00:31:09.310 ⇒ 00:31:10.860 Mitchell Wright: Okay, great. That’s fine.
287 00:31:17.810 ⇒ 00:31:20.266 Uttam Kumaran: And people usually do that. It’s like, if they
288 00:31:21.180 ⇒ 00:31:24.609 Uttam Kumaran: if they’re like, we only want people from our office, or like a couple.
289 00:31:24.610 ⇒ 00:31:25.120 Mitchell Wright: Yeah.
290 00:31:25.120 ⇒ 00:31:27.330 Uttam Kumaran: These places, and then you just do allow list. But
291 00:31:30.250 ⇒ 00:31:36.280 Uttam Kumaran: I feel like we have. If we do duo. If we do key pair, I’m like, really happy cool.
292 00:31:36.780 ⇒ 00:31:38.000 Uttam Kumaran: That won’t be okay.
293 00:31:38.000 ⇒ 00:31:39.690 Mitchell Wright: Yeah, let’s stick with that. Then.
294 00:31:40.850 ⇒ 00:31:44.590 Uttam Kumaran: Okay, cool. So then, yeah, I would love to see if we can get something.
295 00:31:44.700 ⇒ 00:31:49.819 Uttam Kumaran: And so in terms of a biz by next week. And then we’re just we’ll just keep pushing on
296 00:31:50.577 ⇒ 00:31:51.920 Uttam Kumaran: stripe. And yeah.
297 00:31:52.160 ⇒ 00:31:58.110 Uttam Kumaran: whatever you can get from the team in terms of the user or orgs data we can work with.
298 00:31:58.690 ⇒ 00:31:59.430 Uttam Kumaran: Okay.
299 00:31:59.830 ⇒ 00:32:03.790 Uttam Kumaran: So however, it’s easy on them doing the replica stuff. I,
300 00:32:04.120 ⇒ 00:32:05.970 Uttam Kumaran: it just smells like it’s gonna take.
301 00:32:06.230 ⇒ 00:32:06.900 Mitchell Wright: Yeah.
302 00:32:07.570 ⇒ 00:32:09.159 Uttam Kumaran: That’s a really good point.
303 00:32:09.430 ⇒ 00:32:14.849 Uttam Kumaran: Yeah, if they can, they can just run a select star and export it out of the beaver or whatever. Then that would be fine.
304 00:32:19.024 ⇒ 00:32:22.929 Mitchell Wright: Let’s see. Okay, it looks like they might have
305 00:32:25.527 ⇒ 00:32:29.850 Mitchell Wright: set up. Also the slack github integration.
306 00:32:30.550 ⇒ 00:32:37.440 Uttam Kumaran: Yeah. So you’ll just see basically, Prs come through that way.
307 00:32:37.930 ⇒ 00:32:41.350 Uttam Kumaran: Like, if we make comments and stuff, we just try to have it all
308 00:32:41.580 ⇒ 00:32:44.150 Uttam Kumaran: in one place in case we need reviews from you.
309 00:32:44.680 ⇒ 00:32:49.190 Uttam Kumaran: I mean, yeah, I don’t know. We can also help you set up local Dbt.
310 00:32:49.380 ⇒ 00:32:51.519 Uttam Kumaran: Next week. I don’t know again.
311 00:32:52.300 ⇒ 00:32:55.270 Uttam Kumaran: I don’t know how busy you are with other stuff. We can punt it.
312 00:32:56.130 ⇒ 00:32:59.460 Mitchell Wright: Yeah, let’s let’s maybe punt. Just cause right now I’m
313 00:32:59.580 ⇒ 00:33:01.530 Mitchell Wright: I’ve got a lot of other stuff that’s
314 00:33:02.060 ⇒ 00:33:08.039 Mitchell Wright: high priority there. But yeah, so. Documentation, I think, would be another thing of just
315 00:33:09.470 ⇒ 00:33:12.350 Mitchell Wright: so that it’s it’s easy for someone to come in, and
316 00:33:12.530 ⇒ 00:33:22.769 Mitchell Wright: even if they can’t talk to you, they could read the documentation and and get up to speed on how things are architected and and whatever. So if you guys have some time to do that as well. That would be fantastic.
317 00:33:23.220 ⇒ 00:33:26.190 Uttam Kumaran: Perfect. Yeah, we’ll I’ll put the documentation about the grants.
318 00:33:26.713 ⇒ 00:33:32.319 Uttam Kumaran: I’ll also put a put in our docs about how we structure dbt, and then
319 00:33:33.350 ⇒ 00:33:36.120 Uttam Kumaran: look when we do. When we do stripe
320 00:33:36.836 ⇒ 00:33:41.450 Uttam Kumaran: let’s just make we. I already added some comments, but just make sure there’s comments on any logic.
321 00:33:43.590 ⇒ 00:33:51.120 Uttam Kumaran: right as as now. It’s it’s like pretty simple, and then I’ll also put. I’ll also put documentation about the runners. The Github actions there.
322 00:33:51.542 ⇒ 00:33:58.887 Uttam Kumaran: That way you have it. And then, as as soon as you know, you guys bring on anyone, I’ll I’ll sort of walk them through it.
323 00:33:59.150 ⇒ 00:33:59.780 Mitchell Wright: Yeah.
324 00:34:00.250 ⇒ 00:34:02.280 Uttam Kumaran: So that’s perfect. I’ll have all that ready.
325 00:34:05.760 ⇒ 00:34:06.730 Mitchell Wright: All right.
326 00:34:07.129 ⇒ 00:34:14.669 Mitchell Wright: That sounds great to me. Thanks, guys. This is great love to see the progress. And I feel like we’re getting there slowly but surely.
327 00:34:14.969 ⇒ 00:34:18.829 Uttam Kumaran: Yeah, this is what happens when you start like uncovering the rocks
328 00:34:19.204 ⇒ 00:34:21.999 Uttam Kumaran: said, if the data starts coming in, then it becomes our
329 00:34:22.199 ⇒ 00:34:26.440 Uttam Kumaran: becomes more of our problems. And like waiting until that happens. So.
330 00:34:26.460 ⇒ 00:34:31.150 Mitchell Wright: Yeah, I want to get to a point where it’s like we’re, we’re set up on
331 00:34:31.659 ⇒ 00:34:36.549 Mitchell Wright: that side. And we don’t really need them anymore. And then we can just crank on our stuff.
332 00:34:36.960 ⇒ 00:34:42.229 Uttam Kumaran: Yeah, I’m glad, though, like I, I don’t think we’ll really have much to do in Snowflake for a while, so
333 00:34:43.840 ⇒ 00:34:45.060 Uttam Kumaran: sort of set up there.
334 00:34:46.820 ⇒ 00:34:50.660 Mitchell Wright: Alright. Sounds good guys. We’ll talk we’ll talk next week.
335 00:34:51.440 ⇒ 00:34:51.940 Uttam Kumaran: Thank you.
336 00:34:51.940 ⇒ 00:34:52.530 Luke Daque: Thanks.
337 00:34:52.530 ⇒ 00:34:53.440 Mitchell Wright: Okay. Bye-bye.