Meeting Title: Brainforge x CTA: Weekly! Date: 2026-01-23 Meeting participants: Ashwini Sharma, Kyle Wandel, Uttam Kumaran
WEBVTT
1 00:01:59.960 ⇒ 00:02:01.110 Ashwini Sharma: Hey, Kai.
2 00:02:03.160 ⇒ 00:02:04.700 Kyle Wandel: Hey, Ashwini, how you doing?
3 00:02:04.700 ⇒ 00:02:05.890 Ashwini Sharma: I’m good, how are you?
4 00:02:06.180 ⇒ 00:02:07.170 Kyle Wandel: Oh, good.
5 00:02:12.870 ⇒ 00:02:20.140 Kyle Wandel: So in the U.S, it’s supposed to have a major storm in the South and the east this, like, next few days.
6 00:02:20.140 ⇒ 00:02:20.520 Ashwini Sharma: Oh.
7 00:02:20.520 ⇒ 00:02:24.179 Kyle Wandel: Everybody’s… everything, everybody’s panicking, yeah, about it.
8 00:02:24.790 ⇒ 00:02:26.939 Ashwini Sharma: Are you in the south or east?
9 00:02:27.370 ⇒ 00:02:35.279 Kyle Wandel: Yeah, I’m in the… I’m on the East Coast, so we’ll probably get hit by a lot of snow, hopefully, so we’ll see. Like…
10 00:02:35.760 ⇒ 00:02:38.599 Kyle Wandel: 12 inches, at least, is what they’re saying, so we’ll see.
11 00:02:38.600 ⇒ 00:02:41.210 Ashwini Sharma: Whoa, that’s… that’s a lot here.
12 00:02:41.210 ⇒ 00:02:48.940 Kyle Wandel: I… I doubt that it’ll be that much, but, we’ll see. It’s always, like, overhyped. It’s usually probably, like, half of what they say.
13 00:02:50.700 ⇒ 00:02:53.279 Ashwini Sharma: It’s frustrating, right, after such a huge snow?
14 00:02:54.000 ⇒ 00:03:09.030 Kyle Wandel: Yeah, basically. It’s, it’s not too, like, it’s not too bad. I’m from the Midwest, so, snow doesn’t really bother me too much, like, I’m used to it, but, nobody here in DC can, like, can deal with it, if that makes sense.
15 00:03:14.120 ⇒ 00:03:20.270 Kyle Wandel: one year, the last time we had a really bad storm, it took somebody I know
16 00:03:21.190 ⇒ 00:03:32.309 Kyle Wandel: 6 hours to drive normally what takes an hour. So, it’s, like, one of those things where if something in DC, like, shuts down, like, the entire city shuts down.
17 00:03:32.950 ⇒ 00:03:37.549 Ashwini Sharma: Over here, we have landslides, like, it doesn’t snow where I live.
18 00:03:37.740 ⇒ 00:03:41.420 Ashwini Sharma: But there’s a lot of landslides. It rains heavily.
19 00:03:45.340 ⇒ 00:03:47.699 Kyle Wandel: Yeah, I’ve never… because you’re in India, right?
20 00:03:47.700 ⇒ 00:03:48.500 Ashwini Sharma: Yeah.
21 00:03:48.990 ⇒ 00:03:52.799 Kyle Wandel: Yeah, I’ve never been there. My wife is actually gonna go,
22 00:03:53.380 ⇒ 00:04:04.409 Kyle Wandel: a little bit later this year, but we’re… we’re… she’s pregnant right now, so I think the due date would be, like, within a month or something of the… of the trip, so the doctor said, yeah, I shouldn’t do that, but…
23 00:04:04.410 ⇒ 00:04:06.840 Ashwini Sharma: Yeah, yeah, no, not a good idea.
24 00:04:07.530 ⇒ 00:04:09.040 Kyle Wandel: We loved you go one time.
25 00:04:09.240 ⇒ 00:04:10.450 Ashwini Sharma: Sure, sure.
26 00:04:11.640 ⇒ 00:04:13.389 Ashwini Sharma: There’s a lot of nice places.
27 00:04:16.010 ⇒ 00:04:18.419 Kyle Wandel: Yeah, I’m a really big,
28 00:04:18.510 ⇒ 00:04:37.639 Kyle Wandel: there’s not as much in India, but I’m a really big, like, history buff, so I want to go see, like, all of the, not the monuments, what I call it, like, the wonders, the wonders, both ancient and current modern. And I want to go… I know it’s not in… but the Taj Mahal, I know it’s in Indiana, but I want to go see that real bad.
29 00:04:38.260 ⇒ 00:04:42.200 Ashwini Sharma: Oh, yeah, there are some that’s really fascinating, right?
30 00:04:43.600 ⇒ 00:04:48.970 Uttam Kumaran: to see Taj Mahal, like, A year and a half ago, and it was like… Completely worth it.
31 00:04:49.230 ⇒ 00:04:52.609 Uttam Kumaran: It’s, like, the craziest thing. It’s, like, one of the craziest things I’ve ever seen.
32 00:04:53.730 ⇒ 00:04:54.200 Ashwini Sharma: Osman.
33 00:04:54.200 ⇒ 00:04:58.060 Uttam Kumaran: And you, like, you enter, like, into this from this, like.
34 00:04:58.500 ⇒ 00:05:03.970 Uttam Kumaran: Sort of thing, and there’s, like, one entrance, and then, like, you walk, and it, like, opens up, and it’s, like…
35 00:05:04.400 ⇒ 00:05:09.210 Uttam Kumaran: This is, like, it’s just, like, insane. I don’t know, I was, like, I’m not usually…
36 00:05:09.580 ⇒ 00:05:15.809 Uttam Kumaran: impressed, but I was like, holy crap, it’s dope.
37 00:05:15.810 ⇒ 00:05:22.140 Kyle Wandel: I feel like, I mean, the pictures obviously can’t… don’t do it justice. I almost feel like it’s, like, the Great… the Grand Canyon, in terms of just, like.
38 00:05:22.140 ⇒ 00:05:23.190 Uttam Kumaran: Yeah.
39 00:05:23.190 ⇒ 00:05:23.660 Kyle Wandel: Yes.
40 00:05:23.660 ⇒ 00:05:29.840 Uttam Kumaran: see, like, the depth, you’re like, whoa. Almost like your brain kind of, like, zooms out, yeah.
41 00:05:30.160 ⇒ 00:05:33.090 Ashwini Sharma: Taz Mahal is overhyped, trust me.
42 00:05:33.300 ⇒ 00:05:34.240 Uttam Kumaran: You think so?
43 00:05:34.240 ⇒ 00:05:35.330 Ashwini Sharma: Yeah, it is a win.
44 00:05:35.330 ⇒ 00:05:37.309 Uttam Kumaran: It’s definitely crowded. It’s definitely crowded.
45 00:05:37.310 ⇒ 00:05:37.710 Kyle Wandel: Yeah.
46 00:05:37.710 ⇒ 00:05:39.110 Uttam Kumaran: I think it’s worth seeing.
47 00:05:39.840 ⇒ 00:05:52.550 Kyle Wandel: I think the interesting part about it is that there’s, like, there’s just nothing in the area. So, like, at least from the picture perspective, like, it just looks like it’s literally just so picture-act all the time, there’s nothing around it, so, like, it’s, like, a perfect…
48 00:05:52.550 ⇒ 00:05:57.300 Uttam Kumaran: There’s still, like, nothing inside of it. There’s nothing… it’s, like, not a… there’s not, like, things to do. It’s, like…
49 00:05:57.760 ⇒ 00:06:04.180 Uttam Kumaran: It’s a mausoleum like… But it’s just, like… it’s just, like, crazy, I don’t know.
50 00:06:05.000 ⇒ 00:06:05.799 Kyle Wandel: I guess, for example.
51 00:06:05.800 ⇒ 00:06:12.500 Uttam Kumaran: Always had a lot of time on their hands. Well, so the king basically built it for one of his wives, who died.
52 00:06:13.030 ⇒ 00:06:18.330 Uttam Kumaran: I think. I feel like, I think that’s the story. So, in, like, in her honor.
53 00:06:18.470 ⇒ 00:06:21.810 Uttam Kumaran: Or maybe before she died, I don’t know, but then…
54 00:06:21.940 ⇒ 00:06:30.730 Uttam Kumaran: Like, she’s buried there. That’s it. Like, it’s, like, not much else to it. But there’s a lot of… but in that area, is it what? Is it Agra Ashwini?
55 00:06:30.730 ⇒ 00:06:32.299 Ashwini Sharma: Yeah, it’s Agatha, Agra.
56 00:06:32.300 ⇒ 00:06:34.340 Uttam Kumaran: In Agra, there’s a lot of forks.
57 00:06:34.580 ⇒ 00:06:40.609 Uttam Kumaran: So, that’s also, like, one of the things that we did. There’s a lot of these, like, hillside forts where
58 00:06:40.720 ⇒ 00:06:51.139 Uttam Kumaran: they basically, on the side of the mountain, carved out these, like, areas where you see, like, all the places they put cannons, and, like, overlooked the city. And, like, Agra’s very, very famous for the forts as well.
59 00:06:51.600 ⇒ 00:06:55.250 Uttam Kumaran: Yeah, we just spent a day there, doing that.
60 00:06:58.330 ⇒ 00:07:15.439 Kyle Wandel: Yeah, I was just telling Ashwini, we got a… my wife and I really want to, so she was about to go to a work conference in India, this past year, but not this past year, but, like, within the early part of this year, but we’re pregnant, and she’s pregnant and due in late April, and so because of that, we can’t really…
61 00:07:15.440 ⇒ 00:07:20.729 Kyle Wandel: travel during, like, the month of February, but really, really want to go real bad.
62 00:07:21.570 ⇒ 00:07:22.110 Uttam Kumaran: Yeah.
63 00:07:22.500 ⇒ 00:07:29.220 Ashwini Sharma: Yeah, if you come, you should go to that place. It’s in Maharashtra somewhere near Aurangabad, or something.
64 00:07:29.470 ⇒ 00:07:32.130 Ashwini Sharma: It’s called the Kailasa Temple, right?
65 00:07:32.520 ⇒ 00:07:37.690 Ashwini Sharma: if you see that, I mean, Taj comes nowhere closer to that. Oh, really? Yeah.
66 00:07:37.690 ⇒ 00:07:38.630 Kyle Wandel: What is it called?
67 00:07:38.630 ⇒ 00:07:39.380 Uttam Kumaran: Okay.
68 00:07:39.380 ⇒ 00:07:40.720 Ashwini Sharma: Kailasa Temple.
69 00:07:43.250 ⇒ 00:07:44.189 Uttam Kumaran: How do you spell it?
70 00:07:44.190 ⇒ 00:07:46.180 Ashwini Sharma: Let me type it over here.
71 00:07:46.550 ⇒ 00:07:47.360 Uttam Kumaran: with K.
72 00:07:47.600 ⇒ 00:07:48.230 Ashwini Sharma: Yes.
73 00:07:51.370 ⇒ 00:07:58.830 Ashwini Sharma: It’s a rock-cut temple, and unlike the other temples, which are, like, you know, front to back.
74 00:07:58.980 ⇒ 00:08:03.320 Ashwini Sharma: It was cut down from top to bottom. It’s a single rock.
75 00:08:07.490 ⇒ 00:08:08.210 Kyle Wandel: Oh, wow.
76 00:08:08.680 ⇒ 00:08:11.349 Kyle Wandel: The Ramar Temple? No, yeah.
77 00:08:12.280 ⇒ 00:08:13.140 Ashwini Sharma: Let me see…
78 00:08:13.140 ⇒ 00:08:16.399 Kyle Wandel: I’ll look at, I’ll look at the, yeah, I’ll look at the Slack.
79 00:08:26.260 ⇒ 00:08:27.140 Ashwini Sharma: This one.
80 00:08:37.900 ⇒ 00:08:38.640 Kyle Wandel: Let’s true.
81 00:08:41.190 ⇒ 00:08:42.859 Kyle Wandel: Oh, wow, that is pretty cool.
82 00:08:43.100 ⇒ 00:08:45.110 Kyle Wandel: Like, the aerial view of it?
83 00:08:49.460 ⇒ 00:08:51.409 Ashwini Sharma: That’s one single rock.
84 00:08:52.870 ⇒ 00:08:53.779 Kyle Wandel: That’s kinda crazy.
85 00:08:53.780 ⇒ 00:08:54.450 Uttam Kumaran: Damn.
86 00:08:57.390 ⇒ 00:09:03.629 Kyle Wandel: It reminds me of, like, what the, like, I guess, like, the Romans did, but obviously on a bigger, like, a slightly bigger scale.
87 00:09:09.180 ⇒ 00:09:12.089 Kyle Wandel: What are the… because I was just in…
88 00:09:13.380 ⇒ 00:09:16.640 Kyle Wandel: Rome, like, a few years ago. I can’t remember there was, like, the…
89 00:09:17.100 ⇒ 00:09:20.130 Kyle Wandel: The yards, maybe? I can remember what I can’t know what it’s called.
90 00:09:24.740 ⇒ 00:09:30.590 Kyle Wandel: I think Catherine’s joining, but I’m not… I haven’t heard anything. So, she may not be joining.
91 00:09:34.230 ⇒ 00:09:40.050 Kyle Wandel: I knew that she was, meeting with… Christina?
92 00:09:40.150 ⇒ 00:09:48.029 Kyle Wandel: and a couple other people about something, but I don’t… and I think it was today. I knew it was today, but I don’t know when it was today.
93 00:09:48.330 ⇒ 00:09:49.650 Kyle Wandel: But I just reached out.
94 00:09:50.730 ⇒ 00:09:51.240 Uttam Kumaran: Okay.
95 00:10:01.340 ⇒ 00:10:05.500 Uttam Kumaran: Yeah, worst case, we can just use this time, or… Yeah, whatever.
96 00:10:07.250 ⇒ 00:10:19.829 Kyle Wandel: Yeah, well, yesterday, actually, Ashwini and I were on the call for, like, 2 hours, trying to figure out, or creating a dbt model, so I really appreciate that. Like, so that was really, really cool to be able to upload my first, like, basically upload the first table slash dbt.
97 00:10:19.830 ⇒ 00:10:20.750 Uttam Kumaran: Nice.
98 00:10:20.750 ⇒ 00:10:21.909 Kyle Wandel: Like, yeah.
99 00:10:22.320 ⇒ 00:10:28.159 Kyle Wandel: That… that’s the… I think the bigger issue now is trying to go from…
100 00:10:29.290 ⇒ 00:10:42.310 Kyle Wandel: just understanding what I need, like, what I need to do. So, like, for example, obviously, Snowflake, you can still create the queries and do all the things in the queries, but it’s probably better to… I mean, this is what you were talking about yesterday, Ashwini, but basically just do everything
101 00:10:42.860 ⇒ 00:10:45.019 Kyle Wandel: in the DBT model.
102 00:10:45.200 ⇒ 00:10:45.830 Uttam Kumaran: Yes.
103 00:10:46.060 ⇒ 00:10:52.859 Kyle Wandel: create, even then, create summaries and create the various reports that you may need in SQL.
104 00:10:52.860 ⇒ 00:11:05.339 Uttam Kumaran: There’s just very little cost. Yeah. So, like, you can create them as views, and it’s just helpful to have all the logic there, versus going into Power BI or an external to then layer on again
105 00:11:05.580 ⇒ 00:11:08.739 Uttam Kumaran: More segmentation or calculations?
106 00:11:09.020 ⇒ 00:11:21.370 Uttam Kumaran: Because they can get reused. And then, yeah, like, arranging, like, hey, I actually just need, like, a couple of this, and I need, like, a fixed aggregation, and maybe I just want to create a table for that. Go ahead and do that, it’s like…
107 00:11:22.450 ⇒ 00:11:27.099 Uttam Kumaran: You know, way easier, and it maintains a whole lineage, versus…
108 00:11:27.520 ⇒ 00:11:29.750 Uttam Kumaran: Adding an extra layer on top.
109 00:11:29.860 ⇒ 00:11:36.229 Uttam Kumaran: This has always been an argument, you know, in data about, like, how do you centralize logic? And so…
110 00:11:36.330 ⇒ 00:11:37.589 Uttam Kumaran: We found that
111 00:11:38.140 ⇒ 00:11:47.680 Uttam Kumaran: sometimes the BI layer introduces this, like, ability to do logic there, and you just want to try to avoid it if possible, because it’s really hard to govern.
112 00:11:49.950 ⇒ 00:12:06.530 Kyle Wandel: No, that’s a good point, because I think that’s one of the biggest things that we’ve struggled with with Power BI, is getting too much out there, and not just doing, like, typical counts, like, the biggest thing is, like, lists, but that’s easier pulled from Snowflake than, like, the counts and the aggregations of stuff.
113 00:12:19.590 ⇒ 00:12:26.029 Kyle Wandel: So the idea is, like, basically, like, Snowflake is, like, not really the end-all, be-all, but, like, that we should do very minimal…
114 00:12:27.510 ⇒ 00:12:34.630 Kyle Wandel: querying of raw… not raw data, but yeah, but querying of raw data, basically, in Snowflake, or… so I guess the way…
115 00:12:34.630 ⇒ 00:12:35.400 Uttam Kumaran: like…
116 00:12:35.730 ⇒ 00:12:36.140 Kyle Wandel: Yeah, go ahead.
117 00:12:36.140 ⇒ 00:12:37.039 Uttam Kumaran: Yeah, go ahead, go ahead.
118 00:12:37.640 ⇒ 00:12:45.060 Kyle Wandel: I was gonna say, like, the typical way I’ve typically done it in the past is just have a, like, this is very basic stuff, but have the flat table.
119 00:12:45.130 ⇒ 00:13:03.320 Kyle Wandel: Whether it be from a raw or staging or development standpoint, so have that flat table of data, basically, and then you can use Power BI, or Tableau, or another visualization software, or even Excel, to then do your calculations, then to create a report. But it sounds like what’s better is to do…
120 00:13:03.640 ⇒ 00:13:08.300 Kyle Wandel: Do most of that, or the create, like, of the new report or table.
121 00:13:08.450 ⇒ 00:13:18.739 Kyle Wandel: that has all of your aggregations in SQL, and then put that into Snowflake, and then from Snowflake, from there, they can then do visualizations if need be.
122 00:13:19.630 ⇒ 00:13:26.219 Uttam Kumaran: Yeah, so all of this is actually in Snowflake, so I think better, probably, terminology to keep a reference is, like.
123 00:13:26.370 ⇒ 00:13:42.109 Uttam Kumaran: the raw versus, like, what’s in dbt, but, like, in dbt doesn’t… it’s technically in Snowflake. Like, dbt is just the code that executes within Snowflake, right? So, you’re totally right, everything’s in Snowflake. But the kind of… the key thing to think about is that
124 00:13:42.330 ⇒ 00:14:00.120 Uttam Kumaran: Snowflake will have multiple customers or multiple tenants. BI may be one. Integrations may be another. Analysts like us may query directly. So you want all those people pulling from the same thing. You don’t want, like, Power BI to have some logic that, like, oh, how does that…
125 00:14:00.200 ⇒ 00:14:09.099 Uttam Kumaran: column work, oh, like, it’s some logic there, but I need to get that somewhere, I can have to rewrite it in something else, so, like, centralizing that, you know, helps.
126 00:14:09.380 ⇒ 00:14:11.850 Uttam Kumaran: Kind of bring that all into one place.
127 00:14:12.180 ⇒ 00:14:16.280 Kyle Wandel: So Snowflake essentially is like a middleman, is the best way to describe it.
128 00:14:17.050 ⇒ 00:14:21.310 Uttam Kumaran: Yeah, it is a middleman, and yeah, simple way to describe it,
129 00:14:21.460 ⇒ 00:14:26.010 Uttam Kumaran: And ideally, you… we don’t want any tenant of…
130 00:14:26.290 ⇒ 00:14:29.759 Uttam Kumaran: the data to actually go to… straight to raw.
131 00:14:30.100 ⇒ 00:14:34.739 Uttam Kumaran: Like, everything should come through a governed layer that we’re approving
132 00:14:34.910 ⇒ 00:14:47.389 Uttam Kumaran: like, anybody at the BI layer or above can get access to. Because ultimately, like, it’s so… it would be… it’s so challenging to go and replicate how does things join, what IDs do I use, how do we clean this up?
133 00:14:47.660 ⇒ 00:15:03.709 Uttam Kumaran: like, we can’t have that happen every time, especially as our data sources are gonna increase. And for us, we just want to do that, like, one time for every data source, and then be like, hey guys, if you have questions about membership, here are, like, the 10 tables in the membership schema.
134 00:15:04.050 ⇒ 00:15:17.999 Uttam Kumaran: for the most part, you’ll be able to… it’s gonna be clear how to join those if you need to, but, like, you’re gonna be able to answer that there. And then also, like, even if you don’t get it from there, and you’re more of, like, a BI user, that’s what’s powering your BI tables also.
135 00:15:18.110 ⇒ 00:15:23.129 Uttam Kumaran: So everybody’s sort of talking the same language, and then ideally, yes, you want to really prevent
136 00:15:23.410 ⇒ 00:15:42.529 Uttam Kumaran: Anything that isn’t, like, ad hoc calculation, you want to really push that down into Snowflake. So, like, still, people are adding and creating a custom metric or something, like, that’s… that’s safe, that’s whatever, but if it ends up being something that, like, needs to get used more often, or, like, a business definition, like, you can think about,
137 00:15:43.700 ⇒ 00:15:48.199 Uttam Kumaran: like… like, customer segments, right? Like.
138 00:15:48.340 ⇒ 00:16:03.339 Uttam Kumaran: one person just develops that in the BI tool, and then, like, it gets used, and then they come back to us, they’re like, hey, who defined the segment? There’s sort of this lack of governance, right? And so we’re signing up as the data team to basically be like, hey, we’re… we’re gonna own all these definitions.
139 00:16:03.540 ⇒ 00:16:06.759 Uttam Kumaran: And so, like, centralizing the place where we can manage them.
140 00:16:06.900 ⇒ 00:16:18.200 Uttam Kumaran: and then use it in other places, right? So, like, customer segment is probably going to get used not only in memberships, in badge, in other stuff, and so, like, making that definition
141 00:16:18.770 ⇒ 00:16:21.199 Uttam Kumaran: Permeate throughout the layer is important.
142 00:16:23.690 ⇒ 00:16:26.040 Uttam Kumaran: But again, sometimes, like, again, teams…
143 00:16:26.270 ⇒ 00:16:32.000 Uttam Kumaran: BI tools will sell themselves as, like, we can do this. It’s up to the team to be like, we don’t need to do this there.
144 00:16:32.250 ⇒ 00:16:39.849 Uttam Kumaran: Like, that’s… that’s the challenge, I feel like, in data, is that every tool in the stack will be like, do everything in my part, and…
145 00:16:40.180 ⇒ 00:16:43.390 Uttam Kumaran: We… we have to do what’s best for us.
146 00:16:43.730 ⇒ 00:17:03.389 Uttam Kumaran: And typically, it’s, like, using the tool for what it’s good for, not, like, centralizing everything. Because Tableau and Power BI, they were like, oh, this is an end-to-end data platform to do everything here. But their expertise is in visualization and dashboarding and reporting, not in modeling and ETL. But they will…
147 00:17:03.910 ⇒ 00:17:06.270 Uttam Kumaran: They will just pitch that they can do everything, you know?
148 00:17:06.980 ⇒ 00:17:14.949 Kyle Wandel: Yeah, that makes sense. I think the biggest… the biggest challenge that I will struggle with is, understanding when to…
149 00:17:16.420 ⇒ 00:17:33.650 Kyle Wandel: because basically what you do is you create a new view in the dbt model, and that view is what your new tabular data is, if… if need be, if you need tabular data, or you just create the new view for your report, if you want to do aggregations and stuff like that, based on the tabular data that you already
150 00:17:33.760 ⇒ 00:17:49.660 Kyle Wandel: cleaned up, basically. I think the biggest… the biggest… not the issue, the biggest change I’ll have to do with is that I’m so used to just playing with tabular data, basically, so whether it be going from raw to staging to prod, or just be raw to…
151 00:17:49.660 ⇒ 00:18:01.690 Kyle Wandel: broad. I’m so used to tabular data that I… and I’ve used Power BI and Tableau and other visualization softwares in order to then visualize and aggregate that data, obviously Excel, but,
152 00:18:01.750 ⇒ 00:18:02.840 Kyle Wandel: It’ll be…
153 00:18:02.840 ⇒ 00:18:11.730 Uttam Kumaran: For simple… for simple aggregations and, like, rolling things up, that’s not a problem. I think it’s really when you’re, like, doing super complex joins.
154 00:18:11.810 ⇒ 00:18:26.850 Uttam Kumaran: Or, like, further adding business logic on top, you just have to be wary. Look, I think it’s… for everything, it’s gonna be a decision. Like, if it’s something that’s just needed urgently, you can just do it, and then eventually you should… we should find a way to get… move that into the BI layer.
155 00:18:26.850 ⇒ 00:18:34.209 Uttam Kumaran: So this isn’t like a, oh, this is the way it works, like, sort of in this situation, you have to make a call, like, hey, I just need to write this thing and get…
156 00:18:34.400 ⇒ 00:18:38.450 Uttam Kumaran: Get, like, this logic out, but eventually, you want to push that into…
157 00:18:38.560 ⇒ 00:18:45.619 Uttam Kumaran: into dbt. So, like, simple aggregations and things are totally fine. Like, you don’t have to create a table for, like, every single cut.
158 00:18:45.820 ⇒ 00:18:52.409 Uttam Kumaran: you… maybe it’s better to just create a wide table, and then start to aggregate that in the BI tool. Like, that… I guess, like, that type of…
159 00:18:52.680 ⇒ 00:18:56.140 Uttam Kumaran: That type of logic is, like.
160 00:18:56.420 ⇒ 00:19:13.929 Uttam Kumaran: safe. It’s like, you know, no problem with that. Like, you don’t need to create, like, a weekly table, a daily table, monthly, like, just, we’re gonna create a daily, you can aggregate it up, and everything’s safe. But for example, if there are averages or, like, percentages that, like, maybe don’t aggregate up.
161 00:19:14.090 ⇒ 00:19:15.360 Uttam Kumaran: That’s where, like.
162 00:19:15.930 ⇒ 00:19:28.080 Uttam Kumaran: is everyone aware of that? That if you… if you then move this average and sum all the averages, you’re not… it’s, like, not working, you know? So, that’s the kind of situation where this could, like, potentially break.
163 00:19:28.870 ⇒ 00:19:30.110 Kyle Wandel: That makes sense.
164 00:19:31.630 ⇒ 00:19:37.849 Kyle Wandel: And then I guess the, like, the… so, like, if you have just… I’m sorry I’m hitting with all the questions, but I figured this is good.
165 00:19:37.850 ⇒ 00:19:42.960 Uttam Kumaran: No, no, no, please, please, please. I, like, love talking about this. Yeah, this is all we do, so…
166 00:19:43.220 ⇒ 00:19:45.799 Uttam Kumaran: So, like, let’s say we have just…
167 00:19:45.940 ⇒ 00:19:52.989 Kyle Wandel: the registration data environment, basically. So, like, let’s say that’s the schema, so we have multiple years of registration data.
168 00:19:53.250 ⇒ 00:19:58.580 Kyle Wandel: what… basically, what would you… for the DBT model, would you…
169 00:19:59.380 ⇒ 00:20:04.199 Kyle Wandel: Just ingest and clean those datasets, and then combine them.
170 00:20:04.510 ⇒ 00:20:07.530 Kyle Wandel: Or is it more of just, like, a…
171 00:20:09.910 ⇒ 00:20:16.039 Kyle Wandel: I’m not gonna say do counts, but, like, would that include, like, joining tables, or would that include.
172 00:20:16.040 ⇒ 00:20:22.350 Uttam Kumaran: Yeah, so we… so the kind of good way to think about it is, like, what are the final tables that we want to arrive at?
173 00:20:22.440 ⇒ 00:20:37.559 Uttam Kumaran: Right, so on the membership side, do we need to arrive at, like, DIM memberships? Like, DIM organizations, DIM members? And then, so, like, memberships may be a mapping between members and the organization. And then you may have, like,
174 00:20:37.810 ⇒ 00:20:44.310 Uttam Kumaran: You may have, like, a fact table where, yes, you’re actually going from member to the organization to, like.
175 00:20:44.420 ⇒ 00:21:03.690 Uttam Kumaran: different aggregations, like… or, for example, let’s say you have, like, fact organizations. You may have the information about when the organization joined, the number of members, how much money they spent with us, blah blah blah, right? So those are, like, aggregations, simple aggregations that, like.
176 00:21:03.980 ⇒ 00:21:10.430 Uttam Kumaran: They’re gonna get used every time, so we… the data team might as well ship that. Facilitate the joins and ship that.
177 00:21:10.550 ⇒ 00:21:25.400 Uttam Kumaran: And so, like, that’s how to think about it. It’s, like, the mart will be the primitives, which is, like, hey, we have everything around memberships is here, as well as some, like, really helpful summary tables, because not everybody is going to be equipped to…
178 00:21:25.800 ⇒ 00:21:43.310 Uttam Kumaran: rejoin things, but, like, for an analyst, you should be able to go to that, understand the joins, and then, like, okay, I can tack on other stuff if needed. Like, so there will be primary keys and memberships that are also in the Salesforce data, in other data, and so, like, if you need to join that up, you totally can.
179 00:21:43.560 ⇒ 00:22:00.020 Uttam Kumaran: But we will… I think, basically, I like to think about it as, like, we kind of do the 80-20, like, if we can… if we know most of the questions are gonna be, like, how many things, like, blah blah blah, let’s just answer them, and, like, create… create a table for that reporting. And then, like, if needed to add more.
180 00:22:00.260 ⇒ 00:22:12.210 Uttam Kumaran: One is, like, okay, if it’s ad hoc, like, go ahead and do it, that’s fine. If this is like, hey, actually, I just want to add a column, because it’s going to help for a bunch of future asks, then, like, okay, let’s just modify it and add that logic.
181 00:22:13.290 ⇒ 00:22:25.780 Kyle Wandel: That makes sense. And then for… this is my… For… in order to understand the business, like, in order to create the dbt and the business logic, do you create, like, a table that contains various business logic for a certain type of, like.
182 00:22:26.120 ⇒ 00:22:31.039 Kyle Wandel: metrics, basically? Or, like, how does that… or do you just… is it in the work, like…
183 00:22:31.180 ⇒ 00:22:35.229 Kyle Wandel: I mean, it’s probably a combination of both, in the where clause and just, like, aggregation.
184 00:22:35.230 ⇒ 00:22:53.289 Uttam Kumaran: Yeah, so, like, you can think about logic as, like, case whens, casting, like, a WHERE clause, like, anything in the where clause, window functions, right? All of that is gonna live in that, like, intermediate layer. So, we have, like, raw staging, intermediate.
185 00:22:53.470 ⇒ 00:22:56.990 Uttam Kumaran: fraud, and so intermediate is where we do, like, all the muscle work.
186 00:22:57.380 ⇒ 00:23:08.360 Uttam Kumaran: So all of that will live in those SQL files. That’s the… those are the pieces, though, that, like, it’s not relevant to every, like, person that’s querying to understand that.
187 00:23:08.570 ⇒ 00:23:15.730 Uttam Kumaran: Right? And we don’t want them to have to do that every time, so we’ll handle the load of basically doing that and maintaining.
188 00:23:15.810 ⇒ 00:23:31.179 Uttam Kumaran: But yeah, all that logic will be in that step. So, case whens, window functions, sums, some case whens, where clauses, and we’ll put comments, like, hey, we’re filtering out this because this is, like, a backfill, or, like.
189 00:23:31.270 ⇒ 00:23:33.550 Uttam Kumaran: Something is,
190 00:23:33.940 ⇒ 00:23:39.700 Uttam Kumaran: Yeah, like, we always filter this out for this reason, but these are all the gotchas that we just want to take care of for folks.
191 00:23:40.750 ⇒ 00:23:47.970 Kyle Wandel: And then staging’s almost like the comp… the combining of those intermediate data sources in order to get to another production.
192 00:23:47.970 ⇒ 00:23:52.170 Uttam Kumaran: Yeah, Shweeney, how would you… Yeah, do you want to explain, sort of, the layers?
193 00:23:52.170 ⇒ 00:23:58.350 Ashwini Sharma: So, staging is just, where you clean your data, you do aliasing, and,
194 00:23:58.660 ⇒ 00:24:01.969 Ashwini Sharma: You know, data type changes, things like that, right?
195 00:24:01.970 ⇒ 00:24:04.910 Uttam Kumaran: Roth, to staging to intermediate, to production.
196 00:24:04.910 ⇒ 00:24:05.640 Ashwini Sharma: Right.
197 00:24:06.140 ⇒ 00:24:14.690 Ashwini Sharma: And then intermediate is where you use that clean data, or, you know, where you have changed the data types as needed, and then…
198 00:24:14.840 ⇒ 00:24:18.149 Ashwini Sharma: Do joins between multiple other tables to create.
199 00:24:18.320 ⇒ 00:24:25.000 Ashwini Sharma: additional columns, or, you know, as Utam pointed out, case wins, you know, use rolling functions, and all those things.
200 00:24:25.470 ⇒ 00:24:32.790 Ashwini Sharma: And then broad, the MART layer is where you create your facts and dimensions and reports.
201 00:24:33.820 ⇒ 00:24:42.060 Ashwini Sharma: And in the mod layer itself, you can create… you can join between fats and dimensions to create these big fat tables.
202 00:24:42.370 ⇒ 00:24:49.550 Ashwini Sharma: which can be exposed to the BI, and then all the roll-ups and aggregations and filtering can be done on the BI side.
203 00:24:50.580 ⇒ 00:25:00.269 Uttam Kumaran: Yeah, so the big thing to kind of understand is that we’re supporting multiple customers with, like, one work unit, basically, here. So that’s the goal, is like, otherwise, if for every single
204 00:25:00.770 ⇒ 00:25:05.929 Uttam Kumaran: customer, we have, like, tables. It’s just, like, so much… it’s much too hard to maintain.
205 00:25:05.970 ⇒ 00:25:19.270 Uttam Kumaran: So we… we do the majority of the work for you to understand, like, find the gotchas, implement those, and then for some customers, hey, they may want to join further, or…
206 00:25:19.300 ⇒ 00:25:27.480 Uttam Kumaran: You know, and they’re fined, actually. You can just go do that in Snowflake. I would say, in most organizations, a majority of the customers are, like, in BI.
207 00:25:27.600 ⇒ 00:25:33.439 Uttam Kumaran: And so, they have no nuance of the underlying logic. They actually are just, like.
208 00:25:33.680 ⇒ 00:25:37.190 Uttam Kumaran: I’m assuming this is correct, and so that’s really, like.
209 00:25:37.480 ⇒ 00:25:48.080 Uttam Kumaran: how we think about serving both of those key partners, and then the last piece is, like, integrations. So sometimes we’ll be sending this data back into other systems, and so that’s typically another tenant.
210 00:25:49.840 ⇒ 00:25:57.230 Uttam Kumaran: And Ashwini, maybe we can send Kyle our, like, dbt, like, how we organize dbt.
211 00:25:57.500 ⇒ 00:25:57.980 Uttam Kumaran: Sure.
212 00:25:58.310 ⇒ 00:25:59.050 Ashwini Sharma: Yep.
213 00:25:59.830 ⇒ 00:26:04.230 Kyle Wandel: Yeah, I think that would be great, yeah. Learning, I mean, just learning about the different stages is really…
214 00:26:04.420 ⇒ 00:26:06.079 Kyle Wandel: helpful to understand how it’s…
215 00:26:06.090 ⇒ 00:26:07.530 Uttam Kumaran: It all plays out.
216 00:26:07.530 ⇒ 00:26:16.610 Kyle Wandel: So understanding how you guys create the dbt logic behind it to create those different stages would be great, that way I can get just kind of even more insight.
217 00:26:19.240 ⇒ 00:26:20.720 Kyle Wandel: And then…
218 00:26:20.720 ⇒ 00:26:28.619 Ashwini Sharma: For some of the commonly used reports or mods, you could do the aggregation directly in Snowflake, and then keep it over there.
219 00:26:28.750 ⇒ 00:26:34.639 Ashwini Sharma: Rather than doing it in BI, but that has to be a very common use case, right?
220 00:26:34.920 ⇒ 00:26:35.840 Kyle Wandel: Yeah.
221 00:26:38.040 ⇒ 00:26:42.349 Kyle Wandel: And that’s, like, the intermediate stuff, which is you… you basically are…
222 00:26:42.580 ⇒ 00:26:55.650 Kyle Wandel: having these intermediate reports, so it’s one pass staging, and then you’re creating these aggregations of standardized, kind of, counts, basically, and that’s, like, almost like the intermediate stage, or, like, the intermediate of this is the… so, like.
223 00:26:56.410 ⇒ 00:27:13.450 Kyle Wandel: this is the business logic to determine a member, yes or no. This is the… and then this is the list of members, or active members, currently. That would be, like, an intermediate stage, and then, like, the production stage is, like, the counts of active members per group, or the counts of when they joined, basically. Is that correct?
224 00:27:13.710 ⇒ 00:27:18.820 Ashwini Sharma: I would say, yeah, partially, I would say, because…
225 00:27:18.980 ⇒ 00:27:29.019 Ashwini Sharma: there may be a… so, intermediate stage should not be exposed to the BI. It’s only the smart layer that needs to be exposed. So, for example, if there’s a use case where
226 00:27:29.430 ⇒ 00:27:33.700 Ashwini Sharma: Somebody on the BI side needs to see a report of active members.
227 00:27:34.290 ⇒ 00:27:37.470 Ashwini Sharma: then you would be pulling the data from intermediate layer, right? So…
228 00:27:37.580 ⇒ 00:27:41.360 Ashwini Sharma: Having that report on the MART layer is going to be helpful.
229 00:27:41.470 ⇒ 00:27:49.290 Ashwini Sharma: But in addition to that report, you can also aggregate, based on that report, and then expose it as a different table on the Mart layer.
230 00:27:49.750 ⇒ 00:27:53.580 Kyle Wandel: Okay, I think that makes sense. So, like, an intermediate table, essentially.
231 00:27:53.580 ⇒ 00:28:17.969 Kyle Wandel: is usually, typically, either a table review, tabular data, and then it can either be a final list or report based on the need of the analyst, or you can take it one step further and create various aggregations off that intermediate step, and that’s, like, your prod report of count of active members, or count of total registrants per year, or whatever. Like, standardized counts. Okay, that makes… that makes a lot more sense.
232 00:28:18.200 ⇒ 00:28:29.760 Kyle Wandel: And then you can also use those production-based tabular reports to, like, use visualization software, like either Looker or Sigma, in order to do more fun aggregations.
233 00:28:31.010 ⇒ 00:28:41.020 Ashwini Sharma: Right, so maybe, one example I can highlight from my previous experience, right? Let’s say there is a… you want to, analyze the trend of,
234 00:28:41.540 ⇒ 00:28:44.029 Ashwini Sharma: You know, past years of data.
235 00:28:44.170 ⇒ 00:28:46.930 Ashwini Sharma: Right? And,
236 00:28:47.040 ⇒ 00:28:54.850 Ashwini Sharma: Maybe the breakdown could be, week on week, or maybe month on month, and year on year, something like that, right?
237 00:28:54.850 ⇒ 00:29:08.949 Ashwini Sharma: In that case, like, what the BI tool can do is, if you provide it with raw data, it can create those aggregations on the BI tool, right? But that would mean you are pulling a large volume of data from Snowflake into BI, and then doing that aggregation, right?
238 00:29:09.010 ⇒ 00:29:19.359 Ashwini Sharma: So, one thing you can do is, like, you can aggregate the data on the Snowflake itself, and then expose that on the BI. So, that would, you know, eliminate the need to
239 00:29:19.440 ⇒ 00:29:21.949 Ashwini Sharma: pull huge amounts of data into BI.
240 00:29:26.360 ⇒ 00:29:30.219 Ashwini Sharma: And just do aggregations on a smaller grain of data.
241 00:29:37.080 ⇒ 00:29:37.950 Kyle Wandel: Okay.
242 00:29:38.260 ⇒ 00:29:47.899 Kyle Wandel: That… I think that, like, helps me out a lot of understanding of, like, what to create and where to create it, basically. Because I know that
243 00:29:48.200 ⇒ 00:29:51.910 Kyle Wandel: We’ll start to be getting into that very quickly.
244 00:29:52.000 ⇒ 00:29:52.949 Uttam Kumaran: Yeah. In terms of…
245 00:29:52.950 ⇒ 00:30:04.049 Kyle Wandel: visualization softwares, do you guys have an idea? I know we… I don’t know if Catherine told you, but we did talk to SDG, and they gave us a couple of, like, suggestions, basically.
246 00:30:04.390 ⇒ 00:30:04.930 Kyle Wandel: But I don’.
247 00:30:04.930 ⇒ 00:30:06.539 Uttam Kumaran: Yeah, what did they suggest?
248 00:30:07.140 ⇒ 00:30:17.009 Kyle Wandel: That’s a good question, I gotta look at that right this second, but I… it wasn’t… they want us to stay with Power BI, but we basically just told them that’s not gonna work. And then I…
249 00:30:17.200 ⇒ 00:30:27.020 Kyle Wandel: I think it was… A combination of using… either creating the app for in-depth, customizable visualizations, Or…
250 00:30:28.020 ⇒ 00:30:31.420 Kyle Wandel: I think it was Sigma, is what they kind of recommended when it looked like.
251 00:30:31.420 ⇒ 00:30:41.150 Uttam Kumaran: Yeah, I talked to Catherine. I think one thing that we’re gonna try to do is actually enable some level of, like, natural language querying within Snowflake.
252 00:30:41.410 ⇒ 00:30:51.149 Uttam Kumaran: Because I think implementing a BI tool is gonna take a bit of time, even to just, like, select one. So I was talking to her, I was like, hey, maybe…
253 00:30:51.430 ⇒ 00:30:58.650 Uttam Kumaran: Like, there’s probably a percentage of queries that are actually just people needing, like…
254 00:30:58.800 ⇒ 00:31:07.940 Uttam Kumaran: just needing, like, look up something, or pull a quick export, and maybe we can satisfy a lot of those via, like, a Snowflake natural language thing, so we were actually gonna start looking into that.
255 00:31:08.210 ⇒ 00:31:09.520 Uttam Kumaran: Probably next week.
256 00:31:09.860 ⇒ 00:31:15.440 Uttam Kumaran: And then sometime in the next, like, 2-3 months.
257 00:31:16.070 ⇒ 00:31:18.150 Uttam Kumaran: attack, like, BI a bit more.
258 00:31:18.860 ⇒ 00:31:20.700 Uttam Kumaran: Yeah.
259 00:31:21.870 ⇒ 00:31:22.199 Kyle Wandel: Yeah, that makes sense.
260 00:31:22.200 ⇒ 00:31:26.190 Uttam Kumaran: It’s more than likely that a lot of people are just using it for exports and…
261 00:31:26.300 ⇒ 00:31:29.839 Uttam Kumaran: We want to just solve their, like, operational problems, you know?
262 00:31:30.580 ⇒ 00:31:39.009 Kyle Wandel: Yeah, I think even now, like, even right after CES, the biggest thing is, like, a list of exports, like, yeah, export lists, basically.
263 00:31:40.220 ⇒ 00:31:43.000 Kyle Wandel: I don’t have to see.
264 00:31:56.620 ⇒ 00:31:57.590 Kyle Wandel: Bottom there.
265 00:32:26.080 ⇒ 00:32:29.450 Kyle Wandel: Oh, Streamlit. Okay, that was the, UI.
266 00:32:29.650 ⇒ 00:32:31.580 Kyle Wandel: Okay. That was the app.
267 00:32:31.580 ⇒ 00:32:35.120 Uttam Kumaran: Yeah, so we’ll think of something we can do and streamlit with, like, chat.
268 00:32:35.230 ⇒ 00:32:36.360 Uttam Kumaran: shorter term.
269 00:32:36.900 ⇒ 00:32:39.010 Kyle Wandel: Yeah. I think. Yeah, I think that’s good.
270 00:32:40.090 ⇒ 00:32:43.459 Kyle Wandel: Yeah, I think it was Sigma and Streamlit were the two ones that…
271 00:32:43.600 ⇒ 00:32:45.989 Kyle Wandel: They recommended, based off our stuff.
272 00:32:47.110 ⇒ 00:32:48.999 Ashwini Sharma: Wait a second, Streamlight is a BI tool?
273 00:32:50.380 ⇒ 00:32:55.299 Uttam Kumaran: You can do many Python apps. Oh, okay. So, but for example, we would do, like.
274 00:32:55.460 ⇒ 00:33:02.419 Uttam Kumaran: for this idea, I was like, hey, why don’t we just generate something where people can log in and just, like, ask questions over, like, a specific set of data?
275 00:33:02.630 ⇒ 00:33:09.890 Uttam Kumaran: And run basic exports and things like that, and but we’re just, like, planning it out. I was just sort of brainstorming.
276 00:33:14.860 ⇒ 00:33:20.689 Kyle Wandel: Well, I think, I mean, that answers a lot of my… I have a couple more things that I’m working on right now, but,
277 00:33:23.000 ⇒ 00:33:26.119 Kyle Wandel: I don’t want to keep you guys too late, so…
278 00:33:27.780 ⇒ 00:33:44.860 Uttam Kumaran: Cool, yeah, we can adjourn, and then, yeah, I think, Ashwini, on our side, we’re continuing just to start to load a bunch of data, and then Awash, is also starting to help, like, speed up a lot of our ingestion. And then, really, like, that’s something I think, you know, I know CTI has, like.
279 00:33:45.030 ⇒ 00:33:51.970 Uttam Kumaran: 30, 40 different sources, so as long as we get to the… for the P0 ones, we’ll start to move into, like, significant modeling.
280 00:33:52.600 ⇒ 00:33:58.949 Uttam Kumaran: So, yeah, I think, like, probably this next month is gonna be really, really focused on that.
281 00:33:59.410 ⇒ 00:34:15.300 Kyle Wandel: Yeah, that’s what I’m doing, too, is ingesting all of the raw ones that I know of that are lately flat flyers, so Ashwini and I did the conference scans yesterday. And then I think the biggest thing that I want to do, and I don’t know if you guys have added it, but have you guys added the registration data yet?
282 00:34:16.719 ⇒ 00:34:23.299 Ashwini Sharma: Yeah, I had added it in the previous schema. I’ll have to move it and also provide the script.
283 00:34:23.579 ⇒ 00:34:29.319 Ashwini Sharma: So… Yeah, if you want to utilize it temporarily, it’s there in one of the tables.
284 00:34:29.799 ⇒ 00:34:31.439 Ashwini Sharma: I can point it out to you.
285 00:34:31.880 ⇒ 00:34:41.340 Kyle Wandel: Okay. Yes, because eventually I think the next step will be to start to create dbt models off of that, and so there are various, like…
286 00:34:41.380 ⇒ 00:34:54.549 Kyle Wandel: various intermediate lists, or production lists, that people like to use for it. So… and then we have the SQL script mostly created in Postgres, we just need to transfer it over to the dbt language.
287 00:34:55.350 ⇒ 00:35:03.120 Kyle Wandel: But maybe that’s something that we can work on next week, in terms of just starting to hammer out a lot of those views that we have created off of them.
288 00:35:04.450 ⇒ 00:35:05.180 Uttam Kumaran: Okay.
289 00:35:06.090 ⇒ 00:35:08.669 Kyle Wandel: Okay, that’d be great. Alright, bye.
290 00:35:08.670 ⇒ 00:35:09.280 Uttam Kumaran: Okay.
291 00:35:09.280 ⇒ 00:35:15.080 Kyle Wandel: All right. If I have any questions, I’ll reach out, so… But I think you guys appreciate it a lot. Yep.
292 00:35:15.080 ⇒ 00:35:19.220 Uttam Kumaran: Yeah. Thank you, Kyle. Thank you. Thanks, Srini. Okay, talk to you soon. Bye.