Meeting Title: Brainforge x LMNT : Bi Weekly Checkin with Tech Team Date: 2026-01-07 Meeting participants: Awaish Kumar, Jason Wu, Andy Weist, Steve Sizer, Uttam Kumaran, Ashwini Sharma, Shivani Amar, Jeff Warren
WEBVTT
1 00:00:23.480 ⇒ 00:00:24.590 Uttam Kumaran: Hey, everyone!
2 00:00:24.990 ⇒ 00:00:26.190 Jason Wu: Hey, Utem.
3 00:00:29.190 ⇒ 00:00:30.310 Uttam Kumaran: Hey, how’s it going?
4 00:00:31.310 ⇒ 00:00:32.439 Jason Wu: Happy New Year!
5 00:00:32.750 ⇒ 00:00:33.649 Uttam Kumaran: Happy New Year!
6 00:00:35.080 ⇒ 00:00:36.520 Uttam Kumaran: How’s everything going?
7 00:00:37.890 ⇒ 00:00:41.290 Jason Wu: Pretty good. Back to running full speed in our sprints.
8 00:00:41.540 ⇒ 00:00:45.020 Uttam Kumaran: Good. At least no surprises, hopefully, over a break.
9 00:00:45.380 ⇒ 00:00:47.720 Jason Wu: No, not during the break.
10 00:00:48.210 ⇒ 00:00:49.410 Uttam Kumaran: Alright, good, good.
11 00:00:49.410 ⇒ 00:00:50.060 Jason Wu: Yeah.
12 00:00:50.840 ⇒ 00:00:51.750 Uttam Kumaran: Awesome.
13 00:00:52.020 ⇒ 00:00:54.509 Jason Wu: Hopefully, Bit of time as well.
14 00:00:54.510 ⇒ 00:00:57.769 Uttam Kumaran: Yeah, I feel like this was the first,
15 00:00:58.040 ⇒ 00:01:06.179 Uttam Kumaran: this is the first time we had, like, pretty good, like, company-wide break. It’s tough, because we support a lot of clients with different
16 00:01:06.320 ⇒ 00:01:16.750 Uttam Kumaran: schedules, some clients are working all the time, some clients are not, so we… this is the first year, I think we had pretty good coverage across everything, and yeah, it was… it was really nice.
17 00:01:17.960 ⇒ 00:01:18.820 Uttam Kumaran: So…
18 00:01:19.950 ⇒ 00:01:37.780 Uttam Kumaran: Cool, so today, you know, we could probably jump right into things. We have a couple of topics, but really the kind of impetus for this is I think as we’re starting to make a lot of, technical decisions, we want to make sure that, you know, the tech team is, you know, privy to
19 00:01:37.860 ⇒ 00:01:47.430 Uttam Kumaran: you know, all of those. Also, we’re starting to make… we have to make some architecture decisions that we want to make sure that you guys are sort of looped into.
20 00:01:47.790 ⇒ 00:02:00.420 Uttam Kumaran: Yeah, I think, you know, having this, every week, or bi-weekly is sort of gonna be good to kind of batch a lot of that. So, I can go ahead and share, and then, yeah, me and Awash will sort of, kind of be leading.
21 00:02:00.710 ⇒ 00:02:01.384 Uttam Kumaran: Oh…
22 00:02:04.870 ⇒ 00:02:14.310 Uttam Kumaran: Great. So today, I wanted to walk through, a little bit of our current, like, architecture, based on a lot of the decisions we made.
23 00:02:14.410 ⇒ 00:02:26.569 Uttam Kumaran: On the source side, on the ETL side, and then now kind of going into the data warehouse. I want to talk about the Spins API. SPINS is another data source, that I know the team is using that we just…
24 00:02:26.590 ⇒ 00:02:35.460 Uttam Kumaran: signed an agreement to then get, I think, API access to. So I’ll be walking through just, like, some questions we may or may not have the answers today.
25 00:02:35.540 ⇒ 00:02:41.599 Uttam Kumaran: We’re talking about the Emerson sort of data share, and, like, what the kind of ask from our team is gonna be.
26 00:02:41.720 ⇒ 00:02:46.650 Uttam Kumaran: I know Shivani brought up a topic around, like, the Shopify Insider Bundles.
27 00:02:46.750 ⇒ 00:02:50.880 Uttam Kumaran: how that’s gonna potentially talk from Shopify to NetSuite, and like…
28 00:02:50.920 ⇒ 00:03:01.449 Uttam Kumaran: I did want to have a conversation about, like, app integrations versus reporting, and, like, what we should have the reporting infrastructure handle, what we may need direct integrations for.
29 00:03:01.490 ⇒ 00:03:16.169 Uttam Kumaran: And then, we already started some modeling work, so I do want to have you guys, just get an insight into, what we shared before, which is like, okay, now that we’ve landed some data, how we’re actually driving towards,
30 00:03:16.320 ⇒ 00:03:24.970 Uttam Kumaran: data model. So let me go ahead and just walk through… The, sort of, current architecture…
31 00:03:25.580 ⇒ 00:03:26.989 Uttam Kumaran: Give me a sec…
32 00:03:30.090 ⇒ 00:03:31.170 Uttam Kumaran: Great.
33 00:03:40.210 ⇒ 00:03:40.930 Uttam Kumaran: Okay.
34 00:03:40.960 ⇒ 00:03:45.080 Uttam Kumaran: So, you know, I’m… I’m gonna… this is, like.
35 00:03:45.080 ⇒ 00:04:02.729 Uttam Kumaran: pretty similar to what was before, I’ll probably start saving snapshots of this so you can kind of see the progression. But a couple things that are new here. One is we made a decision on a data ingestion tool, so before this was just, you know, an icon. So, basically the green checkmarks are sort of, like, confirmed in place.
36 00:04:02.780 ⇒ 00:04:21.710 Uttam Kumaran: And then anything that’s, like, sort of a cog is, like, you know, sort of in progress, and I can explain what in progress is, depending on the phase. So, starting from left to right, we have, like, all of our core raw data sources. As we identify future sources, we will start adding them here. Of course, like, NetSuite
37 00:04:21.870 ⇒ 00:04:35.689 Uttam Kumaran: I know doesn’t exist, or doesn’t exist in its final form right now for us to kind of consume it, but it is something on our radar, so as we identify more of these sources, I’m sure next we’re going to start to add, some of the information
38 00:04:35.690 ⇒ 00:04:47.329 Uttam Kumaran: Needed for marketing, needed for wholesale, so we’ll add some of that there. So, right now, we’re working on Shopify, Amazon, Walmart, Recharge, and Emerson, and Spins.
39 00:04:47.460 ⇒ 00:04:52.390 Uttam Kumaran: So kind of like everything that we have access to from the side of the business.
40 00:04:52.650 ⇒ 00:04:59.170 Uttam Kumaran: So in terms of, like, what’s in progress, right now we have Shopify,
41 00:04:59.310 ⇒ 00:05:04.929 Uttam Kumaran: and recharge coming in through Polyatomic. The Amazon and Walmart connectors are being built right now.
42 00:05:05.150 ⇒ 00:05:08.969 Uttam Kumaran: Maybe, awaish, we can talk a little bit about, like, what’s…
43 00:05:09.170 ⇒ 00:05:20.100 Uttam Kumaran: Remaining there, but I believe everybody on the team is with… is… has access to the Polytomic UI, and has, like… have you guys logged in in the last two weeks? Maybe I can also…
44 00:05:20.920 ⇒ 00:05:24.800 Uttam Kumaran: potentially log in and just show you, like, what it looks like in the UI, if that’s helpful.
45 00:05:25.440 ⇒ 00:05:34.569 Andy Weist: Quick question, where is our 3PL stored in the grand scheme of things? Are we trying to integrate that data source
46 00:05:35.570 ⇒ 00:05:39.699 Andy Weist: In it, like, early on, or are we pushing that for later?
47 00:05:40.500 ⇒ 00:05:44.899 Uttam Kumaran: Yeah, we’re… we’re… it is part of the,
48 00:05:45.450 ⇒ 00:05:49.719 Uttam Kumaran: It’s… it is somewhat looped in that integration question, but…
49 00:05:50.090 ⇒ 00:05:58.930 Uttam Kumaran: yeah, we need to bring that in. Where to go is the other one that’s not represented here, actually. I missed putting that on, but, we haven’t talked yet about…
50 00:05:58.930 ⇒ 00:06:06.640 Andy Weist: Stored is where to go, so stored required where to go. You can ignore where to go. For all intents and purposes, as of next week, we’re only on stored.
51 00:06:07.090 ⇒ 00:06:20.160 Uttam Kumaran: Okay, okay, so we’ll… so for where to go, then, it may just be a historical backfill that we’ll need? Yep, yep. And then, so stored. But is… does Stored have all the 3PL data, or is that coming… will that come from…
52 00:06:20.750 ⇒ 00:06:25.649 Uttam Kumaran: the 3PL… Like, themselves, I guess, I don’t know what the… yeah.
53 00:06:26.170 ⇒ 00:06:30.350 Andy Weist: Going forward, stored will be our only 3PL for D2C.
54 00:06:30.350 ⇒ 00:06:30.930 Uttam Kumaran: Okay.
55 00:06:31.280 ⇒ 00:06:36.350 Andy Weist: They’ll be the source of truth for all Shopify-filled orders, as well as some others.
56 00:06:37.810 ⇒ 00:06:48.670 Andy Weist: like you said, where to go will have a historical backfill. We should… Steve, I don’t remember where we’re at with… they were gonna give us, literally, a database dump of all the Where2Go data, so…
57 00:06:48.670 ⇒ 00:06:49.050 Uttam Kumaran: Right.
58 00:06:49.490 ⇒ 00:06:53.480 Andy Weist: It may be very easy for us to push that in. We need to circle back on that with them.
59 00:06:53.660 ⇒ 00:07:01.779 Andy Weist: But as of next week, going forward, Stored will be our only fulfillment partner. They have an API we can integrate with, etc.
60 00:07:03.400 ⇒ 00:07:15.140 Uttam Kumaran: Yeah, if you… if you guys even want to toss it… if there was an email about this, or you want to toss us in there, feel free, like, we can push this. We will just copy and make sure we have a copy sitting in Snowflake.
61 00:07:17.260 ⇒ 00:07:17.620 Andy Weist: Okay.
62 00:07:17.620 ⇒ 00:07:21.070 Uttam Kumaran: Wall this, because what we’ll end up doing is stitching these together.
63 00:07:21.170 ⇒ 00:07:24.820 Uttam Kumaran: In the data model to sort of show you the historical data set.
64 00:07:25.640 ⇒ 00:07:26.280 Andy Weist: Okay.
65 00:07:27.950 ⇒ 00:07:31.859 Uttam Kumaran: Cool. So then stored is… is then something we’ll add to the list.
66 00:07:34.870 ⇒ 00:07:42.830 Uttam Kumaran: So yeah, I think, yeah, other than that, like, yeah, if you log into the Polyatomic UI, you’re gonna just see these connectors.
67 00:07:43.570 ⇒ 00:08:03.419 Uttam Kumaran: We can go deeper into that if that’s of interest, but otherwise, it’s… we’ve added the API key. When Awish goes through his sort of demo, you’ll see the representation of the data in Snowflake, as well as, like, how we did the naming conventions on the schemas, and things like that, so…
68 00:08:03.570 ⇒ 00:08:12.150 Uttam Kumaran: you’ll… you’ll kind of see those pieces. So right now, I’m sort of waiting to just hear from, you know, Awash, and team on…
69 00:08:12.430 ⇒ 00:08:27.299 Uttam Kumaran: when these are, like, totally done, that we verify we have all the information. I think there was one sort of access requirement remaining on Shopify, and then I think there was an API issue that Polyatomic is talking to ReCharge about.
70 00:08:27.850 ⇒ 00:08:32.399 Uttam Kumaran: But ultimately, these, I expect to kind of move to green.
71 00:08:32.830 ⇒ 00:08:37.040 Uttam Kumaran: Like, you know, any time now, so… Anything awaish on…
72 00:08:37.580 ⇒ 00:08:39.780 Uttam Kumaran: On any of these that we want to follow up on.
73 00:08:40.080 ⇒ 00:08:44.399 Awaish Kumar: Yeah, like, we… like, for Shopify, I think we got the…
74 00:08:44.690 ⇒ 00:08:49.900 Awaish Kumar: access, because now I’m seeing some rows coming in for, like, transactions data.
75 00:08:51.830 ⇒ 00:09:03.230 Awaish Kumar: like, for recharge, it’s still, like, kind of from the recharge side. There are some bugs the polyatomic is seeing, so there’s nothing, like, we need from tech right now on these two things.
76 00:09:03.480 ⇒ 00:09:11.629 Awaish Kumar: But, like, on the Walmart, I… we still have… don’t have, like, access to… Kind of the dashboard.
77 00:09:12.220 ⇒ 00:09:16.620 Awaish Kumar: And, secondly, Like, the Walmart eComp?
78 00:09:17.110 ⇒ 00:09:26.299 Awaish Kumar: And… Apart from that, we… Yeah, we are good, like,
79 00:09:26.670 ⇒ 00:09:29.369 Awaish Kumar: In terms of, like, the Pyatomic is…
80 00:09:29.580 ⇒ 00:09:33.900 Awaish Kumar: is working on Amazon and where to go connectors.
81 00:09:34.590 ⇒ 00:09:40.200 Awaish Kumar: And now that… I think you have replied for the spins… API, so we can… Yeah.
82 00:09:40.520 ⇒ 00:09:45.649 Awaish Kumar: start to think about implementing the SPINS pipeline to bring in the data.
83 00:09:46.610 ⇒ 00:09:50.050 Uttam Kumaran: So do we need anything remaining on… on Shopify?
84 00:09:51.030 ⇒ 00:09:52.210 Awaish Kumar: Nope.
85 00:09:52.470 ⇒ 00:09:55.100 Uttam Kumaran: Okay, so Shopify’s good, and then we do…
86 00:09:55.600 ⇒ 00:10:00.980 Uttam Kumaran: We do need access to the e-commerce… the Walmart e-commerce UI.
87 00:10:00.980 ⇒ 00:10:06.220 Awaish Kumar: e-commerce, we… we need UI access, but also in terms of,
88 00:10:06.320 ⇒ 00:10:10.319 Awaish Kumar: We are not clear yet, like, what Polytomic needs to pull in the data, right?
89 00:10:10.320 ⇒ 00:10:11.150 Uttam Kumaran: Okay, okay, okay.
90 00:10:11.150 ⇒ 00:10:12.359 Awaish Kumar: Yeah, we’re still figuring out.
91 00:10:12.760 ⇒ 00:10:13.460 Uttam Kumaran: Okay, okay.
92 00:10:13.460 ⇒ 00:10:19.189 Jason Wu: Is the Walmart stuff, is that the snowflake access that we gave you to the Walmart tables, or is that something else?
93 00:10:19.430 ⇒ 00:10:22.999 Uttam Kumaran: Two separate, so the Emerson stuff is all retail.
94 00:10:23.340 ⇒ 00:10:24.310 Jason Wu: Oh, okay.
95 00:10:24.740 ⇒ 00:10:29.309 Uttam Kumaran: their, Walmart e-com will be through, like, a portal, typically.
96 00:10:29.310 ⇒ 00:10:30.230 Jason Wu: Okay, got it.
97 00:10:30.320 ⇒ 00:10:32.950 Uttam Kumaran: Yeah. Who’s, who’s working with you on that one?
98 00:10:34.420 ⇒ 00:10:37.180 Uttam Kumaran: We got it, Carlos.
99 00:10:37.310 ⇒ 00:10:38.460 Awaish Kumar: Is the one. Okay.
100 00:10:38.460 ⇒ 00:10:38.970 Jason Wu: Okay.
101 00:10:38.970 ⇒ 00:10:40.080 Uttam Kumaran: Carlos? Okay.
102 00:10:42.860 ⇒ 00:10:43.960 Uttam Kumaran: Okay, great.
103 00:10:48.310 ⇒ 00:10:49.150 Uttam Kumaran: Great.
104 00:10:50.140 ⇒ 00:10:59.810 Uttam Kumaran: And then, yeah, spins, we actually are just working on a spins pipeline for another client. Ashwini is actually working on that. So we will,
105 00:11:00.110 ⇒ 00:11:05.860 Uttam Kumaran: Yeah, I think… I think I have a… I have a bit of a slide on, like, what we’ll be looking at there, and
106 00:11:06.280 ⇒ 00:11:13.980 Uttam Kumaran: Yeah, so… so we can talk about, that when we get to that topic, but maybe I just kind of continued through. So, we’re landing, we’re…
107 00:11:14.880 ⇒ 00:11:15.930 Uttam Kumaran: Oh, yeah, yeah, please.
108 00:11:15.950 ⇒ 00:11:35.710 Shivani Amar: Sorry, I was raising my hand, but I was like, I’ll just speak. Okay, my question on Shopify, like, and I know we chose Polytopic, but so maybe this is an annoying question, but, like, the speed to ingest the data, is it like, oh, FiveTran would have been… I don’t know how these things work. Is it, like, Fivetron would have…
109 00:11:35.710 ⇒ 00:11:37.980 Uttam Kumaran: It’s a Shopify rate, it’s a Shopify rate limit.
110 00:11:38.330 ⇒ 00:11:43.759 Shivani Amar: There’s a Shopify rate limit. Okay, thank you. I was just like, we chose a good ETL, right?
111 00:11:43.760 ⇒ 00:11:47.759 Uttam Kumaran: Yeah, so it’s all of these… all of these folks
112 00:11:48.030 ⇒ 00:11:52.319 Uttam Kumaran: have various rate limits. You guys sell a lot of stuff.
113 00:11:52.520 ⇒ 00:11:56.290 Uttam Kumaran: So it will take a while. We have customers that, like.
114 00:11:56.410 ⇒ 00:12:02.800 Uttam Kumaran: It could be a week. We have other folks that it’s… it’s actually a very similar time, like, kind of, like, we’ll take a few weeks.
115 00:12:02.900 ⇒ 00:12:03.970 Uttam Kumaran: Yeah.
116 00:12:04.240 ⇒ 00:12:13.959 Uttam Kumaran: The nice thing is, like, the shape of the schema is already there, so we are already continuing modeling, so it doesn’t necessarily restrict us to start modeling, so…
117 00:12:13.960 ⇒ 00:12:14.600 Shivani Amar: Nice.
118 00:12:15.090 ⇒ 00:12:15.660 Uttam Kumaran: Yeah.
119 00:12:15.660 ⇒ 00:12:19.129 Shivani Amar: That’s awesome. So it’s just, like, at this… is it just, like, the…
120 00:12:19.130 ⇒ 00:12:24.350 Uttam Kumaran: Think of, like, a bucket getting filled, but we know, we know the shape of the bucket. Yeah, yeah, it’s like getting.
121 00:12:24.350 ⇒ 00:12:26.820 Shivani Amar: It’s just pulling older data, it’s not necessarily pulling…
122 00:12:26.820 ⇒ 00:12:33.950 Uttam Kumaran: I actually don’t… I actually don’t know, Wish, what direction… They’re… they started in, but…
123 00:12:34.610 ⇒ 00:12:47.769 Uttam Kumaran: Either, like, I guess either way, yeah, we already have the shape, all of the columns, so we know that, so we’re already starting modeling. But like, yeah, each of these, like, their rate limits are not,
124 00:12:48.170 ⇒ 00:12:53.229 Uttam Kumaran: like, they… they just… these are just by default, like, you can’t really do much. Actually, Polytomic does have
125 00:12:53.390 ⇒ 00:12:58.240 Uttam Kumaran: Specific, like, because they move a lot of data from a lot of these folks, they…
126 00:12:58.520 ⇒ 00:13:01.350 Uttam Kumaran: We’ll always go and ask, like, hey, we’re moving, like.
127 00:13:02.000 ⇒ 00:13:09.290 Uttam Kumaran: the, like, a huge amount of rows, like, can you give us exclusive API access? But Shopify just doesn’t do that for anybody, so…
128 00:13:09.290 ⇒ 00:13:17.489 Shivani Amar: And I… I know you were… you have this on your agenda for later, but before… while we’re on the Shopify topic, can we just talk about the Insider Bundle piece?
129 00:13:17.490 ⇒ 00:13:18.900 Uttam Kumaran: Oh, yeah, yeah, sure, yeah, please.
130 00:13:18.900 ⇒ 00:13:36.330 Shivani Amar: So Steve, you might have some context here around what Jeff Warren is, like, looking into. I think he was pinging me, letting me know… I think he was meeting with you, Steve, yesterday. And so I don’t know if you have context or want to, like, just discuss that topic here.
131 00:13:37.740 ⇒ 00:13:48.320 Steve Sizer: Jeff was looking into trying to find or get some location data and inventory data for, basically, the distribution arm of
132 00:13:48.760 ⇒ 00:13:53.190 Steve Sizer: of Element, basically. So he’s in discussions currently.
133 00:13:53.370 ⇒ 00:13:58.490 Steve Sizer: With Encompass on trying to find a solution for that.
134 00:14:00.470 ⇒ 00:14:09.869 Steve Sizer: the API access isn’t available to us unless we do a certification, so I think they’re looking into hiring
135 00:14:10.010 ⇒ 00:14:14.569 Steve Sizer: Sort of a professional services team to take on that work.
136 00:14:16.080 ⇒ 00:14:23.500 Shivani Amar: And the question that they’re trying to answer is… is, like, around specific SKUs we sell and where.
137 00:14:24.990 ⇒ 00:14:25.710 Shivani Amar: We’re like, what.
138 00:14:25.710 ⇒ 00:14:34.239 Steve Sizer: I think it’s… yeah, I think they’re just trying to get access to better information with the, sort of, local distribution, so…
139 00:14:34.360 ⇒ 00:14:38.869 Steve Sizer: same Bozeman, and now it’s gonna be Austin, and… and sort of…
140 00:14:39.310 ⇒ 00:14:45.110 Steve Sizer: What inventory we’re selling in those smaller, like, shopping or retail settings?
141 00:14:46.900 ⇒ 00:14:49.290 Shivani Amar: And so, like, Utham, when you hear that, are you like…
142 00:14:49.290 ⇒ 00:14:51.500 Uttam Kumaran: I… this seems like in our world, kind of.
143 00:14:51.500 ⇒ 00:14:55.200 Shivani Amar: Yeah, like, that’s where, Steve, I was getting confused because he pinged me, and he was like.
144 00:14:55.200 ⇒ 00:15:02.690 Uttam Kumaran: So I heard the two things that you mentioned, Siobhan. There’s one thing about, like, moving data, splitting data up in Shopify, and then moving it to NetSuite.
145 00:15:02.990 ⇒ 00:15:03.450 Shivani Amar: Yeah.
146 00:15:03.450 ⇒ 00:15:07.480 Uttam Kumaran: And this is almost like… Still about insights and reporting.
147 00:15:08.280 ⇒ 00:15:09.330 Shivani Amar: Yeah.
148 00:15:11.000 ⇒ 00:15:17.659 Shivani Amar: Steve, do you have more context behind it? We can also just talk to Jeff directly with Brainforge, if that helps,
149 00:15:19.270 ⇒ 00:15:27.539 Shivani Amar: helps resolve it. Like, I can’t even see if he’s free. I just… I think I was like, why are we adding an app when, like, we’re gonna get all of this granular data and we can always.
150 00:15:27.540 ⇒ 00:15:35.740 Uttam Kumaran: Yeah, like, you mentioned, you mentioned Trackstar. We just talked about stored in the, in the, in the… in the 3PL, so we’ll get the 3PL data
151 00:15:35.920 ⇒ 00:15:37.460 Uttam Kumaran: from that.
152 00:15:37.460 ⇒ 00:15:46.409 Steve Sizer: Encompass is a separate system, and the distribution is almost a separate entity to element.
153 00:15:46.740 ⇒ 00:15:51.570 Steve Sizer: So that’s how they manage all their inventory, is through A separate system.
154 00:15:52.760 ⇒ 00:15:56.540 Uttam Kumaran: So, Encompass is… it’s like, yeah, it looks like some type of ERP.
155 00:15:57.680 ⇒ 00:15:58.620 Uttam Kumaran: the…
156 00:15:59.600 ⇒ 00:16:07.060 Uttam Kumaran: solution, yeah, I know there’s, like, a bunch of these. So, I mean, ultimately, if they’re trying to get data out of this, and this is gonna be helpful.
157 00:16:07.320 ⇒ 00:16:09.600 Uttam Kumaran: For the business in the future, we should just…
158 00:16:10.310 ⇒ 00:16:13.230 Uttam Kumaran: Try to run it through this system, if it’s…
159 00:16:14.240 ⇒ 00:16:18.629 Uttam Kumaran: Like, yeah, I mean, I guess if that’s all I’m hearing.
160 00:16:19.110 ⇒ 00:16:21.459 Uttam Kumaran: Yeah. I feel like it’s not…
161 00:16:21.870 ⇒ 00:16:24.519 Uttam Kumaran: It’s not at all dissimilar to what we’re doing here.
162 00:16:24.520 ⇒ 00:16:36.320 Shivani Amar: Yeah, so, I just pinged Jeff, and I’m just saying, if you want to hop into the Zoom, you’re welcome to, because I was like, this was just an open question that I was kind of like… I guess, like, my…
163 00:16:36.690 ⇒ 00:16:40.159 Shivani Amar: And Jason, feel free to weigh in on this. It’s like, what I…
164 00:16:40.240 ⇒ 00:16:58.089 Shivani Amar: I wasn’t sure if, like, people need more education across the business around, like, the future state of the thing. Like, when Jeff was pinging me, he was kind of saying something like, but when Shopify is linked to NetSuite. And I was like, maybe I need to be educating more… people more on, like, the vision that eventually everything’s connected to Snowflake, and
165 00:16:58.090 ⇒ 00:17:07.529 Shivani Amar: the data’s moving in and out of Snowflake relative to us connecting everything on the outside, and so that was, like, the takeaway for me when I was talking to Jeff.
166 00:17:08.900 ⇒ 00:17:16.429 Shivani Amar: Because I wasn’t quite sure if he’s just, like, looking for an app to make some other connection point easy, if we could just do that through this data stack.
167 00:17:17.020 ⇒ 00:17:18.779 Shivani Amar: Do you have any thoughts, Jason?
168 00:17:19.569 ⇒ 00:17:25.359 Jason Wu: I think it just… I think we just have to have a quick, kind of, like, level cycle. I have not been part of those conversations with Jeff.
169 00:17:25.710 ⇒ 00:17:26.109 Shivani Amar: Yeah.
170 00:17:26.119 ⇒ 00:17:30.799 Jason Wu: about what exactly I was looking for, but from what you just described, it sounds like
171 00:17:31.349 ⇒ 00:17:33.769 Jason Wu: We might be doing a little bit of overlap, so…
172 00:17:33.770 ⇒ 00:17:35.839 Shivani Amar: Yeah, it just felt, like, redundant, so…
173 00:17:36.210 ⇒ 00:17:39.789 Shivani Amar: Great. I’ve invited him, if he’s free, he can join.
174 00:17:40.010 ⇒ 00:17:40.580 Uttam Kumaran: Okay.
175 00:17:41.520 ⇒ 00:17:58.410 Uttam Kumaran: Cool. Yeah, so we can get to the… we can get to the bottom. Again, so typically there’s, like, there’s a couple of use cases, so sometimes people are just trying to get data out of a system to report on it. That’s all we’re doing here. Sometimes people are like, when an order comes into Shopify, I needed to talk to
176 00:17:58.450 ⇒ 00:18:07.580 Uttam Kumaran: to NetSuite, and do some logic in between, and that needs to happen right when the order comes in. That is not something that you should.
177 00:18:08.120 ⇒ 00:18:17.239 Shivani Amar: you should handle through your reporting system, because there are delays. If that’s what he’s looking for, that makes so much more sense to me. He’s like, I need it for, like, handling the order, almost.
178 00:18:17.240 ⇒ 00:18:17.950 Uttam Kumaran: Yes.
179 00:18:18.230 ⇒ 00:18:24.459 Shivani Amar: then it’s like, okay, then it shouldn’t be like, in 6 hours, your data warehouse gets delayed.
180 00:18:24.460 ⇒ 00:18:34.010 Uttam Kumaran: We will still get all of that information, because we’re gonna get NetSuite, and we’re gonna get everything in here, but it’s not for transactional or, like, operational
181 00:18:34.120 ⇒ 00:18:36.820 Uttam Kumaran: Like, the order fulfillment process, you know?
182 00:18:36.820 ⇒ 00:18:39.089 Shivani Amar: That makes… that’s a very helpful distinction.
183 00:18:39.660 ⇒ 00:18:47.639 Andy Weist: Our… so, I think I need some context from Jeff as well, though, but… My understanding was that…
184 00:18:48.020 ⇒ 00:18:53.859 Andy Weist: None of our fulfillment process will be reliant on NetSuite anyway.
185 00:18:54.440 ⇒ 00:19:01.019 Andy Weist: So, I don’t know that we should have the assumption that NetSuite needs to be more real-time than our BI.
186 00:19:01.570 ⇒ 00:19:09.789 Jason Wu: think so either. So that’s where I, like, I think it’s maybe a little bit of a game of telephone. I’m hoping Jeff just joins and explains what he’s thinking.
187 00:19:09.790 ⇒ 00:19:15.410 Andy Weist: Yeah, I think this is more context just for you, Atum, to make sure, like, you have the same understanding that I do.
188 00:19:15.410 ⇒ 00:19:16.140 Uttam Kumaran: Yeah, understood.
189 00:19:16.140 ⇒ 00:19:18.249 Andy Weist: That opinion could change.
190 00:19:18.450 ⇒ 00:19:24.570 Andy Weist: But, like, historically, we have talked about NetSuite not needing to be real-time.
191 00:19:25.050 ⇒ 00:19:30.179 Andy Weist: I think, like, I come from a place… an opinion of… having NetSuite
192 00:19:30.660 ⇒ 00:19:36.579 Andy Weist: following our Snowflake instance would be okay in the way we’re looking at things today.
193 00:19:36.860 ⇒ 00:19:46.509 Andy Weist: So let’s just keep that door open, because I think that may be a cleaner implementation of both of these projects, and I just want to make sure we do things as simply as possible.
194 00:19:46.920 ⇒ 00:19:55.950 Uttam Kumaran: Okay. Yeah, reason I’m saying is people use NetSuite for a whole host of things. So, NetSuite’s like a catch-all for, like, a thousand products, so…
195 00:19:56.100 ⇒ 00:20:11.679 Andy Weist: Like, we’re not, like, managing inventory or anything in there. Honestly, the first iteration of it is mostly going to be for accounting purposes. So, really, latency is not an issue with our NetSuite implementation, as I currently understand it.
196 00:20:12.930 ⇒ 00:20:13.450 Awaish Kumar: Okay.
197 00:20:13.450 ⇒ 00:20:20.309 Uttam Kumaran: So that’s something I think, Awash, like, in our, data source docs, let’s just put some of, like, the core…
198 00:20:20.590 ⇒ 00:20:27.860 Uttam Kumaran: like, use cases on these. That way, especially for NetSuite, where, like, there is a suite of tools, we indicate, like, the kind of the core reason.
199 00:20:28.170 ⇒ 00:20:29.460 Awaish Kumar: Okay, yeah, sure.
200 00:20:31.800 ⇒ 00:20:33.090 Uttam Kumaran: Cool.
201 00:20:33.680 ⇒ 00:20:34.320 Uttam Kumaran: Yeah.
202 00:20:34.620 ⇒ 00:20:40.200 Uttam Kumaran: And then, once things move kind of through polyatomic, they’re ending up in raw.
203 00:20:40.580 ⇒ 00:20:46.780 Uttam Kumaran: Raw is… again, these… so these are all, like, like, sort of…
204 00:20:47.010 ⇒ 00:21:00.210 Uttam Kumaran: digital abstractions of data. RAW is not, like, a system, it’s just a database. What this is mainly indicating is, like, how we’re logically separating different parts of our modeling process.
205 00:21:00.210 ⇒ 00:21:12.760 Uttam Kumaran: So all of the data gets landed. We then, like, clean it up and combine it. So for example, combining Shopify and Amazon, and then producing, like, one
206 00:21:12.790 ⇒ 00:21:21.050 Uttam Kumaran: you know, orders table, which happens in, you know, March. And so one thing, couple things that we’ve done here is, like, GitHub is set up.
207 00:21:21.150 ⇒ 00:21:31.970 Uttam Kumaran: dbt is set up, and, you know, Wish will be walking you through today, like, one of those end-to-end processes, that we did, I think, on some Shopify data.
208 00:21:32.050 ⇒ 00:21:41.219 Uttam Kumaran: And then, as we move to the right here, this is where this data kind of, what we usually call is, like, gets activated, right? So, gets sent
209 00:21:41.240 ⇒ 00:22:00.519 Uttam Kumaran: sent into a visualization tool is probably… it’s more, like, it’s visualized in a Viz tool, so this is a BI tool. It maybe gets used downstream by, you know, this is sort of new as of the last, you know, year, but, like, gets used by AI agents in some way, and then additionally, it may get sent to a CRM, or sent to
210 00:22:00.520 ⇒ 00:22:04.369 Uttam Kumaran: go-to-market tool. A common use case here is, like.
211 00:22:04.380 ⇒ 00:22:20.249 Uttam Kumaran: folks, want to leverage high LTV customers to create a high LTV Klaviyo campaign, and the Klaviyo folks need some of that model data from Snowflake, so we would move that into Klaviyo for them to build more
212 00:22:20.250 ⇒ 00:22:29.229 Uttam Kumaran: targeted campaigns with that rich data. That’s just one use case of why you would move data outside of Snowflake into a tool where, like.
213 00:22:29.270 ⇒ 00:22:33.109 Uttam Kumaran: You know, there’s a… There’s an operational use case for that data.
214 00:22:33.260 ⇒ 00:22:38.910 Uttam Kumaran: The reason you wouldn’t do that all in Klaviyo is because Klaviyo doesn’t have modeling capabilities.
215 00:22:39.070 ⇒ 00:22:55.010 Uttam Kumaran: To this degree. Like, combining a lot of data, doing really sophisticated, LTV or, you know, other sort of analysis is not possible in Klaviyo, so you send it to Snowflake, it processes things, and you move it back there for targeting.
216 00:22:56.720 ⇒ 00:23:11.009 Uttam Kumaran: So that’s, you know, kind of the state of the architecture. What we’ll see over the next, you know, while is, like, kind of more sources will move to green and kind of get checked off. Additionally, we’ll start building marts here.
217 00:23:11.180 ⇒ 00:23:18.880 Uttam Kumaran: I would say this document is not gonna be, like, the ultimate source of truth for what tables exist.
218 00:23:19.000 ⇒ 00:23:30.880 Uttam Kumaran: This will be kind of more visual. We’ll end up with, like, quite a lot of tables, but that spreadsheet, and there will be some other documentation that comes out of dbt that will become the source of truth of, like, what’s in the warehouse.
219 00:23:31.010 ⇒ 00:23:37.629 Uttam Kumaran: But we will maintain this diagram to the best, you know, of our ability, because this is, like, the kind of the snapshot of the process.
220 00:23:37.980 ⇒ 00:23:41.480 Uttam Kumaran: I saw Jeff is here. Do we want to switch back?
221 00:23:41.730 ⇒ 00:23:43.320 Uttam Kumaran: Topics, really quick.
222 00:23:44.790 ⇒ 00:23:56.719 Shivani Amar: We could also do a separate conversation, Jeff. I think… I think the thing that was coming up was, that I brought into the flow was, we were talking about how Shopify data syncs into our warehouse, and…
223 00:23:56.720 ⇒ 00:24:05.660 Shivani Amar: we were just trying to understand if the app that you’re exploring is needed, but I think we’re all lacking a little bit of context around the app, and so maybe we could do a separate call to understand it better.
224 00:24:05.840 ⇒ 00:24:08.440 Jeff Warren: I think Andy and Steve have…
225 00:24:08.770 ⇒ 00:24:18.779 Jeff Warren: all of the context for me, and much better knowledge and thought on how to execute this. Like, I feel like I can be pulled out of the loop on this, because they…
226 00:24:18.950 ⇒ 00:24:22.030 Jeff Warren: Got it from here and know where they’re gonna put it in the workflow.
227 00:24:23.770 ⇒ 00:24:24.410 Shivani Amar: Cool.
228 00:24:25.480 ⇒ 00:24:43.349 Andy Weist: just… let’s clarify. I think I heard a couple different things regarding what Jeff is looking at. One thing, you know, our conversation the other day was based on our product bundling, meaning, like, one insider, one SKU contains multiple products, and how we’re handling that in Shopify, and how it…
229 00:24:43.500 ⇒ 00:24:44.300 Andy Weist: like…
230 00:24:44.480 ⇒ 00:24:52.299 Andy Weist: translates to actual fulfilled items in an order in our… in our warehouse. That’s the app we’re talking about? Is that right?
231 00:24:52.700 ⇒ 00:25:00.289 Andy Weist: Jeff, I know that’s the app we talked about. Shivani, does that make sense as far as, like, what you brought up earlier?
232 00:25:01.770 ⇒ 00:25:09.530 Shivani Amar: Yeah, so Jeff had just pinged me saying that he was looking into an app that could help us disaggregate what an insider bundle is, and my confusion was, isn’t that.
233 00:25:09.530 ⇒ 00:25:09.850 Andy Weist: Okay.
234 00:25:09.850 ⇒ 00:25:29.110 Shivani Amar: data warehouse will do. Like, our data warehouse will, like, have that granular data. And so my question was, is it needed, like, immediately, or is it something that, like, our data team can just report out on, like, this is what the breakdown is, and then you don’t actually need another app layered into the system? So I think I just didn’t totally understand the use case.
235 00:25:31.060 ⇒ 00:25:32.360 Andy Weist: Got it. Okay.
236 00:25:32.790 ⇒ 00:25:39.949 Andy Weist: Yes, then Jeff, you’re right. I have enough context to make sure I push this forward in the right way.
237 00:25:40.070 ⇒ 00:25:42.070 Andy Weist: Okay, thanks for the clarification.
238 00:25:42.340 ⇒ 00:25:43.190 Shivani Amar: Okay, thank you, Jeff.
239 00:25:43.720 ⇒ 00:26:00.910 Jeff Warren: Yeah, Shivani, maybe just, like, to say it in layperson language, the issue is less the, disaggregating it into, like, the warehouse software, like, that’s already happening. Yeah. Like, they’re using, like, a middleware to do that. It’s that it’s making it into QuickBooks.
240 00:26:01.530 ⇒ 00:26:16.739 Jeff Warren: Like, the same way, that, like, there is a syncing across the process so that you can see, like, okay, this insider bundle was actually made up of these three items, here’s the cogs for them, and then, because it was sold as an insider bundle, you know.
241 00:26:16.900 ⇒ 00:26:28.529 Jeff Warren: Instead of it being 27, so there’s $3 of trade spend that is going against each one of these as an insider bundle trade spend line item.
242 00:26:29.060 ⇒ 00:26:29.920 Shivani Amar: Hmm.
243 00:26:31.540 ⇒ 00:26:32.240 Jeff Warren: Cool.
244 00:26:32.750 ⇒ 00:26:34.109 Jeff Warren: Cool. Sweet!
245 00:26:34.510 ⇒ 00:26:34.880 Shivani Amar: Thank you.
246 00:26:34.880 ⇒ 00:26:36.550 Jeff Warren: Okay, catch y’all later.
247 00:26:36.550 ⇒ 00:26:40.190 Shivani Amar: Base… Perfect.
248 00:26:40.190 ⇒ 00:26:40.750 Uttam Kumaran: Okay.
249 00:26:42.190 ⇒ 00:26:46.550 Uttam Kumaran: So let’s go back then to our deck.
250 00:27:06.740 ⇒ 00:27:07.245 Uttam Kumaran: A…
251 00:27:09.810 ⇒ 00:27:16.349 Uttam Kumaran: Cool. And then… so we just got access to… well, I just signed the thing for spins today.
252 00:27:16.490 ⇒ 00:27:21.100 Uttam Kumaran: But, Shivani, I don’t… I was just mentioning, we… we are doing…
253 00:27:21.270 ⇒ 00:27:26.340 Uttam Kumaran: Spins work for another client right now, too. So,
254 00:27:26.500 ⇒ 00:27:34.889 Uttam Kumaran: kind of, like, why we have some of the docs, and so we’re kind of familiar. I’m interested in these questions, like, are we already pulling any data?
255 00:27:35.290 ⇒ 00:27:45.469 Uttam Kumaran: And if not, like, what data? There’s a lot of different endpoints. If we don’t know, then we can also go explore and kind of come see what we find.
256 00:27:46.720 ⇒ 00:27:59.220 Shivani Amar: we, like, from my understanding, people are, like, looking at SPINS reports, and you guys weigh in if you know of any other data actually being pulled, but, like, people are looking at SPINS reports and, like, looking at SPINS data. Like, our distribution team
257 00:27:59.220 ⇒ 00:28:17.479 Shivani Amar: self-distribution team is, like, looking at geographic data, because that’s the places that… that’s the place that geographic data is most clear right now in our business, to try and figure out, like, okay, like, where would we want to do self-distribution? So people are going into Spins and kind of, like, looking at the SPINS dashboards. People are looking at, like.
258 00:28:17.480 ⇒ 00:28:24.500 Shivani Amar: velocity of sparkling relative to other brands of, like, okay, how are we performing relative to…
259 00:28:24.510 ⇒ 00:28:41.540 Shivani Amar: I think I’m gonna have sent you one of the Spins reports, right? Like, people are looking, like, in the ready-to-drink section, like, how are we performing relative to Red Bull and whatever else? So I think that stuff is, like, kind of like these, like, exports or reports versus, like, people, like, cutting the data or anything like that.
260 00:28:41.540 ⇒ 00:28:42.160 Uttam Kumaran: Okay.
261 00:28:42.610 ⇒ 00:28:50.390 Uttam Kumaran: Okay, so I think, Ashwini, maybe what we can do is we can just see what endpoints we end up with access to.
262 00:28:50.660 ⇒ 00:28:58.920 Uttam Kumaran: And in this situation, we’ll just profile and kind of give an output of, like, what we… what we find, and then we can…
263 00:28:59.030 ⇒ 00:29:03.299 Uttam Kumaran: You know, end up… We could decide on if we want to just take everything we find, or…
264 00:29:03.630 ⇒ 00:29:05.379 Uttam Kumaran: Limit it, you know?
265 00:29:06.280 ⇒ 00:29:07.380 Ashwini Sharma: Sure, yeah.
266 00:29:07.580 ⇒ 00:29:09.500 Ashwini Sharma: I think maybe, if…
267 00:29:09.710 ⇒ 00:29:15.229 Ashwini Sharma: If you could find out what’s the smallest granular data that you’re looking at in Spence report?
268 00:29:15.720 ⇒ 00:29:20.029 Ashwini Sharma: Is it a week worth of data at the lowest level, or is it…
269 00:29:20.670 ⇒ 00:29:26.820 Ashwini Sharma: I mean, you never look at a week’s worth of data, and you just look at a month, or maybe four weeks, or a quarter.
270 00:29:28.080 ⇒ 00:29:32.010 Shivani Amar: Jason, do you know anything there about how it’s being used today?
271 00:29:32.610 ⇒ 00:29:37.310 Jason Wu: I don’t… this’ll all be… Will and Russell.
272 00:29:37.310 ⇒ 00:29:39.459 Shivani Amar: Yeah. Like, honestly, like…
273 00:29:39.460 ⇒ 00:29:43.780 Jason Wu: Like, is the question… maybe the question behind the question, like, we don’t want to pull, like.
274 00:29:43.780 ⇒ 00:29:47.900 Shivani Amar: unnecessarily large volumes of data if we don’t need to.
275 00:29:47.900 ⇒ 00:29:49.209 Uttam Kumaran: Well, there’s just, there’s just, like.
276 00:29:49.210 ⇒ 00:29:49.790 Ashwini Sharma: But…
277 00:29:49.790 ⇒ 00:29:54.540 Uttam Kumaran: Yeah, that’s exactly it, and there’s just kind of a bunch of different APIs.
278 00:29:54.840 ⇒ 00:29:56.160 Shivani Amar: Yeah. So…
279 00:29:56.160 ⇒ 00:29:57.759 Uttam Kumaran: Yeah, it’s kind of like…
280 00:29:58.120 ⇒ 00:30:04.499 Uttam Kumaran: I mean, by default, we can just take everything, but it may be, like, may not end up useful.
281 00:30:04.820 ⇒ 00:30:11.829 Shivani Amar: So, okay, so let’s say, like, let’s think about the business question for a second, right? Like, there’s a business question around, like.
282 00:30:12.500 ⇒ 00:30:13.150 Shivani Amar: Like, let me…
283 00:30:13.150 ⇒ 00:30:20.049 Uttam Kumaran: Even just… I’ll bring up the endpoint, even just, like, so we can talk specifics. Like, this is, like, the docs that we have.
284 00:30:20.560 ⇒ 00:30:26.719 Uttam Kumaran: we will get, like, access to this, which is, like, store insights, right, Ashwini? And we have to decide.
285 00:30:27.780 ⇒ 00:30:31.249 Uttam Kumaran: Like, of these, like, 500 things, like, what do we…
286 00:30:31.250 ⇒ 00:30:32.069 Shivani Amar: Gosh, yeah.
287 00:30:32.070 ⇒ 00:30:32.950 Uttam Kumaran: What do we want?
288 00:30:32.950 ⇒ 00:30:33.770 Ashwini Sharma: 60.
289 00:30:34.190 ⇒ 00:30:36.100 Uttam Kumaran: 460, okay, yeah, so… 460.
290 00:30:36.100 ⇒ 00:30:39.439 Ashwini Sharma: 465, 465 for the other client that I’m working on, yeah.
291 00:30:39.940 ⇒ 00:30:40.660 Uttam Kumaran: So…
292 00:30:40.660 ⇒ 00:30:41.260 Shivani Amar: So…
293 00:30:41.770 ⇒ 00:30:42.550 Uttam Kumaran: Yeah.
294 00:30:42.890 ⇒ 00:31:01.290 Shivani Amar: Okay, so let me start with the business question again, because this is a little overwhelming to look at, right? But it’s like, if I’m trying to think about how my… my question is, if I’m trying to think about how my product’s velocity is, I’m already getting some version of that from Emerson, no? Like, am I getting… I’m getting point-of-sale data from Emerson?
295 00:31:01.460 ⇒ 00:31:02.060 Uttam Kumaran: Yeah.
296 00:31:02.520 ⇒ 00:31:08.209 Shivani Amar: So I’m getting a sense of my velocity from Emerson, But then, like…
297 00:31:09.360 ⇒ 00:31:11.220 Uttam Kumaran: Yeah, you’re not getting benchmarks.
298 00:31:11.540 ⇒ 00:31:22.760 Shivani Amar: Yeah, so I’m not getting benchmarks from Emerson, which is what the point is of spins, but am I getting, from Emerson, am I getting… I’m getting velocity, and… sorry, the…
299 00:31:23.250 ⇒ 00:31:25.029 Shivani Amar: like, you just said something, and then I got thrown.
300 00:31:25.030 ⇒ 00:31:30.590 Uttam Kumaran: You’re getting velocity sales, and you’re getting, by… by market, right? Like.
301 00:31:30.990 ⇒ 00:31:31.310 Shivani Amar: One time.
302 00:31:31.310 ⇒ 00:31:32.169 Uttam Kumaran: stores, how many…
303 00:31:32.170 ⇒ 00:31:34.679 Shivani Amar: I’m getting there, I just need to… I need to use one second.
304 00:31:34.680 ⇒ 00:31:35.100 Uttam Kumaran: Oh, okay.
305 00:31:35.100 ⇒ 00:31:41.910 Shivani Amar: Like, you’re getting point of sales, right, from spins. That’s what I’m trying to understand. Are you getting point of sales from both?
306 00:31:44.710 ⇒ 00:31:49.460 Uttam Kumaran: That’s what I… that’s what I… I will have to go in and… we will have to go in and see what… what…
307 00:31:49.860 ⇒ 00:31:55.029 Uttam Kumaran: element is using spins for. I’m not sure, but I don’t know, Ashwini, since you’re already… you just looked at it today.
308 00:31:55.030 ⇒ 00:32:05.019 Ashwini Sharma: Yeah, so it’s more like a report, the spin state, right? There’s no single grain, where you can track the order, right?
309 00:32:05.500 ⇒ 00:32:10.319 Ashwini Sharma: And instead of that, what you see is a week worth of data, right? For example, like.
310 00:32:10.450 ⇒ 00:32:22.250 Ashwini Sharma: what was the number of units sold for this particular product, brand, in the last one week, right? That’s at the lowest grain of data that you can see.
311 00:32:24.800 ⇒ 00:32:33.519 Ashwini Sharma: Yeah, and that’s where I came to this question, right? Do you see it at that level, or is it…
312 00:32:33.750 ⇒ 00:32:40.860 Ashwini Sharma: more like, you know, I don’t see a daily report of how things are changing, I just see weekly data, right?
313 00:32:41.520 ⇒ 00:32:51.579 Ashwini Sharma: That’s the least that we can go. Otherwise, it is a month, quarter, 52 weeks, that kind of… But, Ashwini, are you able to pull.
314 00:32:51.580 ⇒ 00:32:55.350 Uttam Kumaran: Do you… you’re able to… you have to pick the dimensions, though, right, that you need?
315 00:32:55.650 ⇒ 00:33:07.020 Ashwini Sharma: Yeah, those attributes are the dimensions, right? If you see there, there’s attributes and measures, and if you look at the measures, that’s the quantity that you’ll be seeing on the report.
316 00:33:07.350 ⇒ 00:33:10.250 Ashwini Sharma: And those measures are aggregated to
317 00:33:10.570 ⇒ 00:33:15.889 Ashwini Sharma: 1 week, or 2 weeks, or 4 weeks, or more, right? Up to 52 weeks.
318 00:33:16.360 ⇒ 00:33:29.240 Shivani Amar: So tomorrow you’re doing a discovery call with retail, okay? And so, tomorrow you’re doing a discovery call with retail, and we can, like, actually ask some questions to him around, like, how often are you looking at Spins data, and what are you using it for? Okay.
319 00:33:29.240 ⇒ 00:33:51.549 Shivani Amar: My sense is that if Emerson is giving me point-of-sales data, and, like, telling me, like, how much transactions are happening regularly in a Target and in the store and everything else, that Spins is then more for benchmarking, and if Spins is then more for benchmarking, then I don’t understand… I feel like we could do the benchmarking on a monthly basis, as opposed to, like, a daily, weekly transaction thing, if that helps.
320 00:33:51.550 ⇒ 00:33:52.310 Shivani Amar: limit our, like.
321 00:33:52.310 ⇒ 00:33:52.750 Uttam Kumaran: Yes.
322 00:33:53.070 ⇒ 00:34:05.419 Shivani Amar: unnecessary volume of data. So that’s my intuition, is that you can go monthly with spins if the point of sales is in Emerson, so let’s just, like, clarify that, and then we can double-check that hypothesis with Russell tomorrow.
323 00:34:05.420 ⇒ 00:34:09.440 Awaish Kumar: Your data is available in Emerson for Walmart and Target.
324 00:34:10.150 ⇒ 00:34:10.810 Shivani Amar: Great.
325 00:34:12.679 ⇒ 00:34:19.319 Uttam Kumaran: Okay, so let me just note that down. So, tomorrow we’ll ask about Emerson for retail and spins for Benchmark.
326 00:34:19.829 ⇒ 00:34:21.129 Uttam Kumaran: Whoa.
327 00:34:29.170 ⇒ 00:34:32.480 Ashwini Sharma: And do we have the client ID and client secret to explore the.
328 00:34:32.480 ⇒ 00:34:36.010 Uttam Kumaran: Yeah, we don’t have anything yet, yeah. Just signed a thing today.
329 00:34:36.010 ⇒ 00:34:36.600 Awaish Kumar: Yeah.
330 00:34:36.860 ⇒ 00:34:41.659 Awaish Kumar: But, like… We saw in retail, there were other…
331 00:34:43.770 ⇒ 00:34:47.310 Awaish Kumar: stores, other than the Walmart and Target, like…
332 00:34:47.880 ⇒ 00:34:50.569 Awaish Kumar: So, that data is not coming from Immersum.
333 00:34:51.710 ⇒ 00:34:55.689 Uttam Kumaran: Hmm, so maybe if Spins has non-Walmart target.
334 00:34:56.219 ⇒ 00:34:58.589 Shivani Amar: POS. Oh, okay.
335 00:34:58.589 ⇒ 00:35:00.959 Jason Wu: This is all the industry data, right?
336 00:35:01.420 ⇒ 00:35:08.419 Shivani Amar: Yeah, but this must have our point of… so if we’re not getting… like, how are we getting point of sales for Vitamin Shop?
337 00:35:10.100 ⇒ 00:35:12.079 Shivani Amar: Probably in spins, Jason?
338 00:35:12.080 ⇒ 00:35:16.790 Jason Wu: We’re like, how do we… No, I think that’s actually a separate report that they just sent us.
339 00:35:17.500 ⇒ 00:35:20.960 Steve Sizer: Yeah, I think they send a… yeah, I think they send a report to Jessica.
340 00:35:21.240 ⇒ 00:35:21.840 Jason Wu: Yeah.
341 00:35:22.530 ⇒ 00:35:27.870 Shivani Amar: Oh, okay, so then, Utham, when you do the discovery call for retail tomorrow, let’s just know.
342 00:35:27.870 ⇒ 00:35:28.279 Uttam Kumaran: What other.
343 00:35:28.280 ⇒ 00:35:29.770 Shivani Amar: There are other retailers.
344 00:35:29.940 ⇒ 00:35:31.309 Shivani Amar: That we need to get.
345 00:35:31.310 ⇒ 00:35:45.260 Jason Wu: We need to get that stuff into our warehouse. Yeah, I mean, my understanding of spins is that it’s entirely just benchmark, right? Like, how’s, like, the industry performing? And, like, how are we relative to our competitors?
346 00:35:45.700 ⇒ 00:35:47.130 Shivani Amar: And, like, one thing…
347 00:35:47.540 ⇒ 00:36:00.680 Shivani Amar: one thing on spins that I was talking to Will about, like, on Monday was, pricing, okay? It’s like, especially in this world for a sparkling beverage right now, we’re still trying to figure out how we’re doing our pricing.
348 00:36:00.760 ⇒ 00:36:15.749 Shivani Amar: And so, like, so Phil kind of said, can’t you just get pricing from Spins? Why are we, like, we’re having people on our team go around and take pictures when they go to Costco, and take pictures when they go to Walmart, and take pictures when they go to HT, and then so…
349 00:36:15.750 ⇒ 00:36:28.699 Shivani Amar: Phil says, can’t I just get that from Spins? But Will, the head of re… revenue, or commercial, is kind of like, Spins is gonna give me an average, and I actually want to understand, like, how Costco prices us.
350 00:36:28.840 ⇒ 00:36:33.180 Shivani Amar: or… sorry, like, how Walmart prices us in, like.
351 00:36:33.300 ⇒ 00:36:49.199 Shivani Amar: Southside Chicago versus in Beverly Hills kind of thing, because they might have a price variance. So he’s like, I don’t love spins because it’s just giving me an average. So anyways, he still wants to deploy people to take these pictures and, like, do this thing.
352 00:36:49.200 ⇒ 00:36:59.950 Shivani Amar: So, like, pricing data is a thing that I think we would want to look at, but also take with a grain of salt as not, like, the source of truth on how we think about pricing our sparkling beverage.
353 00:36:59.950 ⇒ 00:37:02.710 Uttam Kumaran: Again, maybe that is a… Yeah, go ahead, go ahead.
354 00:37:02.910 ⇒ 00:37:19.639 Shivani Amar: Eventually, when we settle at a price that we’re all feeling good about for sparkling, like, maybe this’ll be, like, a done… like, I’m sure you benchmark over and over in life, but, like, but, like, the exercise right now is very much, like, are we pricing it right? Like, we’re trying to be a premium beverage.
355 00:37:19.740 ⇒ 00:37:31.789 Shivani Amar: But, like, are we pricing too high? Like, how are we feeling about our velocity? And all of that is, like, people looking at the spins data right now to say, okay, like, our velocity’s okay, but it’s also the winter, and…
356 00:37:32.260 ⇒ 00:37:44.109 Shivani Amar: overpricing relative to this, so we haven’t landed… we haven’t, like, totally… we don’t have a high confidence right now, I would say, on how we’re pricing and how our velocity is going with the sparkling beverages.
357 00:37:44.500 ⇒ 00:37:55.240 Uttam Kumaran: Okay, so what I’ll do is we’ll also validate, like, if… is that just because they’re limited about what they see through the UI? Like, does the API end up having more? And then we’ll put together, like, a one-pager on, like.
358 00:37:55.480 ⇒ 00:37:59.740 Uttam Kumaran: what does element have access to as part of spins from the… from the API?
359 00:38:00.170 ⇒ 00:38:15.620 Shivani Amar: And that seems like once you put that one-pager together, then we can pressure test it with Will and Phil, and send it out as a document, and they can actually weigh in. You can pick Russell’s brain tomorrow, but I think that’s spot on, to have, like, people weigh in on, like, I really want this data versus not.
360 00:38:16.080 ⇒ 00:38:17.570 Uttam Kumaran: Okay, okay, great.
361 00:38:17.570 ⇒ 00:38:18.220 Shivani Amar: Thank you.
362 00:38:18.220 ⇒ 00:38:21.900 Uttam Kumaran: Yeah, let me just share this again.
363 00:38:23.010 ⇒ 00:38:23.920 Uttam Kumaran: Perfect.
364 00:38:32.470 ⇒ 00:38:39.380 Uttam Kumaran: Cool, okay, so yeah, we have some questions for retail, and we have some, you know, I’ll add this to our agenda for tomorrow,
365 00:38:39.850 ⇒ 00:38:46.659 Uttam Kumaran: We’ll get understanding, and yes, this is actually a huge insight, so we’ll ask them all other sources of…
366 00:38:46.850 ⇒ 00:38:52.720 Uttam Kumaran: retail or retail distribution data that they’re getting, outside of Emerson and Spins.
367 00:38:52.880 ⇒ 00:38:56.550 Uttam Kumaran: We’ll try to get a list, and we’ll kind of see what the… kind of what the damage is.
368 00:38:56.900 ⇒ 00:39:04.010 Uttam Kumaran: The… basically, like, when I hear this, like, this is pretty classic. It’s, like, not, like,
369 00:39:04.490 ⇒ 00:39:07.690 Uttam Kumaran: not a surprise. The problem is, is, like.
370 00:39:07.870 ⇒ 00:39:13.710 Uttam Kumaran: Supporting the nth retailer and the nth retailer’s format is, like, the difficulty here.
371 00:39:13.860 ⇒ 00:39:16.220 Uttam Kumaran: So, most likely.
372 00:39:16.480 ⇒ 00:39:24.499 Uttam Kumaran: if, like, we get a list of, like, there’s, like, 30 other people that are sending CSVs and PDFs, we will prioritize by…
373 00:39:24.750 ⇒ 00:39:29.789 Uttam Kumaran: Revenue, or, like, immediate need for reporting, basically.
374 00:39:30.230 ⇒ 00:39:32.880 Uttam Kumaran: Like, if,
375 00:39:33.270 ⇒ 00:39:43.370 Uttam Kumaran: Joe’s Crab Shack is selling element, like, on the beach, and they’re selling, like, 10. We don’t… like, it may not be the highest priority, but a vitamin shop, I’m sure, is a big one, so…
376 00:39:43.830 ⇒ 00:39:44.510 Uttam Kumaran: Cool.
377 00:39:44.890 ⇒ 00:39:59.440 Uttam Kumaran: And then on the Emerson side, so, this is something, Jason, probably a question for you. So we… we have Emerson’s, Snowflake instance. We now have our own Snowflake instance. You can’t…
378 00:39:59.990 ⇒ 00:40:03.060 Uttam Kumaran: You can’t share someone else’s share.
379 00:40:04.010 ⇒ 00:40:09.640 Uttam Kumaran: So we just need them to share with our thing now.
380 00:40:09.810 ⇒ 00:40:16.229 Uttam Kumaran: And so we can provide you with all of the necessary… they just need our account ID and a couple things.
381 00:40:16.460 ⇒ 00:40:21.479 Uttam Kumaran: And we can probably draft that email. If you want to loop us in, we can talk to them.
382 00:40:21.730 ⇒ 00:40:28.800 Uttam Kumaran: I don’t think they’ll have any problem based on the last email they sent. They seem pretty sophisticated, so…
383 00:40:29.530 ⇒ 00:40:35.350 Jason Wu: Yeah, if you can give me the… the draft and the credential, I’ll… I’ll get the going.
384 00:40:35.790 ⇒ 00:40:36.330 Uttam Kumaran: Great.
385 00:40:37.460 ⇒ 00:40:39.350 Uttam Kumaran: Okay. Awaesh, anything else there?
386 00:40:40.230 ⇒ 00:40:41.920 Awaish Kumar: No, yeah, that’s it.
387 00:40:44.640 ⇒ 00:40:50.050 Uttam Kumaran: We already talked about this one, and then, yeah, maybe I wish I can hand it to you for…
388 00:40:50.230 ⇒ 00:40:51.869 Uttam Kumaran: By walking through dbt.
389 00:40:56.110 ⇒ 00:41:00.050 Awaish Kumar: Sure, I can… Share my screen.
390 00:41:09.470 ⇒ 00:41:19.410 Awaish Kumar: Okay, basically… Yeah, so I can just give an overview of Polytomic. This is how it looks like.
391 00:41:19.600 ⇒ 00:41:25.660 Awaish Kumar: And we have connections to Recharge, Shopify, and snowflake Warehouse.
392 00:41:25.970 ⇒ 00:41:36.099 Awaish Kumar: And these are the bulk syncs, which basically are syncing our data from Shopify to Snowflake. And in the history, you can see how it’s doing.
393 00:41:36.320 ⇒ 00:41:38.320 Awaish Kumar: So, the…
394 00:41:38.880 ⇒ 00:41:47.519 Awaish Kumar: And we can see here that it is taking some time for the initial sync, and that’s the reason we don’t have order line data yet.
395 00:41:47.750 ⇒ 00:41:49.239 Awaish Kumar: And because.
396 00:41:49.240 ⇒ 00:41:57.630 Uttam Kumaran: Maybe I can pause here, Wish? Yeah, if you scroll this, so see how it’s hitting around 2.26? Like, that’s a… that’s a…
397 00:41:57.940 ⇒ 00:42:00.079 Uttam Kumaran: That’s a Shopify cap.
398 00:42:00.350 ⇒ 00:42:04.979 Uttam Kumaran: Typically. So within, like, 10 hours or so, within every hour.
399 00:42:05.090 ⇒ 00:42:10.289 Awaish Kumar: Of course, it’ll be, like, something on the milliseconds, you can only hit the API so many times and get a certain payload.
400 00:42:10.410 ⇒ 00:42:16.529 Uttam Kumaran: So… Since we kicked it off, we’ve been landing data
401 00:42:17.030 ⇒ 00:42:32.490 Uttam Kumaran: basically consistently. 2 million, it doesn’t mean anything about… instead of, like, what you’re looking at, right? So, but when they… from the API side, they don’t really care much. Like, they see it as, like, amount of records.
402 00:42:32.530 ⇒ 00:42:37.990 Uttam Kumaran: For us, we are pulling specific tables, so if you want to click into one of those syncs, Awash.
403 00:42:38.430 ⇒ 00:42:40.700 Awaish Kumar: Yeah.
404 00:42:40.750 ⇒ 00:42:47.850 Uttam Kumaran: So to show the team exact… yeah, like, one of the… exactly. So this is kind of, like, what all the objects that we’re getting.
405 00:42:50.370 ⇒ 00:42:54.880 Awaish Kumar: Yes, and each, each run, what, table was basically…
406 00:42:55.180 ⇒ 00:43:00.920 Awaish Kumar: data for what Twitch table we pulled in. It just shows all of that in here, and
407 00:43:01.190 ⇒ 00:43:02.710 Awaish Kumar: We see that…
408 00:43:03.890 ⇒ 00:43:11.810 Awaish Kumar: The orders table basically has all the data now, but order line table, like, if we just look at this, this is still in progress.
409 00:43:11.950 ⇒ 00:43:16.880 Awaish Kumar: So, that’s what we are missing. Once it is done, like, we will have the…
410 00:43:17.670 ⇒ 00:43:20.999 Awaish Kumar: Next incremental runs, and it should be just fine.
411 00:43:22.130 ⇒ 00:43:37.689 Awaish Kumar: Yeah, now moving on to the dbt part. So in our BrainForge assessment repo, we have added a dbt project folder, which is basically… which basically contains the dbt project.
412 00:43:38.140 ⇒ 00:43:43.090 Awaish Kumar: And, like, this is structured in a normal dbt.
413 00:43:44.010 ⇒ 00:43:57.179 Awaish Kumar: standard way, how dbt structures its projects. So, in here, we have the models, which… where we normally work, and then there’s some… a folder called macros, which basically have…
414 00:43:57.290 ⇒ 00:44:08.210 Awaish Kumar: Like, here we play some… Sql, which is, like, standardizes our business context, or the… Some, like…
415 00:44:08.430 ⇒ 00:44:19.079 Awaish Kumar: Like, if we want to make some custom naming, or, like, so the piece of code which we want to reuse in our multiple models, that’s what we can put in here.
416 00:44:19.490 ⇒ 00:44:36.580 Awaish Kumar: And then if we go back to models, this is how we are structuring it. So we have raw, mods, intermediate, and mods. Raw is basically just the SQL queries, which are basically showing the structure of the table inside of the database, or data warehouse, which is Snowflake.
417 00:44:36.820 ⇒ 00:44:50.149 Awaish Kumar: Right now, I have, like, we have two sources, like Shopify and Recharge. If we go into the Shopify, basically, these are all the tables which are syncing for Shopify.
418 00:44:50.270 ⇒ 00:44:51.850 Awaish Kumar: And,
419 00:44:52.320 ⇒ 00:45:04.710 Awaish Kumar: And if we just… if I open any one of them, like, raw customers, so these are all the columns which are… we are getting, through Polyatomic, for all the Shopify customers.
420 00:45:07.200 ⇒ 00:45:15.030 Awaish Kumar: So, normally, at this point of time, like, in the raw, it’s just mainly one-on-one mapping with whatever is in the…
421 00:45:15.240 ⇒ 00:45:16.670 Awaish Kumar: the database.
422 00:45:16.870 ⇒ 00:45:25.810 Awaish Kumar: And then, we go to the intermediate, where we start to model stuff, and…
423 00:45:26.140 ⇒ 00:45:33.519 Awaish Kumar: do some transformation, or cleanups, and things like that. So I’ve started adding some models, like.
424 00:45:33.670 ⇒ 00:45:41.680 Awaish Kumar: deemed user and the orders, so it’s like, so here I’m trying to get… all the…
425 00:45:41.750 ⇒ 00:45:56.150 Awaish Kumar: fields which we can get from the Shopify customers table, and then on top of it, we can have some more fields from orders, like, for each customer, how much is the total revenue generated by this customer, how many…
426 00:45:56.190 ⇒ 00:46:03.190 Awaish Kumar: Wholesale orders, like the lifetime wholesale revenue for this specific customer, and lifetime
427 00:46:03.240 ⇒ 00:46:06.139 Awaish Kumar: Retail revenue, like, which is not the custom, no.
428 00:46:06.300 ⇒ 00:46:08.030 Awaish Kumar: Which is not the wholesale.
429 00:46:08.190 ⇒ 00:46:10.160 Awaish Kumar: And, for further discussion… Yeah.
430 00:46:10.160 ⇒ 00:46:14.579 Uttam Kumaran: If we could pause… pause here, so, like, like, even one step out of this, like.
431 00:46:14.780 ⇒ 00:46:23.500 Uttam Kumaran: most of our work here is gonna be writing, like, a ton of SQL like this. So this is, like, where we live as, like, analytics engineers.
432 00:46:23.830 ⇒ 00:46:37.379 Uttam Kumaran: Right? We’re just writing, you know, data models. As you see here, there are, like, of course, like, several types of transformations, like, we’re doing countless stinks, we’re doing sums, but we’re also doing logic, where, for example.
433 00:46:37.560 ⇒ 00:46:39.259 Uttam Kumaran: We have,
434 00:46:39.440 ⇒ 00:46:52.520 Uttam Kumaran: we, for example, have, like, sum of total price, which is lifetime total revenue. In addition, right, some business may request, hey, I actually want to see, in addition to total revenue, a column for wholesale revenue and retail revenue.
435 00:46:52.630 ⇒ 00:47:04.620 Uttam Kumaran: Right? So, like, we’ve… we forecast that that will be a question, so we go ahead and build, that… those columns. So you see here, we say, case when is wholesale? Is wholesale as a boolean?
436 00:47:04.680 ⇒ 00:47:17.619 Uttam Kumaran: sum anything where wholesale is true, sum the total price where wholesale is true, and put that into this bucket. And so these are all the types of transformations. I would say these are fairly simple transformations.
437 00:47:17.620 ⇒ 00:47:33.159 Uttam Kumaran: When we get into, what complex transformations, when we get into, like, subscription, churn, so we’ll be looking at net new subscriptions, net new expansion, reductions, churn, cancellation,
438 00:47:33.280 ⇒ 00:47:42.500 Uttam Kumaran: you know, resurrection, so there’s a lot of complex modeling kind of paradigms that we’ll be doing. But of course, a lot of, actually, the complication is stitching systems together.
439 00:47:42.550 ⇒ 00:47:57.779 Uttam Kumaran: So, this you’re seeing is just something for Shopify users. The Amazon customers table looks a lot different, has different columns. The source columns, although you may, like, see the name and say they’re very defined. Similarly, they’ll be defined differently, and so…
440 00:47:58.090 ⇒ 00:48:05.799 Uttam Kumaran: We have just worked with some of this data, so we kind of skipped past some of the figuring out, but in the past, yeah, you have to go read the docs and sort of understand.
441 00:48:05.930 ⇒ 00:48:11.549 Uttam Kumaran: How Shopify describes an order, how Amazon describes an order, and how we should
442 00:48:11.670 ⇒ 00:48:13.940 Uttam Kumaran: stitch those together, you know, in SQL.
443 00:48:15.080 ⇒ 00:48:23.820 Andy Weist: Do these run based on an event, or a time, like a cron schedule, or at query time, like…
444 00:48:24.100 ⇒ 00:48:28.669 Andy Weist: I guess the question is, what’s the latency of, like, something like this, which is, like…
445 00:48:28.830 ⇒ 00:48:29.590 Awaish Kumar: Wrong.
446 00:48:31.190 ⇒ 00:48:35.700 Awaish Kumar: Yeah, I can show you, like, for this to run, I have set up this CICD
447 00:48:35.840 ⇒ 00:48:42.480 Awaish Kumar: And… which is basically in the GitHub Actions. So, in the GitHub Actions, I… right now, I have added two different
448 00:48:43.830 ⇒ 00:48:53.179 Awaish Kumar: workflows. One is on the PR validation, which basically just validates whenever we push new things, it will just test it if it is working.
449 00:48:53.180 ⇒ 00:49:05.630 Awaish Kumar: And then the other one, and whenever we merge it, it will also run that, workflow. But the second one, which is running in the production, it’s basically, right now, it runs,
450 00:49:05.770 ⇒ 00:49:09.729 Awaish Kumar: on a schedule, like,
451 00:49:10.730 ⇒ 00:49:20.590 Awaish Kumar: on an hourly base, like, but we… it depends on how we want it to… want it to be. It can be moved to, like, we need to refresh all the data once a day, or…
452 00:49:20.740 ⇒ 00:49:23.870 Awaish Kumar: Twice a day or twice a day, and we can just set that up.
453 00:49:24.790 ⇒ 00:49:30.600 Andy Weist: Okay, so it’s scheduled, and the source of the scheduling is GitHub Actions.
454 00:49:30.970 ⇒ 00:49:31.850 Awaish Kumar: Yes.
455 00:49:32.470 ⇒ 00:49:33.429 Andy Weist: That makes sense. Cool.
456 00:49:33.430 ⇒ 00:49:46.790 Uttam Kumaran: And the ordering happens, you know, like, you run dbt run, it runs a compile steps, it figures out that this model references the last model, because it’s all Jinja templated, and then it runs it in order.
457 00:49:46.800 ⇒ 00:50:04.720 Uttam Kumaran: But running a query… running a dbt model, doesn’t necessarily mean it will get materialized in the… in the… in the warehouse. And so, materialization means, like, you materialize a view, or you materialize as a table. A view is a query that gets run on query time.
458 00:50:04.750 ⇒ 00:50:19.519 Uttam Kumaran: Versus a materialized table, you can think of it as a fixed table, that when a query hits, it runs on that. So, that is actually configured at the model level, so if you go back to that table, Oasia, you can just show the config at the top.
459 00:50:19.700 ⇒ 00:50:22.529 Uttam Kumaran: Where we, we sort of… you kind of put in the…
460 00:50:22.530 ⇒ 00:50:23.720 Awaish Kumar: Oh, the macro.
461 00:50:24.210 ⇒ 00:50:30.839 Uttam Kumaran: Yeah, or, you know, you could just show the, at the top, you put in the table config for the DIM users table.
462 00:50:39.570 ⇒ 00:50:47.519 Uttam Kumaran: Yep, so the materialization schedule is table, meaning it will get written as a table in Snowflake.
463 00:50:48.700 ⇒ 00:51:04.439 Uttam Kumaran: there’s… there’s… there’s just reasons for having views versus table. Sometimes, if tables are, directly similar to other tables, you may not want to just materialize the whole thing. Like, think about one table that’s 100 million rows, and another table that just…
464 00:51:04.460 ⇒ 00:51:18.100 Uttam Kumaran: changes one column from that, you may not want to materialize that again, because now you have 200 million rows. Another, common materialization strategy is ephemeral. This is, like, this is, basically…
465 00:51:18.210 ⇒ 00:51:36.219 Uttam Kumaran: This is not a view, this is actually just creating a dbt model, as a way to represent, like, almost like a CTE, which is a common table expression. If a dbt model gets, like, a thousand lines, it’s really hard to read, and so we commonly will split things up.
466 00:51:36.290 ⇒ 00:51:44.950 Uttam Kumaran: The last one is incremental, so again, we may end up with tables with hundreds of millions of rows. You don’t want to run that and
467 00:51:45.060 ⇒ 00:52:03.060 Uttam Kumaran: drop it, rerun that every time, so you instead just want to add the latest, like, add the latest orders, add the latest transactions. Those are incremental. So, there are different configuration types, that dbt, again, helps you manage that just through this sort of templating.
468 00:52:04.490 ⇒ 00:52:05.190 Andy Weist: Got it.
469 00:52:05.460 ⇒ 00:52:11.400 Awaish Kumar: Yeah, if I just show this in Snowflake, so this is how it is structured right now.
470 00:52:12.110 ⇒ 00:52:17.710 Awaish Kumar: for each folder, like, the raw is coming from this raw, so we are basically…
471 00:52:17.880 ⇒ 00:52:26.659 Awaish Kumar: the data from Polyatomic goes into this raw tape, raw database, and that’s, like, our queries and raw just reflect
472 00:52:26.930 ⇒ 00:52:37.010 Awaish Kumar: this structure, what is in there, and then apart from that, we have intermediate and then mods. So, like, and they are, like, if we… if you see, we have…
473 00:52:37.270 ⇒ 00:52:54.819 Awaish Kumar: two databases, like Prod, Intermediate, and ProdMarts, which basically show prod is the environment. So the first part of this name says the environment name, and the second part is just the folder name. So anything which you find in intermediate goes into the intermediate database.
474 00:52:55.000 ⇒ 00:53:00.040 Awaish Kumar: And all the models which are in Mars go… go create a table in this,
475 00:53:00.440 ⇒ 00:53:05.320 Awaish Kumar: basically, the MARTS database. In the MARTS, we have the customers.
476 00:53:05.680 ⇒ 00:53:10.219 Awaish Kumar: mart, and in the customer’s mart, we have a DIM users table.
477 00:53:10.630 ⇒ 00:53:21.310 Awaish Kumar: Similarly, for intermediate, we have Shopify. For Shopify, we created two different tables. So it’s exactly the same structure as you see in the…
478 00:53:21.540 ⇒ 00:53:23.040 Awaish Kumar: In the GitHub code.
479 00:53:23.160 ⇒ 00:53:37.639 Awaish Kumar: And then the first chunk shows the environment, which is, like, if it is running in production, you will see in the prod. If we are running it in our, like, local dev environment for testing, then it will end up in the dev databases.
480 00:53:37.920 ⇒ 00:53:44.920 Awaish Kumar: And our CICD… we have a CSID pipeline, which is, like, staging, and for that, it will end up in SDG.
481 00:53:45.140 ⇒ 00:53:46.150 Awaish Kumar: databases.
482 00:53:52.020 ⇒ 00:53:58.760 Awaish Kumar: Yeah, and the… all the other, like, the… Folders will become datasets.
483 00:53:58.980 ⇒ 00:54:03.799 Awaish Kumar: And inside of that folder, if there’s a model, it becomes a table in the snowflake.
484 00:54:06.210 ⇒ 00:54:14.189 Uttam Kumaran: So there’s this matching, of the repo structure to schemas and tables. But see, that is all just, like…
485 00:54:14.480 ⇒ 00:54:18.440 Uttam Kumaran: you don’t need… you don’t, have to do that to run dbt, but…
486 00:54:18.590 ⇒ 00:54:26.650 Uttam Kumaran: after doing dbt for a long time, like, it’s just very nice to kind of keep things organized like this. This is more ergonomics. So we match…
487 00:54:26.820 ⇒ 00:54:36.020 Uttam Kumaran: the first layer, folder structure to schemas, just so it’s easy to go from Snowflake to, the next, so…
488 00:54:37.120 ⇒ 00:54:39.149 Awaish Kumar: Yeah, just to add, like.
489 00:54:39.270 ⇒ 00:54:45.860 Awaish Kumar: Here, like, the kind of transformations we did, if, like.
490 00:54:46.040 ⇒ 00:54:54.550 Awaish Kumar: they are just for Shopify, so it looks, like, really straightforward right now. It becomes really complex when we have multiple sources.
491 00:54:54.570 ⇒ 00:55:12.410 Awaish Kumar: when the data from Amazon, and when we have data from Walmart, and then where to go, then all of that needs to be joined, and we don’t have just one model like GIM customer. Then we might generate, like, fact orders, fact order lines, and then we have some summary models, which
492 00:55:12.760 ⇒ 00:55:18.990 Awaish Kumar: Which are, like, we need an order summary, or we need a product summary, things like that, and…
493 00:55:19.330 ⇒ 00:55:24.259 Awaish Kumar: Yeah, that will just then… End up being more and more models in there.
494 00:55:26.380 ⇒ 00:55:29.089 Steve Sizer: With the… with that processing of that data.
495 00:55:29.200 ⇒ 00:55:33.509 Steve Sizer: you’re saying it runs on a schedule of, like, every hour. How does it know
496 00:55:34.060 ⇒ 00:55:36.760 Steve Sizer: What stuff it’s already processed, and where it…
497 00:55:37.030 ⇒ 00:55:41.610 Steve Sizer: needs to start again for the next process. How do you… how do you map them?
498 00:55:42.590 ⇒ 00:55:50.690 Awaish Kumar: So yeah, that is handled in dbt, as in, like, like, there are multiple different, like, kind of configurations.
499 00:55:50.940 ⇒ 00:55:53.859 Awaish Kumar: For each dbt model.
500 00:55:54.000 ⇒ 00:55:56.160 Awaish Kumar: Right? So, right now.
501 00:55:56.250 ⇒ 00:56:02.970 Awaish Kumar: Like, we have just a really simple model, I’m just saying it as a table, but then there are different materializations.
502 00:56:03.010 ⇒ 00:56:22.170 Awaish Kumar: So, one materialization is called incremental. If we use that materialization, what happens is that we use primary key of the table to figure out what is already being processed, and what now needs to process. So, we will have a primary key, and we will have, like, last updated timestamp.
503 00:56:22.190 ⇒ 00:56:40.869 Awaish Kumar: Using those two columns, dbt will figure out, okay, we already processed this data, and I need only… need to process last seven days of data, and in that also, like, we have hundreds of same customers, then it will just leave them and update the columns, like, it will use the
504 00:56:40.970 ⇒ 00:56:50.809 Awaish Kumar: Like, the absurd kind of… the strategy, so it will update the fields which were needed to be updated, or it will insert the new ones.
505 00:57:01.400 ⇒ 00:57:07.069 Awaish Kumar: So yeah, normally we will be using incremental strategy, because that’s,
506 00:57:07.380 ⇒ 00:57:11.329 Awaish Kumar: And that’s how we can optimize our processing time and the cost.
507 00:57:14.860 ⇒ 00:57:19.709 Uttam Kumaran: Yeah, and that’s just one of the common ways that, like, again, you’re gonna see that we’ll end up with, like, hundreds of models.
508 00:57:20.070 ⇒ 00:57:36.140 Uttam Kumaran: And to run all of those, to materialize all of those, and a lot of those will become big, this is a common way that dbt projects get bloated. So, off the start, we do incremental, we use ephemeral, and we just try to stay, like, organized.
509 00:57:36.260 ⇒ 00:57:40.240 Uttam Kumaran: Because the… the sprawl will get really, really big here.
510 00:57:44.840 ⇒ 00:57:45.670 Uttam Kumaran: Cool.
511 00:57:45.940 ⇒ 00:57:55.280 Uttam Kumaran: I know we’re at time, so I think as we start to do more modeling, more of our weekly textings, we’ll talk a little bit about the logic that we’re implementing.
512 00:57:56.680 ⇒ 00:58:11.329 Uttam Kumaran: you know, I think over the… we’ll never stop talking about, like, ingesting new sources, but ideally, we’ll try to push as much through Polyatomic as we can, but most of our conversation, I think, will talk about some of the modeling logic, and I think as we start to get into the
513 00:58:11.340 ⇒ 00:58:17.730 Uttam Kumaran: business intelligence layer and, like, AI and stuff, that… that’s where, kind of, these conversations will go as well, but…
514 00:58:18.260 ⇒ 00:58:21.329 Uttam Kumaran: Happy to answer any questions about,
515 00:58:21.580 ⇒ 00:58:24.580 Uttam Kumaran: you know, DPT, or… or…
516 00:58:24.880 ⇒ 00:58:30.250 Uttam Kumaran: You know, how to do modeling, and we hope that everybody can kind of get into the repo and start pushing stuff, too, so…
517 00:58:34.190 ⇒ 00:58:34.960 Uttam Kumaran: Great.
518 00:58:35.390 ⇒ 00:58:43.130 Uttam Kumaran: Okay, so Jason, I’ll follow up with the Emerson email. I think, Shivani, I’ll… I’m gonna have the… I’m gonna update the retail agenda with…
519 00:58:43.250 ⇒ 00:58:50.800 Uttam Kumaran: stuff for… for spins tomorrow. Perfect. And I can send that, and yeah, I…
520 00:58:51.430 ⇒ 00:58:54.220 Uttam Kumaran: I think that’s it. Let me know if there’s anything else.
521 00:58:55.260 ⇒ 00:59:00.890 Uttam Kumaran: And, well, we’re gonna update the… now that we’re starting to land data, we’re updating the Gantt chart for…
522 00:59:01.160 ⇒ 00:59:06.300 Uttam Kumaran: For all the… with all the actual data models that we’re building, like the core output data models.
523 00:59:06.610 ⇒ 00:59:11.249 Uttam Kumaran: So we’ll have that. I think we’ll have some of that to read tomorrow.
524 00:59:13.400 ⇒ 00:59:14.780 Shivani Amar: Sounds good, thank you.
525 00:59:16.790 ⇒ 00:59:17.530 Uttam Kumaran: Okay.
526 00:59:17.530 ⇒ 00:59:17.910 Jason Wu: Thank you.
527 00:59:17.910 ⇒ 00:59:18.580 Uttam Kumaran: Thanks, everyone.
528 00:59:19.590 ⇒ 00:59:20.340 Shivani Amar: Thank you.
529 00:59:20.340 ⇒ 00:59:21.440 Uttam Kumaran: Perfect. Talk soon.
530 00:59:21.440 ⇒ 00:59:22.000 Jason Wu: Nope.
531 00:59:22.200 ⇒ 00:59:22.760 Awaish Kumar: Right?