Meeting Title: Brainforge x LMNT : Bi Weekly Checkin with Tech Team Date: 2026-03-18 Meeting participants: Andy Weist, Uttam Kumaran, Awaish Kumar, Shivani Amar, Jason Wu, Amber Lin
WEBVTT
1 00:00:14.850 ⇒ 00:00:15.750 Uttam Kumaran: Handy.
2 00:00:35.180 ⇒ 00:00:36.480 Andy Weist: Adam, how are you?
3 00:00:36.950 ⇒ 00:00:38.100 Uttam Kumaran: Good, how are you?
4 00:00:38.100 ⇒ 00:00:42.699 Andy Weist: Sorry, I was trying to finish a quick lunch before making you look at that.
5 00:00:42.700 ⇒ 00:00:44.930 Uttam Kumaran: What’s for lunch? I’m always interested.
6 00:00:45.200 ⇒ 00:00:47.710 Andy Weist: Leftovers, fried rice type stuff.
7 00:00:47.920 ⇒ 00:00:49.910 Uttam Kumaran: Alright, that’s great.
8 00:00:50.260 ⇒ 00:00:50.940 Uttam Kumaran: iPhone 5.
9 00:00:50.940 ⇒ 00:00:51.440 Andy Weist: No, no.
10 00:00:51.440 ⇒ 00:00:54.530 Uttam Kumaran: I need to go do lunch after this, I think.
11 00:00:55.010 ⇒ 00:01:02.120 Uttam Kumaran: Some… what do I… yeah, I think some type of leftovers. I made this, like, sheet pan hash browns. We made some, like, the other day, so that’s…
12 00:01:02.120 ⇒ 00:01:02.860 Andy Weist: Yeah, we’re nice.
13 00:01:04.080 ⇒ 00:01:04.550 Awaish Kumar: Aye.
14 00:01:04.550 ⇒ 00:01:09.900 Andy Weist: Yeah, I just cook, like, 8 meals at once, and then pick away at whatever for the whole week. However you wish.
15 00:01:11.700 ⇒ 00:01:12.350 Awaish Kumar: Bye.
16 00:01:15.460 ⇒ 00:01:23.339 Uttam Kumaran: Yeah, lunch is always tough, I feel like I’m always a little bit in a scramble. But I really like to cook, so at dinner, I try to just block off time to…
17 00:01:23.900 ⇒ 00:01:26.929 Uttam Kumaran: To do that, and it’s a little bit relaxing, so…
18 00:01:27.350 ⇒ 00:01:28.489 Andy Weist: Yeah, same.
19 00:01:30.880 ⇒ 00:01:38.079 Andy Weist: Steve and Jason are in Bozeman. They’ll be joining, they’re there. I just figured they probably need a minute to find a space or something.
20 00:01:38.380 ⇒ 00:01:38.760 Jason Wu: Yep.
21 00:01:38.760 ⇒ 00:01:41.610 Uttam Kumaran: or… Hey guys.
22 00:01:42.610 ⇒ 00:01:46.670 Jason Wu: How’s it going? Hey, how’s it going? We’re all sat together, me, Jason, and Chevac.
23 00:01:46.670 ⇒ 00:01:48.220 Uttam Kumaran: Oh, great, okay.
24 00:01:48.390 ⇒ 00:01:51.370 Uttam Kumaran: Cool, cool, cool. I’m jealous.
25 00:01:51.610 ⇒ 00:01:52.889 Jason Wu: I’m like, I wanna…
26 00:01:52.890 ⇒ 00:01:56.710 Uttam Kumaran: be somewhere… oh, I guess it’s… it’s probably… Really cold there, but…
27 00:01:56.710 ⇒ 00:01:59.669 Jason Wu: It’s actually, like, 70 degrees right now.
28 00:01:59.670 ⇒ 00:02:04.290 Uttam Kumaran: Oh, okay, now, okay, then I’m 100% jealous, not just 60%.
29 00:02:04.530 ⇒ 00:02:14.520 Uttam Kumaran: Cool, I think today’s gonna be kind of like a lot of show and tell. I think we have significant parts of
30 00:02:14.620 ⇒ 00:02:16.790 Uttam Kumaran: The entire system.
31 00:02:16.870 ⇒ 00:02:30.459 Uttam Kumaran: churning all the way into our BI tool. So I kind of want to just recap, you know, since we’ve last presented, where each of those pieces are, and actually kind of show
32 00:02:30.460 ⇒ 00:02:48.270 Uttam Kumaran: the flow all the way into Omni, which is our pilot, that we’re running with. And then just… I… I suspect there’ll be, some questions, so I’ll just… it’ll be sort of engineering-focused about each part, today. So let me just go ahead and share, and let’s just jump into it. Please, like, feel free to…
33 00:02:49.100 ⇒ 00:02:55.970 Uttam Kumaran: Stop me at any point, during the presentation, and… Okay…
34 00:03:01.710 ⇒ 00:03:03.609 Uttam Kumaran: Great. One second…
35 00:03:03.610 ⇒ 00:03:10.569 Jason Wu: In terms of agenda, I just had an interesting conversation with Phil and Brianna, maybe even just, like, a few minutes after.
36 00:03:10.830 ⇒ 00:03:13.379 Jason Wu: Just a few minutes saved for me to share some stuff.
37 00:03:13.380 ⇒ 00:03:14.980 Uttam Kumaran: Sure, perfect, okay, okay.
38 00:03:16.350 ⇒ 00:03:18.800 Uttam Kumaran: Great.
39 00:03:20.330 ⇒ 00:03:21.750 Uttam Kumaran: Yeah, so…
40 00:03:21.830 ⇒ 00:03:31.710 Uttam Kumaran: I think just to give you a sense of progress, and then I just want to sort of jump right into sharing a couple pieces, I think overall, we are continuing on our ingestion pace of four sources at a time.
41 00:03:31.710 ⇒ 00:03:43.340 Uttam Kumaran: I appreciate you guys, where I know we’re pinging for access, and sure, it’s not something that, you know, the team is doing very often, but it’s really helping us just make sure that we have visibility into what data
42 00:03:43.340 ⇒ 00:03:52.060 Uttam Kumaran: And especially for QAing sources, our process for QAing something like Amazon is we are actually looking at the raw data
43 00:03:52.060 ⇒ 00:04:09.959 Uttam Kumaran: in addition to what is coming out of the UI, because oftentimes the business users at Element are using the UI as a source of truth. So, you know, as much access as you can provide us to that is actually really helpful. I know we have some outstanding requests on, you know, access to a couple more marketing sources, but
44 00:04:09.960 ⇒ 00:04:27.950 Uttam Kumaran: to kind of recap, I feel like we’re close to the finish line on most of what we considered P0, Pier 1 sources. What’s still remaining, is we’re ingesting Amazon. I know there was some movement on Walmart as well, so those are the two core P0s that we’re still looking to land.
45 00:04:28.020 ⇒ 00:04:31.459 Uttam Kumaran: Spins is another API that,
46 00:04:31.950 ⇒ 00:04:46.199 Uttam Kumaran: that us and the business team are making a decision on getting access to, and that will be next month. And Spins is really about competitive intelligence. How is Element versus, you know, another competitor sitting in Target? What is a market share?
47 00:04:46.280 ⇒ 00:05:01.180 Uttam Kumaran: And so we’re really trying to continue just to hammer landing as much data into Snowflake as possible. On the modeling side, we’re also making pretty steady progress. We have where to go stored, and a… oh, sorry, sorry, I didn’t see your hand.
48 00:05:02.770 ⇒ 00:05:05.909 Uttam Kumaran: Sorry, I can’t hear you, Shivani, it’s kind of…
49 00:05:10.030 ⇒ 00:05:12.580 Jason Wu: Okay, on the spins piece, can you hear me?
50 00:05:13.050 ⇒ 00:05:14.070 Uttam Kumaran: Yes, yes.
51 00:05:14.070 ⇒ 00:05:21.070 Jason Wu: On the spins piece, like, they sent documentation about, like, what it included, and, like, who…
52 00:05:21.070 ⇒ 00:05:37.820 Jason Wu: so basically, for context here, it’s, like, 15K a year or something like that to get this API access, and then they sent some documentation, but I didn’t want to… I’m not… I didn’t want to, like, review it and say, like, okay, yes, let’s go for it. I either wanted OTM’s eyes on it, or our tech team’s eyes on it, to say, like.
53 00:05:37.820 ⇒ 00:05:43.489 Jason Wu: recommend making this purchase. So… Has your team reviewed that with them?
54 00:05:43.900 ⇒ 00:06:03.629 Uttam Kumaran: I’ve reviewed it, and it’s fairly standard to what we’ve seen on our other clients. I think for this team, since it’s gonna be net new, I’m happy to present, like, a little bit deeper on, like, what’s in spins, because often it’s not 100% just element data, it’s actually, like, market intelligence and competitive intelligence.
55 00:06:03.630 ⇒ 00:06:07.789 Uttam Kumaran: I can also share a little bit about how the API works.
56 00:06:07.910 ⇒ 00:06:12.880 Uttam Kumaran: I didn’t… I didn’t confirm that, but yeah, I took a look at it, so…
57 00:06:13.200 ⇒ 00:06:15.529 Jason Wu: Would that be helpful for…
58 00:06:15.690 ⇒ 00:06:31.869 Jason Wu: Honestly, that’s not our call. It’s, does the business want the data or not? I think… I think people want the data, but I think they were like, let’s just make sure that what the… what we get with the 15K is, like, worthwhile. Okay. And so, I think in terms of, like, who… like, maybe it’s just not, like, a full…
59 00:06:32.100 ⇒ 00:06:32.820 Uttam Kumaran: Yeah, yeah, yeah.
60 00:06:32.820 ⇒ 00:06:35.030 Jason Wu: But, like, just, like, the little thing.
61 00:06:35.030 ⇒ 00:06:35.670 Uttam Kumaran: Yeah.
62 00:06:35.670 ⇒ 00:06:45.779 Jason Wu: like, I recommend making this decision so that we can block it, because I… I think that would be, like, a nice deliverable for the four box or something, then Phil could just be like, okay, I bless this, because…
63 00:06:45.780 ⇒ 00:07:02.730 Uttam Kumaran: Perfect, and to give you guys just a, like, a sneak peek into, like, what that is gonna look like, like, this is documents… this is documentation that we reference a bunch for some other clients. There are a few different APIs in here around stores and markets,
64 00:07:02.850 ⇒ 00:07:18.339 Uttam Kumaran: away… we were working with this data really heavily with another client right now, it’s a little bit… there’s a lot going on here, so I want to be really clear on, like, what it is that we are interested in. As you can see, their API is not really…
65 00:07:18.440 ⇒ 00:07:26.380 Uttam Kumaran: well documented, and there’s so many fields, so I will go ahead and put, like, a short one-pager on what are the things that this unlocks.
66 00:07:26.580 ⇒ 00:07:39.639 Uttam Kumaran: also based on the fact that there are several folks during our discovery period that, are asking for this information or are currently leveraging it. And then I can talk a little bit about… there’s some nuances to the API around data releases and
67 00:07:39.840 ⇒ 00:07:44.400 Uttam Kumaran: like, the format, so I can just, like, toss that into something, for sure.
68 00:07:44.400 ⇒ 00:07:49.019 Jason Wu: We currently get this data by downloading a report, though, right? So the…
69 00:07:49.500 ⇒ 00:07:59.190 Jason Wu: the whole reason we’re using the API is to pull it automatically instead of downloading the report manually. Okay. So let’s say, like, eventually with Omni.
70 00:07:59.190 ⇒ 00:08:13.220 Jason Wu: if we’re ingesting this data, we can say, how did this perform relative to Red Bull or whatever, and then that’s actually, like, not as something to go out to an external system, it’s just in our set of data.
71 00:08:14.190 ⇒ 00:08:33.150 Uttam Kumaran: Yeah, so this is market share, store share, shelf share, comparison between people in your category, but you’re exactly right. Similar to other sources, what this is gonna unlock is eliminating the need for that manual refresh, and of course, joining that with other data.
72 00:08:33.490 ⇒ 00:08:40.840 Uttam Kumaran: For example, this will be able to join to geographic data for our actual sales, you know, and start to compare, so…
73 00:08:41.710 ⇒ 00:08:42.409 Uttam Kumaran: Okay.
74 00:08:42.669 ⇒ 00:08:43.470 Uttam Kumaran: Great.
75 00:08:43.789 ⇒ 00:08:47.400 Uttam Kumaran: And so, yeah, I feel… yeah, sorry, go ahead.
76 00:08:48.470 ⇒ 00:09:02.329 Awaish Kumar: Yeah, I was just saying if we can get those reports, like, that are being sent via the email, or downloaded via UI, so we can… actually can map the fields that are being used to the API, and figure out what needs to be…
77 00:09:02.490 ⇒ 00:09:03.919 Awaish Kumar: Collected from the API.
78 00:09:04.290 ⇒ 00:09:13.709 Uttam Kumaran: Yeah, I can follow up from our discovery notes on who was using spins, and just… I can also run the memo by them and just get their feedback.
79 00:09:13.970 ⇒ 00:09:16.069 Uttam Kumaran: So, yeah, that’d be great.
80 00:09:17.570 ⇒ 00:09:37.000 Uttam Kumaran: And then on the modeling side, we are… we basically are typically modeling one to two marts at a time. So, when I say modeled, the first versions of these data models are going to be, and I’ll just kind of show you what the sort of net-net is, they’re going to be collections of
81 00:09:37.090 ⇒ 00:09:46.659 Uttam Kumaran: of, like, cleaned tables that are… that are joinable. And so, if I was to show you some versions of this, as you can see here, on the bottom left.
82 00:09:47.230 ⇒ 00:10:00.259 Uttam Kumaran: We have marts, around e-commerce, fulfillment, and so if I was to take fulfillment as an example, we now have 5 clean tables for fulfillment. So, SKU, warehouse, inbound shipments, orders, RMAs.
83 00:10:00.790 ⇒ 00:10:13.900 Uttam Kumaran: you may… like, our goal initially is to model this in the best practice way that we’ve seen in the past, and then drive towards getting something that the business can get feedback on. Unfortunately, like, sometimes
84 00:10:14.000 ⇒ 00:10:17.250 Uttam Kumaran: It’s hard to get feedback from the business on, like.
85 00:10:17.670 ⇒ 00:10:31.839 Uttam Kumaran: on, like, what tables and what is a star schema. So instead, what we’d be looking for is business feedback on the accuracy and on the number of dimensions or metrics that are made available. So if we were to, like, take one of these,
86 00:10:32.580 ⇒ 00:10:44.380 Uttam Kumaran: If we were to just take one of these, for example, you’ll see in… in this table, and I’ll just show you this, you’ll see this is, just SKU-level information. Sorry, this is kind of a…
87 00:10:44.810 ⇒ 00:10:47.840 Uttam Kumaran: Small table, let’s show you, like…
88 00:10:48.360 ⇒ 00:10:55.430 Uttam Kumaran: This is every… this is shipments. And so now, what you can see is it’s actually really helpful for us to have
89 00:10:55.710 ⇒ 00:10:58.040 Uttam Kumaran: IDs that can join to other systems.
90 00:10:58.140 ⇒ 00:11:14.499 Uttam Kumaran: We have things like statuses, and then we have, sort of, metadata fields with additional information. So across every single mart, we are going from the raw data that comes from the source system into these clean tables that can now be visualized in Omni.
91 00:11:15.230 ⇒ 00:11:17.549 Uttam Kumaran: Does that make sense in terms of, like.
92 00:11:17.840 ⇒ 00:11:25.149 Uttam Kumaran: like, our supply chain in terms of data models, and then I’ll sort of show, kind of, like, the orientation in Omni as well.
93 00:11:27.010 ⇒ 00:11:28.690 Jason Wu: Yes, this makes sense to me.
94 00:11:28.690 ⇒ 00:11:29.220 Uttam Kumaran: Okay.
95 00:11:29.490 ⇒ 00:11:30.130 Uttam Kumaran: And the code…
96 00:11:30.130 ⇒ 00:11:36.030 Andy Weist: What? You said, like, the… I guess, how do we validate
97 00:11:36.450 ⇒ 00:11:43.560 Andy Weist: correctness of the marked tables eventually. You said we’ll validate, like, correctness of the data.
98 00:11:44.000 ⇒ 00:11:45.329 Andy Weist: I guess, like, what is.
99 00:11:45.330 ⇒ 00:11:46.250 Uttam Kumaran: Great question.
100 00:11:46.250 ⇒ 00:11:51.560 Andy Weist: in asking for feedback from Element and saying, okay, this is the finalized
101 00:11:51.970 ⇒ 00:11:54.939 Andy Weist: schema of all of our marked tables.
102 00:11:55.210 ⇒ 00:12:09.790 Uttam Kumaran: Yeah, so to kind of give you the rudimentary process that’s happening, today, like, this is a good example of something that we just did, we’re in progress with Russell on, is we have our tables for…
103 00:12:09.820 ⇒ 00:12:27.310 Uttam Kumaran: retail data. And in particular, we were trying to get confirmation on retail sales information, and confirm that this matches with Russell… with what Russell is putting together, in the OKRs. And so right now, it’s very, like, caveman process of, like.
104 00:12:27.350 ⇒ 00:12:33.169 Uttam Kumaran: Hey, we pulled this from this raw data source, we aggregated it based on some logic.
105 00:12:33.440 ⇒ 00:12:45.120 Uttam Kumaran: does, like, does the number for, your end of quarter make sense? And so yesterday, or yesterday or Monday, when we were doing this with Russell, we found, yes, we were about $1,000 off.
106 00:12:45.120 ⇒ 00:12:58.400 Uttam Kumaran: I think we’ve basically isolated that there’s some timestamp mismatch. And so that’s the first QA, is that, are we basically matching? We did a very similar exercise with Bess on the finance side. But of course, as you know, that’s like, we’re not going to…
107 00:12:58.640 ⇒ 00:13:15.520 Uttam Kumaran: we’re not gonna, every day, be like, you tell me your number, and I’ll tell you my number. So, first of all, it’s like, is the SQL logic for where we’re aggregating accurate, right? At that point, we now know that, like, okay, we’re comfortable that our process from going from the raw
108 00:13:15.560 ⇒ 00:13:23.989 Uttam Kumaran: rows into this aggregate number is fair. The next piece of, like, observability is understanding a couple things. One is data fresh.
109 00:13:24.090 ⇒ 00:13:38.060 Uttam Kumaran: Right? So, are we getting the most recent data? That may mean looking at when Polytomic runs. That also may mean looking at, hey, Polyatomic ran, but maybe it didn’t load any data. So we’ll be looking at both of those for every single source.
110 00:13:38.310 ⇒ 00:13:45.029 Uttam Kumaran: Part of this will be part of some of the work that we’re gonna do around observability, that… that is a little bit later in the slide, but…
111 00:13:45.060 ⇒ 00:14:04.889 Uttam Kumaran: We’re gonna look at data freshness. We’re also gonna look at, like, completeness. So, let’s say data does land, let’s say Polyton ran, data does land, but it’s only a subset, right? The way we catch that could be on row counts. It could also be on, hey, we noticed that the sum is, like, quite a bit off. For example, let’s say this said 500,000.
112 00:14:05.150 ⇒ 00:14:15.159 Uttam Kumaran: We will be writing tests that sort of verify that these numbers stay within a forecasted range, so that any out-of-bounds metrics will cause a flag to get raised.
113 00:14:15.540 ⇒ 00:14:23.500 Uttam Kumaran: And then lastly, There’s gonna be much more complex metrics than just summing revenue.
114 00:14:23.600 ⇒ 00:14:34.009 Uttam Kumaran: And so, a lot of these metrics have actually never been measured in this way by the business. And so there may not be a point… there may not be another QA
115 00:14:34.100 ⇒ 00:14:43.620 Uttam Kumaran: they’re not… there may not be a comparable QA that exists for some of these metrics, because we didn’t have this level of dimensionality yet. And so…
116 00:14:44.250 ⇒ 00:14:48.800 Uttam Kumaran: This is actually just gonna become the source of truth, for these metrics.
117 00:14:48.950 ⇒ 00:14:54.660 Uttam Kumaran: And really having observability into how we’re moving from the individual row that comes in the raw
118 00:14:54.800 ⇒ 00:14:59.820 Uttam Kumaran: Database all the way to here is what this new system provides.
119 00:15:00.000 ⇒ 00:15:16.600 Uttam Kumaran: If you take Russell’s example, he’s downloading a pre-aggregated report from Emerson to produce this, meaning he himself cannot go through and tell me all of the orders that… and from all the orders, and what SKUs, and what stores this came from.
120 00:15:16.970 ⇒ 00:15:18.580 Uttam Kumaran: So, ultimately.
121 00:15:18.720 ⇒ 00:15:25.759 Uttam Kumaran: We can do that on our side, and we’re just matching sums right now, because there is no granularity in
122 00:15:25.960 ⇒ 00:15:27.640 Uttam Kumaran: In the… in that world right now.
123 00:15:27.850 ⇒ 00:15:29.569 Uttam Kumaran: So, that’s a little bit of a…
124 00:15:29.780 ⇒ 00:15:38.819 Uttam Kumaran: overview on, like, how we’re doing this QA, does that, like, make sense? I know we didn’t go too far into the observability piece, but that’s kind of, like, the.
125 00:15:38.820 ⇒ 00:15:42.610 Andy Weist: On the accuracy part, it does. I assume, like, eventually…
126 00:15:42.770 ⇒ 00:15:48.480 Andy Weist: probably through Omni, we’ll want to validate, like, the shape of the schema as well, and just.
127 00:15:48.480 ⇒ 00:15:49.390 Uttam Kumaran: Yeah.
128 00:15:49.390 ⇒ 00:15:54.500 Andy Weist: like, are all the attributes that we need for a SKU on this SKU, are the attributes on an outbound order?
129 00:15:54.620 ⇒ 00:15:58.660 Andy Weist: tracking information, blah blah blah, from stored on that order. I think that’s more of a,
130 00:15:59.800 ⇒ 00:16:14.360 Andy Weist: gonna be a manual process for the team to validate after the fact. Follow-up question on the accuracy part. Do we have unit or integration tests that we’re using mock data for to… to validate the dbt stuff we’re building? Like, how do we do…
131 00:16:14.890 ⇒ 00:16:17.990 Andy Weist: like, test-driven development against dbt.
132 00:16:17.990 ⇒ 00:16:33.050 Uttam Kumaran: Yeah, so, and we… I could show you, sort of, what an example of this, but within dbt, there are unit tests and broader, accuracy and shape-related tests. So, like, I think they’ll have a good example here,
133 00:16:33.690 ⇒ 00:16:48.299 Uttam Kumaran: One good example you could have is, hey, check whether an email address is a valid email, right? So, for example, we could say, make sure that there are no orders that are attributed to someone with a drink element email, which may be
134 00:16:48.330 ⇒ 00:17:01.909 Uttam Kumaran: like, you know, just employee orders, just as an example. This is the test that gets run, and so from the software side, you know, it’s the same concept of, like, a unit test, but it is actually on a column on a table.
135 00:17:01.950 ⇒ 00:17:16.000 Uttam Kumaran: So, you could look at things like test whether a valid email address. You could also think about test whether a column is ever negative, right? So, like, sums of a revenue should never be negative. You can add tests. Test whether a column is unique.
136 00:17:16.119 ⇒ 00:17:21.519 Uttam Kumaran: Right? So, like, if it’s a primary key, making sure there’s no duplicates. So we will be adding these unit tests.
137 00:17:21.730 ⇒ 00:17:26.820 Uttam Kumaran: These all get run as part of our dbt job. And so, just to sort of…
138 00:17:26.920 ⇒ 00:17:38.730 Uttam Kumaran: this is also another piece that I was going to share today, is just to give you an example of, like, how dbt is running, we are currently running, dbt on a schedule through GitHub Actions.
139 00:17:39.790 ⇒ 00:17:51.689 Uttam Kumaran: it’s not only just a little bit easy to configure, we’re not… we don’t have to pay for dbt Cloud. And so, there is a step in our dbt, you know, CLI function to actually execute what’s called dbt test.
140 00:17:51.770 ⇒ 00:18:00.160 Uttam Kumaran: Right now we don’t have any tests, but we will be populating these, and these will run not only on every PR that gets pushed.
141 00:18:00.250 ⇒ 00:18:05.910 Uttam Kumaran: So, when we open a PR for a model change, like, if I could give you a good example of one.
142 00:18:06.170 ⇒ 00:18:09.580 Uttam Kumaran: Like, let’s take…
143 00:18:10.470 ⇒ 00:18:23.369 Uttam Kumaran: for example, like, this was sort of the… adding the initial models for where to go. On this PR, there are actually, checks that run. And so we actually run a dbt PR validation
144 00:18:23.560 ⇒ 00:18:31.989 Uttam Kumaran: CICD step that not only executes the entire model, but then eventually will run our tests before it enters into the system.
145 00:18:32.070 ⇒ 00:18:46.949 Uttam Kumaran: And so, that’s why we have dev, staging, and production environments, so that we’ll recreate a version of production in staging to execute any new tests, any new models that get run, in order to verify that the entire test suite
146 00:18:47.500 ⇒ 00:18:49.969 Uttam Kumaran: You know, is… is valid.
147 00:18:50.420 ⇒ 00:18:53.139 Andy Weist: That makes sense. I’m thinking more, like, do we have…
148 00:18:53.830 ⇒ 00:18:58.310 Andy Weist: More of an integration test, like… Mock raw data gets…
149 00:18:58.960 ⇒ 00:19:10.370 Andy Weist: populate it into a MARC table, end-to-end test that way? Like, are we developing those right now? Is that part of this process? Is that… not familiar enough with the dbt test library to understand if that’s…
150 00:19:10.610 ⇒ 00:19:18.619 Andy Weist: what you said seems more like validation. Like, validate that revenue’s not negative. That’s not necessarily what I’m thinking of as far as integration tests, like.
151 00:19:18.910 ⇒ 00:19:23.270 Andy Weist: Let’s mock a… a raw table.
152 00:19:23.810 ⇒ 00:19:29.900 Andy Weist: run dbt against it, validate that the output in the BART table is what we expect.
153 00:19:30.360 ⇒ 00:19:32.990 Andy Weist: Do we have expectations of that being implemented?
154 00:19:33.160 ⇒ 00:19:45.439 Uttam Kumaran: Yeah, fair. I don’t think it’s in our… I don’t think we typically will implement, like, an end-to-end data… end-to-end dbt test with mock data. Typically, we will just use the
155 00:19:45.440 ⇒ 00:19:55.759 Uttam Kumaran: The raw data that comes in, as the assumption that it is, like, the data that gets used, and we’re… we’re basically running tests versus values or definitions.
156 00:19:56.000 ⇒ 00:20:03.530 Uttam Kumaran: I think, to your point, you’re… you’re saying, does the combination of rows resolve in the way we expect at the end?
157 00:20:04.020 ⇒ 00:20:20.379 Andy Weist: Yeah, like, I mean, if you’re writing normal software, you would say, okay, let me mock a day’s… a thousand records of orders in a day. I know what the sum total should be of those, because I’m populating the data. Let me make sure after it runs through all these functions that the mark table at the end of the
158 00:20:20.790 ⇒ 00:20:23.980 Andy Weist: run is that total, right? Just…
159 00:20:24.150 ⇒ 00:20:28.600 Uttam Kumaran: Yeah, so we will actually… we won’t use mock data for that, we will take a subset of…
160 00:20:29.060 ⇒ 00:20:32.799 Uttam Kumaran: Passed data to ensure that it’s never overwritten.
161 00:20:32.800 ⇒ 00:20:34.459 Andy Weist: Right, that’s static data.
162 00:20:34.460 ⇒ 00:20:44.250 Uttam Kumaran: Yeah, but static, yeah. So, like, so, like, you could say December 2024 revenue always resolves to this. There is no net new change that gets added.
163 00:20:44.540 ⇒ 00:20:51.950 Uttam Kumaran: In order to go one step further, we could always clone the existing backfill, make sure that stays static, and always confirm that
164 00:20:52.060 ⇒ 00:20:57.540 Uttam Kumaran: the values change. The values stay… stay static. I think… both…
165 00:20:57.970 ⇒ 00:21:02.779 Uttam Kumaran: processes, I think, achieve the same thing. For example, we were to say December 2024 revenue always
166 00:21:02.890 ⇒ 00:21:09.890 Uttam Kumaran: resolves to 100,000. Anytime a new change gets implemented and that changes, that’s a flag, right?
167 00:21:09.890 ⇒ 00:21:10.500 Andy Weist: Got it, okay.
168 00:21:10.500 ⇒ 00:21:11.789 Uttam Kumaran: I will see I will fail.
169 00:21:12.110 ⇒ 00:21:22.459 Andy Weist: That makes sense. I don’t want to eat up all the time. I’m interested in getting a little more involved in that part, since I’m going to be doing a little development against the NetSuite side, so I’ll likely follow up on this part, and would.
170 00:21:22.460 ⇒ 00:21:23.140 Uttam Kumaran: Great, yeah.
171 00:21:23.140 ⇒ 00:21:25.119 Andy Weist: Got it, but we can move on to…
172 00:21:25.120 ⇒ 00:21:25.600 Uttam Kumaran: Cool.
173 00:21:25.600 ⇒ 00:21:27.600 Andy Weist: I don’t want to hijack the whole thing.
174 00:21:28.120 ⇒ 00:21:37.220 Jason Wu: Is there any best practices in regards to test coverage for DBT? Like, is there a set percentage that you should be hitting, or anything like that?
175 00:21:37.630 ⇒ 00:21:43.599 Uttam Kumaran: Yeah, so… it’s a little bit… it’s a little bit tough, because… like, I would say, like.
176 00:21:45.160 ⇒ 00:21:57.850 Uttam Kumaran: It’s not like… like the typical software development lifecycle, where there are, like, test coverage percentages, because oftentimes one metric can have multiple tests run on it, and you’re… you’re…
177 00:21:57.930 ⇒ 00:22:02.099 Uttam Kumaran: Ultimately, if we were to have a test on every column.
178 00:22:02.130 ⇒ 00:22:15.060 Uttam Kumaran: like, I would say it’s fairly overkill. Some columns are just IDs. What we will actually orient is, based on the column type, we will have a baseline number of tests.
179 00:22:15.060 ⇒ 00:22:28.679 Uttam Kumaran: So if the column type is numeric, we’re always going to make sure that that’s always numeric. If it… if it… should it ever go negative? For example, if something’s an ID, it should never resolve to, like, a numeric, it should always be a string, it should be…
180 00:22:28.700 ⇒ 00:22:39.809 Uttam Kumaran: you know, should it be non-null? So, we will… I will say, we will try to have baseline tests for as many columns as possible. The complication comes in is when we have logic tests.
181 00:22:40.100 ⇒ 00:22:58.240 Uttam Kumaran: similar to what, Andy, we were talking about, is, like, this revenue always needs to resolve to this. Another example is, let’s say you have marketing channels. We should never see a marketing channel outside of this list. Those are things that are hard to infer when developing the model without the external
182 00:22:58.360 ⇒ 00:23:11.499 Uttam Kumaran: stakeholders giving you feedback on, like, what tests we should run. So, I would say we’re expecting there to be… to give you… put it a different way, we are expecting there to be several tests on every single MARTS model.
183 00:23:11.870 ⇒ 00:23:21.569 Uttam Kumaran: Whether there are tests on raw and intermediate, I think is sort of TBD, depending on how we develop, but on every single MARTS model, yes, there will be
184 00:23:21.760 ⇒ 00:23:24.810 Uttam Kumaran: A series of tests, and those will grow over time.
185 00:23:25.900 ⇒ 00:23:39.350 Andy Weist: I think, Steve, if we hold ourselves to every column of every mark table should be checked via a test, that could give us, like, a coverage number, so to speak. And that’s not unrealistic, in my opinion.
186 00:23:40.450 ⇒ 00:23:51.210 Andy Weist: You’re saying, like, we can’t test every permutation of every value, like, that’s unreasonable. I think, like, coverage can be defined as, are we validating a value in every column?
187 00:23:51.640 ⇒ 00:23:54.000 Andy Weist: of every table, every marked table.
188 00:23:54.260 ⇒ 00:23:54.820 Uttam Kumaran: Yeah.
189 00:23:55.260 ⇒ 00:24:00.559 Uttam Kumaran: Ultimately, some tests are gonna… you kind of… we actually want to drive towards more of that… those business logic tests.
190 00:24:00.660 ⇒ 00:24:03.649 Uttam Kumaran: Because those are the actual things that fail
191 00:24:03.850 ⇒ 00:24:06.630 Uttam Kumaran: More than, like, is this ever a string or not?
192 00:24:06.850 ⇒ 00:24:16.630 Uttam Kumaran: Those also fail, but it’s oftentimes, like, an out-of-bounds value that you never wrote a test for that enters the system, where it passed all the type checks.
193 00:24:16.820 ⇒ 00:24:34.179 Uttam Kumaran: if dbt ran totally fine. That’s the stuff that is the meat of the test that we will… I actually want to prioritize first, because that ultimately affects… if a number is off, it really hits, like, the trust on the business, and so we will prioritize the business logic checks.
194 00:24:34.350 ⇒ 00:24:43.410 Uttam Kumaran: if we were to… if we were to consider, like, the priority, because adding the ID and numeric value, non-null, unique tests is… is fairly trivial.
195 00:24:47.690 ⇒ 00:25:05.020 Uttam Kumaran: Great. I think one thing I wanted to… so we actually covered… I was gonna walk through, dbt. This is just… I wanted… I wanted to show everybody where you can go in and see how dbt is running. Right now, it’s… it’s running right now. It’s taking roughly, like, 4-5 minutes. We will actually end up
196 00:25:05.090 ⇒ 00:25:07.900 Uttam Kumaran: breaking this out.
197 00:25:08.150 ⇒ 00:25:24.560 Uttam Kumaran: so that’s not just in one, like, huge run. What this is doing is creating our views and our tables in Snowflake. Oftentimes, there’s a couple strategies for separating out workflows. Typically, we recommend trying to align
198 00:25:24.730 ⇒ 00:25:37.809 Uttam Kumaran: the models that you run with the SLAs from the business, and then the next version is sort of, by business domain, or by mart. That way, if something fails in the e-commerce mart, it doesn’t
199 00:25:38.100 ⇒ 00:25:40.569 Uttam Kumaran: Delay the retail mart.
200 00:25:40.690 ⇒ 00:25:47.830 Uttam Kumaran: And so, there is… This is where it just gets into more, like, workflow, orchestration, optimization.
201 00:25:49.280 ⇒ 00:25:55.740 Uttam Kumaran: don’t… there’s no red flags right now. This is pretty… it’s pretty stable, given how much we’re doing, so…
202 00:25:58.810 ⇒ 00:26:03.610 Uttam Kumaran: Okay, I wanted to just… also just give a little bit of a…
203 00:26:04.060 ⇒ 00:26:23.420 Uttam Kumaran: overview of, like, what we’re doing within Omni. So I’m really pumped we have, sort of, our Omni trial instance set up. We’ve made some good progress this week on creating, the baseline topics in Omni, and that’s something that I want to just,
204 00:26:23.660 ⇒ 00:26:25.520 Uttam Kumaran: share with this group is…
205 00:26:26.250 ⇒ 00:26:42.820 Uttam Kumaran: sort of the definitions, of how we’re gonna actually model and display data in Omni. It’s sort of similar to how we’ve been describing dbt and Snowflake, but kind of the way I want to sort of talk about it is the core piece, that I think
206 00:26:42.820 ⇒ 00:26:59.919 Uttam Kumaran: is really gonna be where the art is here, is, like, modeling topics. And so if anyone here has used something like Looker or Tableau before, you may be used to, hey, we’re just gonna join every table together and give business stakeholders, like, hundreds and hundreds of metrics.
207 00:26:59.970 ⇒ 00:27:11.780 Uttam Kumaran: Where they have a customer ID here, they have a customer ID here, and it’s very, very confusing. Instead, sort of, Omni introduces this concept of topics, which… it’s basically curated, data sets.
208 00:27:11.810 ⇒ 00:27:21.689 Uttam Kumaran: a good topic example is wholesale data versus retail data versus executive, dashboard views. And so this is where it’s…
209 00:27:21.730 ⇒ 00:27:38.369 Uttam Kumaran: there’s not a great, like, rule of thumb, except for this is sort of, like, the curation that our team will do. Additionally, the way Topics actually impacts Omni is really based on how the AI is used. And so one example, that I can just…
210 00:27:38.420 ⇒ 00:27:43.109 Uttam Kumaran: share is, like, right now, if you go into Omni, you can click on
211 00:27:43.230 ⇒ 00:27:56.540 Uttam Kumaran: salty, on the left. You can actually look at that there are topics based on finance, retail, and wholesale. I don’t know, Amber, which is the topic, like, for wholesale that we were working on today?
212 00:27:56.980 ⇒ 00:28:04.229 Amber Lin: Today we are looking at customers, so we wanted to get their address and their most recent shipping address.
213 00:28:04.660 ⇒ 00:28:14.480 Uttam Kumaran: So I can go in here and just say, tell me about what data exists on wholesale customers. And so what’s happening under the hood, is…
214 00:28:15.410 ⇒ 00:28:22.169 Uttam Kumaran: Omni is actually taking this, it’s going to run, a query… set of queries, and…
215 00:28:22.250 ⇒ 00:28:33.649 Uttam Kumaran: and set of queries on both contexts we’ve loaded into Omni, as well as on the actual data it has available, to basically put together, you know, this document. And so it is an agentic
216 00:28:33.720 ⇒ 00:28:52.859 Uttam Kumaran: service that runs on all the contacts made available to it. So ultimately, yes, we are driving towards creating dashboards, but the reason why our team tries to recommend Omnis is really because of this AI feature. We sort of have a perspective that not… the AI feature not only opens up the
217 00:28:52.900 ⇒ 00:29:05.199 Uttam Kumaran: access to data for a lot more people than Dashboards does, but we have a feeling that people are actually going to like interacting with business questions in this manner. When you interact with a data analyst, you don’t just…
218 00:29:05.290 ⇒ 00:29:18.970 Uttam Kumaran: say, okay, give me a printout of a dashboard, and then, like, I’ll go away. You ask questions. And so, the way that natural language, allows for that is actually, like, really amazing. And so, you can see that
219 00:29:18.970 ⇒ 00:29:28.759 Uttam Kumaran: Lobby, or the service, actually pulls that information together, gives you some information, and then actually gives you some additional things to explore.
220 00:29:28.760 ⇒ 00:29:48.340 Uttam Kumaran: There’s functionality around feedback, there’s functionality around exporting, but this is just one core part of Omni that I think is really the core highlight. I can go deeper into showing you how context gets added, but just wanted to stop here. If there’s anything particular about Omni that
221 00:29:48.960 ⇒ 00:29:53.549 Uttam Kumaran: Any questions or any thoughts on this before I just continue forward?
222 00:29:54.090 ⇒ 00:30:05.289 Andy Weist: You asked for an actual set of data through natural language, like, give me the last 10 customers who placed orders or something. Does it provide the underlying, like, underlying SQL that ran?
223 00:30:06.350 ⇒ 00:30:07.179 Uttam Kumaran: Yeah, so lemon.
224 00:30:07.180 ⇒ 00:30:11.710 Andy Weist: Which is how do we… how do we worry about deterministic
225 00:30:11.860 ⇒ 00:30:16.650 Andy Weist: actions within a natural language agent platform.
226 00:30:16.650 ⇒ 00:30:26.569 Uttam Kumaran: Yeah, so it will… it will give you the query, that ran. I think this is where it will just take a lot of steering, from our team.
227 00:30:26.900 ⇒ 00:30:40.880 Uttam Kumaran: And so what do you… what are you seeing here is exactly… it… it ran a query, which it said, go to the wholesale customer’s topic, pull the last 100 rows where the last order date is in the 30 days. These are wholesale partners. And it’s actually gonna…
228 00:30:41.050 ⇒ 00:30:51.010 Uttam Kumaran: give you not only, like, a sort of rich export of that data, it’s actually gonna do some summarization. All of this, though, is very steerable, meaning this is, like.
229 00:30:51.320 ⇒ 00:31:00.579 Uttam Kumaran: We just set this up, like, 3 days ago, it’s very out of the box. So, we can steer it to be conservative, more liberal with the way it answers.
230 00:31:00.580 ⇒ 00:31:15.940 Uttam Kumaran: more verbose, and we will actually be pushing a lot of context of what we learned through our discovery process into all the metadata fields available. So that’s column-level definitions, table-level definitions, topic-level definitions, and
231 00:31:15.990 ⇒ 00:31:21.699 Uttam Kumaran: the AI will actually read from that to form its answer. I think your question is about determinism.
232 00:31:23.000 ⇒ 00:31:33.839 Uttam Kumaran: there’s… there’s sort of… the way this gets steered is it’s only able to write a query based on the metrics that it has access to. So ultimately, if we feel like this is the governed layer where everything in here
233 00:31:34.060 ⇒ 00:31:42.359 Uttam Kumaran: combined is accurate, then I think that’s where I would say we feel really comfortable in allowing this. Additional… yeah.
234 00:31:42.790 ⇒ 00:31:46.359 Jason Wu: Is there a place where somebody could just click in and see the SQL that ran?
235 00:31:47.790 ⇒ 00:31:49.689 Andy Weist: Yeah, I’m thinking ahead towards, like.
236 00:31:49.890 ⇒ 00:31:54.740 Andy Weist: Shivani runs something, I run something, we get two different datasets, because…
237 00:31:54.860 ⇒ 00:32:04.159 Andy Weist: call it, she’s in a different time zone, right? And cut off at midnight or something like that. Like, these are real examples I’ve seen from people trying to run reports in our systems already.
238 00:32:04.990 ⇒ 00:32:06.040 Andy Weist: I think…
239 00:32:06.210 ⇒ 00:32:14.570 Andy Weist: like, that is where the raw SQL would be helpful. Andy, why is your data different than mine? To answer that question.
240 00:32:14.570 ⇒ 00:32:15.000 Jason Wu: Totally.
241 00:32:15.000 ⇒ 00:32:19.250 Andy Weist: I worry about someone coming in here saying.
242 00:32:19.590 ⇒ 00:32:35.549 Andy Weist: you know, a much more complex thing than give me the last 10 customers, and the way it will join that being non-deterministic from time to time, because of the way I worded the question slightly differently than Shivani. So I just want a way to sort of audit
243 00:32:35.740 ⇒ 00:32:38.979 Andy Weist: Where the data came from when it does mismatch, because it will happen.
244 00:32:39.850 ⇒ 00:32:52.229 Uttam Kumaran: Fair. I think there’s two pieces. One, the topic is actually what’s gonna start to fix what joins are available, but you’re right, if you say something today versus the data refreshes tomorrow and there’s
245 00:32:52.230 ⇒ 00:33:05.340 Uttam Kumaran: you kind of look at it, it could be different. You can always explore and just immediately see, like, the query that run. I’m fairly confident we can go edit that so it’s a little bit more forward here with what the query it ran.
246 00:33:05.340 ⇒ 00:33:21.850 Uttam Kumaran: But again, this is just, like, this is the refinement process of this tool, is, like, what is our confidence level? And we will actually be doing a lot of unit tests on this, which is, like, testing the fact that it responds in the way we expect as the data model and topics change.
247 00:33:21.930 ⇒ 00:33:22.880 Uttam Kumaran: You know, so that’s where…
248 00:33:22.880 ⇒ 00:33:24.749 Andy Weist: You can unit test this? Like, how would you go about that?
249 00:33:24.750 ⇒ 00:33:31.989 Uttam Kumaran: we… it has… there’s an MCP, and there’s an… and they have a… yeah, either through the MCP or through their CLI.
250 00:33:32.050 ⇒ 00:33:45.229 Uttam Kumaran: we would just issue the query, see the response, and then you kind of run… in typical, like, an agentic development, you’re… you’re almost, like, running evaluations, because oftentimes, the answer is, like, free text, right? So…
251 00:33:45.230 ⇒ 00:34:00.150 Uttam Kumaran: Instead, there’s a couple ways. You could, one, ask for just a simple result out of the SQL. So then you’re doing more, like, is it generating the right SQL? The other piece is you could have, sort of, LLM as a judge, where another LLM is sort of judging, hey, are these things answering similarly?
252 00:34:00.160 ⇒ 00:34:19.680 Uttam Kumaran: I think, to your point about asking things differently, yes, there will have to be, like, some real significant training on the team on the way to interact with this tool, but that isn’t more of, like, oh, you have to specify the join. It’s actually just, how do you go validate the references in the way it answered, you know?
253 00:34:20.170 ⇒ 00:34:22.969 Uttam Kumaran: Like, where did it pull from? Where did you get this from?
254 00:34:23.250 ⇒ 00:34:24.530 Uttam Kumaran: But ultimately.
255 00:34:25.000 ⇒ 00:34:34.890 Uttam Kumaran: like, it’s gonna be on the data team to make sure that all the metadata, all the columns are oriented in a way where that is clear. For you to jump from this to the explore and be like, okay, I’m…
256 00:34:35.000 ⇒ 00:34:45.009 Uttam Kumaran: fairly comfortable looking at the raw data that contributed to this, how it got formed. Not everyone, though, is going to be able to see this and give that, but yes, our team.
257 00:34:45.010 ⇒ 00:34:45.440 Andy Weist: Correct.
258 00:34:45.440 ⇒ 00:34:48.349 Uttam Kumaran: We’ll look at that and say, why did… why did this differ?
259 00:34:48.620 ⇒ 00:34:56.130 Andy Weist: Yeah, I think I… you know, this is more of an internal element thing, but, like, I think that we need to make sure the entire team’s diligent about, like.
260 00:34:57.420 ⇒ 00:35:01.330 Andy Weist: Auditing the source of the information, documenting the source.
261 00:35:01.570 ⇒ 00:35:17.749 Andy Weist: So that, if these are business-critical reports and stuff that we’re using, like, there’s some step of, okay, yes, this is pulling correctly. Not everyone can read the sequel, you’re right, but, like, it should pass by someone who can, if it’s a mission-critical thing, and I just… I…
262 00:35:17.820 ⇒ 00:35:21.659 Andy Weist: I want that to be kind of, like, a discussion point as we keep moving forward with this.
263 00:35:21.660 ⇒ 00:35:23.670 Uttam Kumaran: Yeah, yeah, going on.
264 00:35:24.290 ⇒ 00:35:35.250 Jason Wu: Oh, I was gonna say, I think to avoid that also, like, once you kind of go past that test, I think I saw this in the Omni demo, right? But there’s a… is there a way to basically kind of say, let’s lock this in, and then that becomes, like, something that you can.
265 00:35:35.250 ⇒ 00:35:44.440 Uttam Kumaran: Yeah, yeah, so you can basically lock in certain types of questions, that are frequently answered, and within Omni, too,
266 00:35:44.440 ⇒ 00:35:57.919 Andy Weist: I mean, you can build dashboards and stuff, right? Like, once we generate the query once, it should be the source of truth, and it should be on a dashboard. I’m more concerned about someone’s like, I just want this data real quick, how do they know that they’re getting the exact right thing that they’re actually looking for?
267 00:35:58.530 ⇒ 00:36:03.990 Uttam Kumaran: Yeah, you may say I want retail sales data, I want retail PO, like, revenue data, and so part of this is, like.
268 00:36:03.990 ⇒ 00:36:04.590 Andy Weist: Yeah, right.
269 00:36:04.590 ⇒ 00:36:23.840 Uttam Kumaran: Yeah, we… I think we just need to… we need to really do it over this next pilot, is we’re just planning on doing, really, a lot of stress testing with this… with this crew on this call. So that we feel comfortable putting ourselves in the shoes of the business users, asking in these various ways, and Omni responding in the way that we feel comfortable.
270 00:36:23.900 ⇒ 00:36:30.889 Uttam Kumaran: Ultimately, some people will go through the dashboards, and that’ll be fixed, those are fixed queries. But I really…
271 00:36:32.340 ⇒ 00:36:51.790 Uttam Kumaran: I’m… I feel like a lot of people are going to leverage the AI, and so how can we both observe what people are asking, listen to some of the feedback, like, that’s coming through thumbs up and thumbs down, and then we will try our… we will… we will do those unit tests that I described, which is, like, having the AI issue a query, respond in a way that’s predictable.
272 00:36:53.250 ⇒ 00:36:58.940 Uttam Kumaran: But again, it’s… I think this is both a feature and something that we have to design our process around.
273 00:36:59.790 ⇒ 00:37:06.349 Jason Wu: So, do we get visibility in that sort of feedback loop, then, from the end user?
274 00:37:06.350 ⇒ 00:37:06.910 Uttam Kumaran: Yes.
275 00:37:06.910 ⇒ 00:37:07.470 Jason Wu: and…
276 00:37:08.020 ⇒ 00:37:12.110 Uttam Kumaran: So this isn’t… this isn’t, like, totally put together, this is their out-of-the-box, like.
277 00:37:12.150 ⇒ 00:37:30.429 Uttam Kumaran: what are people doing in Omni. Nicely, all of these are Omni topics as well, but we will actually get to see, like, who’s using it, what are they using it for, some of the logs. And so we can start to flag things like thumbs up versus thumbs down feedback, long-running queries.
278 00:37:30.610 ⇒ 00:37:35.419 Uttam Kumaran: You know, places where people are, yeah, long-running queries,
279 00:37:36.250 ⇒ 00:37:45.229 Uttam Kumaran: when we can start to sort of judge these things a little bit, you know, more heavily. A good example here is you can just start to see, like, what are all the prompts that were asked in a given session.
280 00:37:45.480 ⇒ 00:37:48.849 Uttam Kumaran: So we have all the logs.
281 00:37:50.300 ⇒ 00:37:51.490 Jason Wu: Nice. Okay.
282 00:37:52.370 ⇒ 00:38:01.280 Uttam Kumaran: And so this is kind of one thing, and then, Shivani, I just want to make sure you have time. The last piece I wanted to just talk about is, like, adoption and observability. It’s two things that we haven’t really…
283 00:38:01.360 ⇒ 00:38:12.649 Uttam Kumaran: been presenting on, which, like, ideally, now that we have an avenue where people will start using it, we want to start to look at how often people are using Omni, and so the
284 00:38:12.770 ⇒ 00:38:24.590 Uttam Kumaran: whether this dashboard or another one, you’ll start to be able to see what content people are looking at, what models they’re querying, and then through the AI, it’s what question they’re asking. Ideally, that should help us define our roadmap.
285 00:38:24.770 ⇒ 00:38:27.260 Uttam Kumaran: And so that’s one piece. And then observability…
286 00:38:27.600 ⇒ 00:38:40.569 Uttam Kumaran: it’s basically the health of the entire system. So, from polyatomic to Snowflake to dbt to here, where are all those breaking points? And so, we’ll start presenting as we start to layer on tests and things across the entire system, so…
287 00:38:42.040 ⇒ 00:38:45.389 Andy Weist: What platform are you gonna use for observability?
288 00:38:46.190 ⇒ 00:38:52.550 Uttam Kumaran: Yeah, so… I… we were actually just gonna pipe it into, Omni as a dashboard.
289 00:38:52.770 ⇒ 00:39:03.110 Uttam Kumaran: So, we were gonna have logs land in Snowflake, around tests for each of those systems, and then present on issues or, you know,
290 00:39:03.280 ⇒ 00:39:13.850 Uttam Kumaran: long-running dbt models, and we’re gonna log that in Snowflake and present it also as a dashboard within Omni. There are other options on more, like, you could say, like.
291 00:39:13.990 ⇒ 00:39:21.859 Uttam Kumaran: For, like, for example, if a metric is, like, 3 standard deviations up, there are some tools that are, like, have that more, that are, like.
292 00:39:22.100 ⇒ 00:39:27.159 Uttam Kumaran: Forecast a metric out, find the anomaly, like, anomaly detection things, but for the most part.
293 00:39:27.370 ⇒ 00:39:31.330 Uttam Kumaran: We will be logging these deterministic tests and then displaying them here in Omni.
294 00:39:31.790 ⇒ 00:39:34.729 Andy Weist: Okay, yeah, we’re leveraging Datadog for other software here, so…
295 00:39:34.730 ⇒ 00:39:35.120 Uttam Kumaran: Okay, great.
296 00:39:35.120 ⇒ 00:39:41.560 Andy Weist: I’m wondering if, like, it’s very similar as far as dashboards, but that’s where our software observability’s gonna land. Okay.
297 00:39:41.560 ⇒ 00:39:48.300 Uttam Kumaran: Yeah, so we use this, we use this tool called Metaplane, which Datadog actually purchased.
298 00:39:48.540 ⇒ 00:39:54.010 Uttam Kumaran: We’re… this is, like, really great tool for this type of stuff. So, like.
299 00:39:54.010 ⇒ 00:39:54.680 Andy Weist: Okay.
300 00:39:54.680 ⇒ 00:40:00.339 Uttam Kumaran: This metric, like, has been trending like this, and then one day it just flies, like, help me diagnose.
301 00:40:00.560 ⇒ 00:40:15.329 Uttam Kumaran: So this is… I… yeah, curious to see if this is already… if this is integrated into your Datadog plan, but I think Datadog may also have their own native version, but, like, would love to just pipe logs into there. We can pipe Snowflake DPT logs, whatever, into there, too.
302 00:40:16.590 ⇒ 00:40:35.000 Andy Weist: If it’s not too inconvenient, I’d like to consolidate those. Steve, Jason, we should talk about it more, but as far as observability, I’d like to consolidate it to one place. It sounds like it’d be a very similar tooling step between Omni and Datadog, so if we can leverage what we have already, I think that.
303 00:40:35.000 ⇒ 00:40:42.050 Uttam Kumaran: Yeah, and we would rely on the same, like, alerting processes. Like, if you guys are gonna do PagerDuty or have it go to Slack, like, I would love to…
304 00:40:42.300 ⇒ 00:40:43.279 Uttam Kumaran: use Datadocs.
305 00:40:43.280 ⇒ 00:40:55.920 Andy Weist: Learning Chain is already set up in Datadog, that’s why it’s, like, kind of easy. As long as we… I can set you guys up with API access and everything, if we just start dumping things there, it’s… it’s easier than controlling in two different places.
306 00:40:55.920 ⇒ 00:40:57.780 Uttam Kumaran: Cool, okay, perfect. Great.
307 00:40:58.670 ⇒ 00:41:02.939 Jason Wu: And what’s… what’s the pricing model of Omni? Is it token-based?
308 00:41:03.430 ⇒ 00:41:12.739 Uttam Kumaran: So there’s two… there’s two pricing, so it’s one is gonna be users, and then they give you a cap, they give you some tiering on the number of tokens.
309 00:41:13.320 ⇒ 00:41:26.760 Uttam Kumaran: this is where I think, like, there… we can totally negotiate. I don’t think we have a clear understanding of, like, how many tokens are gonna get used yet, so our biggest barometer for this initial sort of contract is really gonna be on the number of seats.
310 00:41:26.900 ⇒ 00:41:46.790 Uttam Kumaran: and the type of seat, whether it’s a developer seat or a viewer seat, it comes with a standard, sort of, like, set of, tokens. So that’s, I think, probably the thing I want to drive towards soon, is for this crew to start using Omni, and then, Max will give us a call for sure and be like, how’s it going? What do you want to do? So…
311 00:41:47.150 ⇒ 00:41:50.349 Jason Wu: Watson, can you ride a skid? How long is our, pilot with them?
312 00:41:50.510 ⇒ 00:42:02.819 Uttam Kumaran: Yeah, so for us, like, we’ve been… typically, they do, like, two-week pilots. We’ve, like, because of our relationship, they’re… they’re open to doing 4-6 week pilot, and so that’s… we’re kind of in, like, week 2 of that right now.
313 00:42:03.670 ⇒ 00:42:14.919 Uttam Kumaran: they’re pretty easy to… to get along with. I think the biggest thing I want to try to find is a… is a time, maybe next week, to sync back with Max. And I don’t know, Shivani, I don’t know if you’re going to the Omni Conference.
314 00:42:15.100 ⇒ 00:42:18.070 Uttam Kumaran: But I was just gonna propose that
315 00:42:18.200 ⇒ 00:42:21.199 Uttam Kumaran: Once we start to have you guys in the platform, then he can…
316 00:42:21.310 ⇒ 00:42:24.939 Uttam Kumaran: Sort of have the conversation again and refresh on, like, how pricing works.
317 00:42:25.110 ⇒ 00:42:26.890 Uttam Kumaran: Yeah.
318 00:42:27.950 ⇒ 00:42:28.610 Jason Wu: Alright.
319 00:42:30.620 ⇒ 00:42:36.609 Uttam Kumaran: Okay, sorry, Shiva, 3 minutes… I got you 3 minutes, but I’m… I’m free after, so…
320 00:42:36.990 ⇒ 00:42:41.000 Jason Wu: Okay, let’s see, so I…
321 00:42:42.560 ⇒ 00:42:54.429 Jason Wu: I was talking with Brianna and Phil today, and I just wanted to, like, share… I’m still working through it, but I think I just wanted to share, like, what Brianna is yearning for.
322 00:42:54.750 ⇒ 00:42:59.730 Jason Wu: She’s the COO at Element, and… Is hoping to, like.
323 00:43:00.010 ⇒ 00:43:17.449 Jason Wu: get a more 50,000-foot view of this project and make it, like, just, like, really simple, like, verb-led, noun ending, like, what is happening and when, in terms of just, like, communicating the project. And so, I’ve been just drafting something based off of,
324 00:43:17.870 ⇒ 00:43:27.100 Jason Wu: Based off of what we spoke about, and it’s, like, almost like phase one was, like, the vendor pilot, like, do we feel like we found the right vendor to work with to, like, make this work happen?
325 00:43:27.570 ⇒ 00:43:35.609 Jason Wu: Phase 1B is, like, it’s not that you haven’t started building this infrastructure, but it’s, like, really, like, being comprehensive, and…
326 00:43:36.190 ⇒ 00:43:52.419 Jason Wu: getting all of the data sources congested, standardized, modeled, QA’d, connected to our BI tool. And that’s, like, this push that we’re currently, like, exploring, like, the revised contract and things like that, right, with them, so I’m just trying to explain what it is.
327 00:43:54.350 ⇒ 00:44:08.590 Jason Wu: Then there’s, like, internal rollout. So, like, when you’re talking about how many seats and things, there are folks that, like, I think I’ve shared a little bit, but there’s, like, this incoming VP layer, and it’s like, what will that look like? Like, for the VP layer, maybe what we’re doing is, like.
328 00:44:09.080 ⇒ 00:44:20.419 Jason Wu: in their first 90 days, we’re making sure that they’re well equipped with the dashboards that they need to understand their business. And so, like, that’s, like, a part of their, like, onboarding, is, like, we’re working with them to, like, define those dashboards.
329 00:44:20.560 ⇒ 00:44:25.509 Jason Wu: But then it’s, like, also identifying the other members of the Element team, which I think
330 00:44:26.320 ⇒ 00:44:35.980 Jason Wu: the… you all can, like, help me, like, come up with that list of saying, like, who else would be in Omni? And, like, who else would be, like, users of this data?
331 00:44:36.250 ⇒ 00:44:51.639 Jason Wu: And then what does it mean for us to, like, drive adoption, right? Like, training people on not only, like, what is this data stack that we’ve built, but also, like, what are good prompts that you can ask? And, like, what are the, like, kind of boundary conditions that this, like, BI tool has?
332 00:44:51.870 ⇒ 00:45:00.590 Jason Wu: And so, it’s like training, whether it’s team leads, or VPs, or, like, who are the people internally that we’re going to
333 00:45:00.660 ⇒ 00:45:12.519 Jason Wu: get on board with, like, this, like, phase here of, like, internal rollout, and then what Phil was calling me is, like, almost, like, 2A is internal rollout, but then we also have, like, the external connection, which is integrating
334 00:45:12.620 ⇒ 00:45:20.420 Jason Wu: what everything we’re doing with NetSuite, with Atomic, maybe, maybe with Keen, a marketing mix model. And so…
335 00:45:20.790 ⇒ 00:45:29.679 Jason Wu: I would say that that’s kind of, like, April through December, as opposed to just, like… I think earlier, I was kind of like, that will not start until we’ve done
336 00:45:29.780 ⇒ 00:45:35.350 Jason Wu: 1B, and yet I’m hearing on one hand, like, how can we connect atomic sooner?
337 00:45:35.530 ⇒ 00:45:48.949 Jason Wu: Right? So, like, that is, like, this thing that’s, like… I thought it was just gonna be, like, we’re not even talking to Atomic until September, but it might be that we’re actually, like, trying to figure out how to link Atomic to Snowflake sooner. So this is just, like, not even…
338 00:45:48.960 ⇒ 00:46:05.400 Jason Wu: I’m just sharing this, it’s not even, like, looking for… I mean, yes, feedback is welcome, but I’m still working through this. I think I just want to share, like, what we’re trying to get to is, like, a very simple view of this project for anybody at Element to be like, okay, like, I, like, understand what phase we’re in right now.
339 00:46:06.440 ⇒ 00:46:07.810 Jason Wu: I just want to share that.
340 00:46:08.520 ⇒ 00:46:09.070 Uttam Kumaran: Cool.
341 00:46:10.320 ⇒ 00:46:13.989 Jason Wu: Because I feel like we’ve gone through so many different versions with them of, like.
342 00:46:14.360 ⇒ 00:46:14.760 Uttam Kumaran: Yeah.
343 00:46:14.760 ⇒ 00:46:32.000 Jason Wu: week by week, and, like, Gantt chart, and I think what she’s saying, what Brianna’s saying is, like, yes, you need all of those things to run an effective project, and what is the communication tool internally? So, like, people, like, are like, okay, I can now understand that I’m in this phase of this project.
344 00:46:32.790 ⇒ 00:46:37.210 Uttam Kumaran: Cool. Yeah, I think the external connection, I agree, is the piece that I’m sort of, like.
345 00:46:37.980 ⇒ 00:46:42.480 Uttam Kumaran: we’ve kind of gone in and out on, so I totally agree with you there.
346 00:46:42.800 ⇒ 00:47:01.149 Jason Wu: Yeah, like, initially, I was thinking that’s, like, September onward, like, we’re not even thinking about that, but now, obviously, tech team is thinking about how to, like, do some stuff in Snowflake to get it into NetSuite, so it’s, like, that’s probably the murkiest one to me, also. So just my follow-up to figure out how to name that more clearly.
347 00:47:02.040 ⇒ 00:47:02.550 Uttam Kumaran: Okay.
348 00:47:04.140 ⇒ 00:47:05.040 Jason Wu: Okay.
349 00:47:08.080 ⇒ 00:47:17.789 Andy Weist: And just to call out, in the next month, I’m gonna be doing a pretty deep dive into the ERP NetSuite integration piece, so…
350 00:47:18.360 ⇒ 00:47:24.329 Andy Weist: I’m gonna be shoring up a lot of the, kind of, infrastructure answers to how that’s gonna work.
351 00:47:24.430 ⇒ 00:47:25.350 Jason Wu: But…
352 00:47:25.350 ⇒ 00:47:34.179 Andy Weist: Adam, for your… Cool. Context, like, there’ll be a lot of hands-on stuff in April for…
353 00:47:34.840 ⇒ 00:47:41.199 Andy Weist: whatever, call over versus ETL connection between Snowflake and… and, NetSuite.
354 00:47:41.600 ⇒ 00:47:44.370 Uttam Kumaran: Great, yeah, so I feel like as soon as I can…
355 00:47:44.690 ⇒ 00:47:57.629 Uttam Kumaran: you know, help you maintain… we have sort of roles and everything there, but as soon as you kind of have some visibility to that, I can… we can talk about roles, like Snowflake tasks, however the methodology is, and we can be a good pair there.
356 00:47:58.480 ⇒ 00:47:59.030 Andy Weist: Cool.
357 00:48:05.820 ⇒ 00:48:08.130 Jason Wu: I’ll think about other ways to say this, but…
358 00:48:09.250 ⇒ 00:48:17.650 Uttam Kumaran: I’m gonna make sure, also, this whole… this whole group is in Omni as well. I think this is the crew that really, yeah, I just want to really stress test and find…
359 00:48:17.680 ⇒ 00:48:28.710 Uttam Kumaran: And I think, Andy, for you, Steve, to press on the AI tool, I… the dashboarding is something that, you know, we’ve done for, you know, our whole careers.
360 00:48:28.710 ⇒ 00:48:45.480 Uttam Kumaran: But the AI piece, to your point, is something that we need to have… this team needs to be really, really confident on. We’re going to be layering on more and more context, and so we expect it to get more accurate, but you’re right in that it’s very user-driven. And so, whether it’s, like, we need to enforce more input validation.
361 00:48:45.530 ⇒ 00:48:50.270 Uttam Kumaran: Whether we need to limit it to certain topics, totally on the table.
362 00:48:50.600 ⇒ 00:48:56.099 Andy Weist: Do we have a… Full sandbox or staging stack of…
363 00:48:56.270 ⇒ 00:48:59.090 Andy Weist: Snowflake and Omni, by chance. I don’t know, like.
364 00:48:59.090 ⇒ 00:49:00.220 Uttam Kumaran: Not Omni yet.
365 00:49:00.380 ⇒ 00:49:04.450 Andy Weist: Okay. Yeah. You said yet. Is that anticipated?
366 00:49:04.780 ⇒ 00:49:08.839 Uttam Kumaran: We will end up setting up, like, staging dashboards.
367 00:49:09.200 ⇒ 00:49:15.280 Uttam Kumaran: So that we can do, sort of… but Omni already… Omni has a process of, like.
368 00:49:15.860 ⇒ 00:49:24.409 Uttam Kumaran: editing and then publishing, but content is different than, like, topic development. So, for example, nothing can change an underlying, but you may change a chart.
369 00:49:24.560 ⇒ 00:49:34.340 Uttam Kumaran: they have version control for that. They also have version control for more Omni-inner workings, like model joins. So, yes, we will have a complete end-to-end
370 00:49:34.670 ⇒ 00:49:36.450 Uttam Kumaran: Staging and production.
371 00:49:36.790 ⇒ 00:49:42.190 Uttam Kumaran: you know, throughout the whole process, except for Polyatomic.
372 00:49:42.510 ⇒ 00:49:48.530 Andy Weist: I think I’m a little… it’s not a separate environment, though, in Omni. Like, a separate login, separate domain.
373 00:49:48.530 ⇒ 00:49:52.869 Uttam Kumaran: No, no, no, it’s not a separate instance. It is… it will just be…
374 00:49:53.370 ⇒ 00:49:53.950 Jason Wu: It’s changing.
375 00:49:53.950 ⇒ 00:49:57.280 Uttam Kumaran: there’s just multiple layers of things in Omni.
376 00:49:57.700 ⇒ 00:50:08.649 Uttam Kumaran: For example, do we want to have the exact same dashboard cloned staging, and production, which is, like, the bar chart? Or is it, like, do we want to have topics cloned either way? That’s…
377 00:50:08.650 ⇒ 00:50:19.610 Andy Weist: Like, if I wanted to push a day’s worth of data into Polytomic, into staging Snowflake, then be able to interact with Omni via CLI, and be like.
378 00:50:19.620 ⇒ 00:50:30.989 Andy Weist: let me prompt you with this, did I get the expected data out to do, like, a fully end-to-end test? Yeah. That is possible, because Ami will still have access to the staging layer in Snowflake.
379 00:50:30.990 ⇒ 00:50:34.710 Uttam Kumaran: For your user. Isolated. For your user, yeah, exactly.
380 00:50:35.100 ⇒ 00:50:36.330 Andy Weist: Okay, yeah, that sounds good.
381 00:50:36.330 ⇒ 00:50:43.770 Uttam Kumaran: There’s role-based access control at every level, but Omni in particular will use whatever your access is to then pass your queries through.
382 00:50:44.230 ⇒ 00:50:45.519 Andy Weist: Got it. Cool.
383 00:50:49.030 ⇒ 00:50:49.640 Uttam Kumaran: Okay.
384 00:50:50.120 ⇒ 00:50:50.870 Uttam Kumaran: Great.
385 00:50:52.040 ⇒ 00:50:53.070 Uttam Kumaran: I think that’s it.
386 00:50:53.310 ⇒ 00:50:54.480 Uttam Kumaran: Anything else?
387 00:50:55.370 ⇒ 00:50:56.260 Jason Wu: No.
388 00:50:56.260 ⇒ 00:50:57.260 Awaish Kumar: This… I…
389 00:50:58.040 ⇒ 00:51:08.840 Awaish Kumar: I just would love to ask about, like, I wanted to connect Omni with Snowflake for the version control thing, and I don’t have access to create, like, add deploy keys and…
390 00:51:09.100 ⇒ 00:51:18.269 Awaish Kumar: stuff like that in the GitHub repo. So, like, I might send you some keys, and maybe someone can help me with adding those to the GitHub repo.
391 00:51:19.370 ⇒ 00:51:20.279 Jason Wu: Okay, yeah.
392 00:51:20.770 ⇒ 00:51:26.999 Andy Weist: If it’s easier, we can grant you the right permissions to the GitHub repo, at least temporarily.
393 00:51:27.200 ⇒ 00:51:28.170 Andy Weist: Probably.
394 00:51:29.070 ⇒ 00:51:29.430 Awaish Kumar: Yeah.
395 00:51:29.430 ⇒ 00:51:33.840 Andy Weist: I feel like that’s easier than back and forth. I’ll take a look at the permissions.
396 00:51:33.970 ⇒ 00:51:36.540 Andy Weist: Steve, Jason, as long as that’s cool with you guys.
397 00:51:36.820 ⇒ 00:51:41.120 Jason Wu: Yeah, that’s cool with me, I was gonna say, if not, then you could always just send him via, like, a…
398 00:51:41.230 ⇒ 00:51:43.320 Jason Wu: 1Password, secure, notice.
399 00:51:43.320 ⇒ 00:51:44.060 Andy Weist: Yeah.
400 00:51:44.320 ⇒ 00:51:46.689 Andy Weist: It’s probably quicker to just grant you access.
401 00:51:47.910 ⇒ 00:51:50.170 Andy Weist: I’ll look at that. We’ll follow up async.
402 00:51:50.310 ⇒ 00:51:50.850 Uttam Kumaran: Okay.
403 00:51:51.010 ⇒ 00:51:52.130 Awaish Kumar: Alright, thank you.
404 00:51:53.590 ⇒ 00:51:54.180 Uttam Kumaran: Okay.
405 00:51:54.480 ⇒ 00:51:56.339 Uttam Kumaran: Thank you, everyone. Appreciate it.
406 00:51:57.200 ⇒ 00:51:58.700 Jason Wu: Thanks, Jim. See ya.
407 00:51:58.700 ⇒ 00:52:00.220 Uttam Kumaran: Talk to you soon. Bye.