Meeting Title: Plan Data Warehouse Setup Date: 2026-01-26 Meeting participants: Pranav Narahari, Casie Aviles, Samuel Roberts
WEBVTT
1 00:00:17.840 ⇒ 00:00:18.830 Pranav Narahari: Hey guys.
2 00:00:32.299 ⇒ 00:00:35.739 Samuel Roberts: Hey, sorry, Zoom’s being kind of slow on this thing right now.
3 00:00:36.290 ⇒ 00:00:37.790 Pranav Narahari: All good, all good.
4 00:00:39.000 ⇒ 00:00:39.630 Samuel Roberts: Alright.
5 00:00:41.050 ⇒ 00:00:42.640 Pranav Narahari: Yeah, so…
6 00:00:42.980 ⇒ 00:00:47.090 Pranav Narahari: Sam, yeah, I’ll let you take the lead. I think we just kind of need to discuss a little bit, just like…
7 00:00:47.270 ⇒ 00:00:58.930 Pranav Narahari: Kind of the… how we can make this data warehouse, like, setup kind of, like, more bite-sized, and we can all kind of contribute if needed, to kind of getting it set up by this week, so…
8 00:00:59.060 ⇒ 00:01:02.509 Pranav Narahari: Yeah, I have a few thoughts, but yeah, Sam, if you want to go ahead, go for it.
9 00:01:03.030 ⇒ 00:01:10.430 Samuel Roberts: Yeah, I mean, I think my main concerns here are… Airbyte cloud versus self-hosted.
10 00:01:10.580 ⇒ 00:01:16.099 Samuel Roberts: And then DuckDB on Mother Duck, or on a volume on Ritway.
11 00:01:16.220 ⇒ 00:01:19.849 Samuel Roberts: I’m kind of inclined to lead towards cloud and stuff.
12 00:01:20.120 ⇒ 00:01:22.029 Samuel Roberts: If it does what we need?
13 00:01:22.770 ⇒ 00:01:28.079 Samuel Roberts: But I don’t know for a fact, how the AirByte Cloud will handle, like.
14 00:01:28.270 ⇒ 00:01:36.490 Samuel Roberts: When you add a new brand, what do we have to do? Like, how can we add all 70 brands, how does it set all those up? So that’s kind of what I’m wondering most now.
15 00:01:36.980 ⇒ 00:01:38.070 Samuel Roberts: Excuse me.
16 00:01:39.680 ⇒ 00:01:40.490 Pranav Narahari: Gotcha.
17 00:01:40.780 ⇒ 00:01:46.200 Pranav Narahari: So, we have, two different, like, I guess,
18 00:01:46.490 ⇒ 00:01:54.810 Pranav Narahari: tasks that we have set up for… for the project, like, in the… in the Gantt. One of them is, like, the ETL tool and connector setup.
19 00:01:54.970 ⇒ 00:02:03.359 Pranav Narahari: Which I would say is more so, like, aligned with that air bite. And then… for the POC,
20 00:02:03.860 ⇒ 00:02:08.860 Pranav Narahari: like, milestone, that is what is, like, we should get the data warehouse set up by then.
21 00:02:09.050 ⇒ 00:02:15.519 Pranav Narahari: the ETL tool can actually go a little bit past, like, POC complete, and then probably be ready for, like, the MVP.
22 00:02:15.710 ⇒ 00:02:28.719 Pranav Narahari: So, I’m wondering, does it make sense to not do AirByte right now, and set things up for, you know, just using Mother Duck or DuckDB?
23 00:02:30.000 ⇒ 00:02:31.340 Pranav Narahari: And then…
24 00:02:31.540 ⇒ 00:02:38.190 Pranav Narahari: Or will that cause, like, a lot of extra work, and it makes more sense to just implement and set up, like, AirByte now?
25 00:02:39.140 ⇒ 00:02:43.189 Samuel Roberts: Yeah, I think part of it depends on the schema from AirByte.
26 00:02:44.210 ⇒ 00:02:45.859 Pranav Narahari: That’s kind of the…
27 00:02:46.080 ⇒ 00:02:51.649 Samuel Roberts: Like, we don’t want to map everything to a schema we defined from Shopify versus what AirByte does.
28 00:02:51.760 ⇒ 00:02:59.899 Samuel Roberts: Which hopefully isn’t all that different, but I think it’s probably worth getting air by working for, like, a single Shopify connection, and then we can continue exploring how to…
29 00:03:00.160 ⇒ 00:03:04.600 Samuel Roberts: Expand that out to other brands.
30 00:03:04.800 ⇒ 00:03:06.639 Samuel Roberts: And other data sources.
31 00:03:08.400 ⇒ 00:03:12.590 Samuel Roberts: Like, unless we’re gonna go nowhere bite, I don’t think it’s worth skipping over it now.
32 00:03:14.670 ⇒ 00:03:15.480 Pranav Narahari: Gotcha.
33 00:03:15.610 ⇒ 00:03:30.349 Pranav Narahari: Yeah, so, for this POC, right, so, like, how… how the end state should look is, basically, we already have the data for, Shopify right now, for the last 2 years for Newton, I think, or… I think that was the… the…
34 00:03:30.350 ⇒ 00:03:32.529 Samuel Roberts: That’s what that’s… that’s what I… yeah, that’s what I got out of there.
35 00:03:32.530 ⇒ 00:03:36.309 Pranav Narahari: Right? Yeah, so we have the last two years’ data for Newton, so…
36 00:03:37.030 ⇒ 00:03:43.410 Pranav Narahari: Since we already have that data, it’s essentially just, like, putting that in the data warehouse, and then…
37 00:03:43.670 ⇒ 00:03:54.320 Pranav Narahari: having that… then… like, having that data then be visible within the Stitch platform.
38 00:03:54.470 ⇒ 00:03:59.129 Pranav Narahari: So the whole ETL setup, we actually don’t need to write any, like.
39 00:03:59.500 ⇒ 00:04:05.739 Pranav Narahari: custom code for even right now, because it’s done pretty manually, and pretty, like…
40 00:04:05.990 ⇒ 00:04:11.449 Pranav Narahari: you know, it’s not like a… like an automated process right now, so…
41 00:04:12.970 ⇒ 00:04:20.980 Pranav Narahari: you know, at some point, we will have to do that. I’m wondering if… if it makes sense to just, like, let’s just tackle, like, the data warehouse setup right now, and then, like…
42 00:04:21.230 ⇒ 00:04:26.320 Pranav Narahari: We don’t even need to make a decision on air bite or anything until next week.
43 00:04:27.940 ⇒ 00:04:32.120 Samuel Roberts: I see what you’re saying. Yeah. So I guess my question becomes then is…
44 00:04:32.330 ⇒ 00:04:40.459 Samuel Roberts: what does the data warehouse look like if we’re not mapping to the schema that AirByte is gonna map to? Like, just what we have, and then we’ll have to remap that later to the…
45 00:04:40.640 ⇒ 00:04:41.540 Samuel Roberts: Sun End?
46 00:04:43.170 ⇒ 00:04:45.749 Pranav Narahari: Gotcha, yeah. That’s…
47 00:04:45.750 ⇒ 00:04:49.730 Samuel Roberts: That’s fine, I mean, we can do that, I just don’t know… you know…
48 00:04:49.730 ⇒ 00:04:53.810 Pranav Narahari: I see your point, like, I don’t… like, my whole thing is, like, I don’t want to do…
49 00:04:53.970 ⇒ 00:05:00.700 Pranav Narahari: Extra work, you know, like… So… Interesting.
50 00:05:00.700 ⇒ 00:05:09.029 Samuel Roberts: Yeah, I mean, I think… I think what we can do, probably, in the immediate future, is probably get that Shopify data into DuckDB.
51 00:05:09.200 ⇒ 00:05:10.220 Pranav Narahari: Yup.
52 00:05:10.400 ⇒ 00:05:24.079 Samuel Roberts: we could probably do that right on railway, or run it locally for now, if you’re still working on the, you know, if you want to figure out the best way to talk to that. I don’t think it’s too much of a waste of time to get that working,
53 00:05:24.770 ⇒ 00:05:31.049 Samuel Roberts: The question becomes, like, well, yeah, what do we have to change once we move to, like, the air byte syncing?
54 00:05:32.570 ⇒ 00:05:33.250 Pranav Narahari: Right.
55 00:05:33.540 ⇒ 00:05:38.129 Samuel Roberts: So it’s probably worth a little bit of a dive into AirByte.
56 00:05:38.240 ⇒ 00:05:39.260 Samuel Roberts: Burst?
57 00:05:39.590 ⇒ 00:05:41.260 Samuel Roberts: Or in parallel, at least.
58 00:05:41.550 ⇒ 00:05:42.600 Samuel Roberts: Okay.
59 00:05:43.370 ⇒ 00:05:45.040 Pranav Narahari: I would say…
60 00:05:48.820 ⇒ 00:05:55.449 Samuel Roberts: Yeah, I think that makes sense. We have the Shopify data, so we can… I mean, you’re already… I mean, where are we with the, like, front-end…
61 00:05:55.710 ⇒ 00:05:57.530 Samuel Roberts: forecasting UI.
62 00:05:57.890 ⇒ 00:06:03.020 Pranav Narahari: Yeah, that’s just kind of on pause for right now while I’m doing the report stuff.
63 00:06:03.080 ⇒ 00:06:13.540 Samuel Roberts: Got it. Okay, so then, that’s fine. I just wasn’t sure if it was, like, we’re just ready to plug in data, in which case we could just, you know, pretty easily plug in the Shopify data. And if not, I think let’s spend some time on AirByte first.
64 00:06:16.270 ⇒ 00:06:21.069 Samuel Roberts: While you’re still sorting out the reports, maybe, because even that will help with reports, potentially, too, so…
65 00:06:21.070 ⇒ 00:06:22.470 Pranav Narahari: Right, yeah, I definitely would.
66 00:06:25.000 ⇒ 00:06:26.190 Samuel Roberts: So…
67 00:06:29.370 ⇒ 00:06:35.939 Samuel Roberts: Sorry, I got my kid here, he’s… I’m trying to keep him entertained with the little toys, and he’s making noise, I don’t know if you guys are hearing it.
68 00:06:37.750 ⇒ 00:06:45.189 Samuel Roberts: Let’s see… Yeah, I think spiking on air bite is probably a good idea.
69 00:06:46.640 ⇒ 00:06:47.300 Pranav Narahari: Okay.
70 00:06:47.910 ⇒ 00:06:49.950 Samuel Roberts: Just to make, like, a plan of attack.
71 00:06:50.580 ⇒ 00:06:53.220 Samuel Roberts: Yeah. I think
72 00:06:58.050 ⇒ 00:06:59.149 Samuel Roberts: Yeah, I think just…
73 00:06:59.500 ⇒ 00:07:06.920 Samuel Roberts: even just setting that up to stink Newton, to Mother Duck or something is probably the best Quick win.
74 00:07:12.290 ⇒ 00:07:13.250 Samuel Roberts: Yeah.
75 00:07:17.970 ⇒ 00:07:20.109 Samuel Roberts: Sorry, my brain’s just trying to think through, like…
76 00:07:21.270 ⇒ 00:07:24.590 Samuel Roberts: Maybe it’s even worth digging into the docs right now and just seeing
77 00:07:28.630 ⇒ 00:07:30.600 Samuel Roberts: Casey, do you have any experience with AirByte?
78 00:07:32.720 ⇒ 00:07:36.819 Casie Aviles: No, but I can spike on this,
79 00:07:37.390 ⇒ 00:07:41.480 Casie Aviles: I’ve… I’ve connected to Mother Duck before.
80 00:07:42.690 ⇒ 00:07:46.209 Samuel Roberts: Yeah, okay. I haven’t used AirBytes specifically.
81 00:07:46.820 ⇒ 00:07:50.540 Samuel Roberts: Okay, well, let’s… let’s do a quick sanity check here.
82 00:07:51.070 ⇒ 00:07:52.160 Samuel Roberts: -
83 00:07:52.620 ⇒ 00:08:00.060 Samuel Roberts: I’m just opening up AirByte just to check the, like… Docs and stuff.
84 00:08:01.150 ⇒ 00:08:05.149 Samuel Roberts: Individuals and small teams… oh, right, their pricing is not helpful here, is it?
85 00:08:09.320 ⇒ 00:08:12.940 Casie Aviles: I believe I have, like, a free trial, Airbus, but…
86 00:08:13.350 ⇒ 00:08:14.010 Samuel Roberts: Yeah.
87 00:08:14.010 ⇒ 00:08:16.590 Casie Aviles: have one, I think they can add us there.
88 00:08:17.260 ⇒ 00:08:19.410 Samuel Roberts: I don’t think they do, but they may want to.
89 00:08:27.280 ⇒ 00:08:30.870 Samuel Roberts: that… the plan, then.
90 00:08:32.860 ⇒ 00:08:36.579 Samuel Roberts: No, I still want to dig into the docs. I don’t feel comfortable not knowing exactly…
91 00:08:36.789 ⇒ 00:08:43.929 Samuel Roberts: how this is all gonna work yet, so let me go to… Shopify…
92 00:08:47.170 ⇒ 00:08:52.800 Samuel Roberts: I would share my screen, but it’s… it’s crawling right now anyways, can’t even get to what I’m trying to do right now.
93 00:08:52.960 ⇒ 00:08:56.910 Samuel Roberts: Alright, hold on.
94 00:08:59.310 ⇒ 00:09:04.830 Casie Aviles: Oh, I can share what they need me to share?
95 00:09:07.820 ⇒ 00:09:11.689 Samuel Roberts: Wait, hundreds of connector? The shop? Oh, that’s, that’s the…
96 00:09:12.970 ⇒ 00:09:15.519 Samuel Roberts: I was just looking at the Airbite Connectors page.
97 00:09:19.430 ⇒ 00:09:21.980 Samuel Roberts: I don’t even see Shopify on here, what am I missing?
98 00:09:25.060 ⇒ 00:09:26.219 Casie Aviles: Oh, is it this one?
99 00:09:28.290 ⇒ 00:09:31.859 Samuel Roberts: Yeah, why is that… why is that not showing up on earbite.com slash connectors?
100 00:09:33.390 ⇒ 00:09:34.050 Samuel Roberts: Huh.
101 00:09:35.830 ⇒ 00:09:38.880 Samuel Roberts: Docs, yeah, integrations, maybe that’s different than connectors?
102 00:09:42.170 ⇒ 00:09:45.779 Pranav Narahari: I’m able to see it on the airbite.com slash connectors.
103 00:09:46.370 ⇒ 00:09:48.680 Samuel Roberts: Okay, where are you seeing it? Like…
104 00:09:48.790 ⇒ 00:09:50.779 Samuel Roberts: Like, we just type in Shopify and it shows up?
105 00:09:51.560 ⇒ 00:09:54.680 Pranav Narahari: Yeah, yeah, and the filter’s on the left-hand side, yeah.
106 00:09:54.680 ⇒ 00:09:57.570 Samuel Roberts: Oh, there we go. For some reason it was… it was limited to, like, 100.
107 00:09:57.900 ⇒ 00:10:00.129 Pranav Narahari: Oh, yeah, it took a second to load, yeah.
108 00:10:00.130 ⇒ 00:10:02.679 Samuel Roberts: Yeah, I think I just started it too fast.
109 00:10:02.680 ⇒ 00:10:04.020 Pranav Narahari: Oh, gotcha, gotcha.
110 00:10:04.160 ⇒ 00:10:05.410 Samuel Roberts: That must be it, okay.
111 00:10:07.670 ⇒ 00:10:14.189 Samuel Roberts: There it is. Okay, phew. I was like, for a second there, I was panicking. But yeah, as long as you had a case, we were good. Okay.
112 00:10:16.330 ⇒ 00:10:21.440 Samuel Roberts: Yeah, I would say let’s just… let’s just get that working, or get,
113 00:10:23.630 ⇒ 00:10:25.840 Samuel Roberts: What does it need here?
114 00:10:28.230 ⇒ 00:10:31.920 Samuel Roberts: Yeah, let’s just try to connect it to Newton and sync it to Mother Duck, I guess.
115 00:10:32.160 ⇒ 00:10:34.649 Samuel Roberts: I don’t see a reason to not try that.
116 00:10:36.980 ⇒ 00:10:37.590 Pranav Narahari: Yeah.
117 00:10:42.360 ⇒ 00:10:47.160 Samuel Roberts: Yeah, I think that’s probably… I think that’s it, really. Let’s just see if that works, and… Mega.
118 00:10:47.500 ⇒ 00:10:54.530 Samuel Roberts: Decision once we either get that shrinking, or… Hit a roadblock.
119 00:10:55.280 ⇒ 00:11:00.340 Pranav Narahari: Okay, cool. Do we still need a spike on AirByte? I just created that ticket, but I can delete it.
120 00:11:00.340 ⇒ 00:11:03.810 Samuel Roberts: I think, I mean, I just call it, like, that’s probably just it, like, let’s just get Airbite.
121 00:11:04.030 ⇒ 00:11:04.480 Pranav Narahari: Yeah.
122 00:11:04.480 ⇒ 00:11:06.640 Samuel Roberts: do a dive into that. Spiking…
123 00:11:06.880 ⇒ 00:11:12.519 Samuel Roberts: Yeah, not like a research thing, just like a… let’s see if we can get air bytes syncing to ShopifyDataTomothe kind of thing.
124 00:11:12.730 ⇒ 00:11:14.139 Pranav Narahari: Yeah, I think that’s a good call.
125 00:11:14.140 ⇒ 00:11:14.969 Samuel Roberts: I think that’s it.
126 00:11:17.160 ⇒ 00:11:21.610 Pranav Narahari: Yeah. Okay, yeah, Casey, I see you’re in that task right now.
127 00:11:25.000 ⇒ 00:11:29.360 Samuel Roberts: Yeah, I mean, what is it… how does it… how are you guys looking this week with that? Like…
128 00:11:29.540 ⇒ 00:11:36.019 Samuel Roberts: Is that… something you can grab, Casey, or.
129 00:11:36.410 ⇒ 00:11:37.819 Samuel Roberts: take a stab at it, I just…
130 00:11:38.340 ⇒ 00:11:40.629 Samuel Roberts: Don’t want to overload you with other stuff, too.
131 00:11:41.260 ⇒ 00:11:44.290 Casie Aviles: Yeah, I won’t be doing much on…
132 00:11:44.520 ⇒ 00:11:48.649 Casie Aviles: Eden this week, I believe, so I think I should have at least time.
133 00:11:48.950 ⇒ 00:11:50.750 Casie Aviles: for Lilo and ABCs.
134 00:11:51.210 ⇒ 00:11:54.210 Samuel Roberts: Okay, cool. Yeah, I would say,
135 00:11:54.360 ⇒ 00:11:58.340 Samuel Roberts: Set up a mother duck with the…
136 00:11:58.970 ⇒ 00:12:02.290 Samuel Roberts: the, what is it, spitch at Lilo Social email?
137 00:12:05.370 ⇒ 00:12:22.889 Samuel Roberts: And, same with Airbyte, and then use the Newton creds to see if we can sync that. And then, I guess the real question in my mind is gonna be doing this more programmatically with the API keys in the database, triggering Airbyte with those, are we gonna have to store them separately? Like, that’s where I’m, like, not 100% sure how this plays out.
138 00:12:25.160 ⇒ 00:12:26.330 Samuel Roberts: Does that make sense?
139 00:12:26.580 ⇒ 00:12:29.519 Pranav Narahari: Yeah, yeah, like, as they add more brands.
140 00:12:29.820 ⇒ 00:12:37.599 Samuel Roberts: Right, like, how do we add a new connection with a new schedule? How do we do that initial sync? How do we use the API keys that we’re storing
141 00:12:38.170 ⇒ 00:12:42.219 Samuel Roberts: you know, encrypted to trigger that.
142 00:12:43.580 ⇒ 00:12:53.280 Samuel Roberts: Make sure that everybody can handle all that. I’m hope… I mean, there’s got to be some kind of API we can use, I just don’t know exactly what that looks like, and if it’s different, self-hosted versus cloud.
143 00:12:55.030 ⇒ 00:12:57.459 Samuel Roberts: That’s kind of my big, big questions right now.
144 00:13:00.470 ⇒ 00:13:07.790 Pranav Narahari: Gotcha. Yep, that makes sense. That’s…
145 00:13:07.990 ⇒ 00:13:13.499 Pranav Narahari: a good thing for us to look into. We can look into that either now or after the POC.
146 00:13:16.720 ⇒ 00:13:21.309 Samuel Roberts: Oh, okay, so it is OAuth? What is the… sorry, Casey, can you pull that back up? I was just starting to look at that.
147 00:13:21.790 ⇒ 00:13:22.849 Casie Aviles: Oh, okay.
148 00:13:22.850 ⇒ 00:13:29.370 Samuel Roberts: Yeah, thank you. I wasn’t… we might as well just spend a little time in it now, just to make sure we can probably answer a couple of these questions pretty quickly, isn’t it?
149 00:13:29.370 ⇒ 00:13:29.990 Pranav Narahari: Yeah.
150 00:13:31.740 ⇒ 00:13:36.660 Samuel Roberts: Okay, so OAuth is an… Hmm.
151 00:13:37.990 ⇒ 00:13:41.410 Samuel Roberts: We have an API key, are we able to… does this have to be OAuth, you think?
152 00:13:46.080 ⇒ 00:13:50.719 Pranav Narahari: They all say optional, so we can probably just type in the access token and then the Shopify store.
153 00:13:51.380 ⇒ 00:13:53.329 Samuel Roberts: Yeah, maybe I’ll try that.
154 00:14:04.240 ⇒ 00:14:06.069 Pranav Narahari: It’s the bottom one.
155 00:14:06.070 ⇒ 00:14:12.309 Samuel Roberts: No, you can honestly just delete that middle one, I just… before I had confirmed the second one was good, I didn’t want to just erase the other one.
156 00:14:12.480 ⇒ 00:14:14.600 Samuel Roberts: But you can delete the one at any time.
157 00:14:22.550 ⇒ 00:14:24.369 Samuel Roberts: Let me see what happens here…
158 00:14:45.260 ⇒ 00:14:51.639 Samuel Roberts: I don’t know where these API keys came from, if it’s… Thick, okay, that’s good news.
159 00:14:54.920 ⇒ 00:14:56.550 Samuel Roberts: Okay, yeah.
160 00:15:01.560 ⇒ 00:15:02.920 Samuel Roberts: Is there a search?
161 00:15:03.760 ⇒ 00:15:05.909 Samuel Roberts: the top here, I think.
162 00:15:09.940 ⇒ 00:15:11.500 Samuel Roberts: Marketplace.
163 00:15:13.940 ⇒ 00:15:17.560 Casie Aviles: Mother ducks… wait, what’s the difference again? I forgot.
164 00:15:21.500 ⇒ 00:15:23.949 Samuel Roberts: So there… yeah, sorry, Mike, again.
165 00:15:25.520 ⇒ 00:15:28.410 Samuel Roberts: Same successor to other students.
166 00:15:32.750 ⇒ 00:15:33.530 Casie Aviles: Hmm.
167 00:15:33.530 ⇒ 00:15:36.109 Samuel Roberts: Yeah, I would go ahead, yeah, I would go ahead and set up a…
168 00:15:36.840 ⇒ 00:15:38.569 Samuel Roberts: I don’t think we have anything for them yet.
169 00:15:41.240 ⇒ 00:15:45.160 Samuel Roberts: I would just set up a new Mother Duck with their login.
170 00:16:30.740 ⇒ 00:16:33.909 Samuel Roberts: Yeah, I would change… this to Leela, too.
171 00:16:36.620 ⇒ 00:16:37.320 Samuel Roberts: Cool.
172 00:16:38.350 ⇒ 00:16:39.400 Samuel Roberts: Lappings.
173 00:16:45.330 ⇒ 00:16:46.050 Samuel Roberts: Cool.
174 00:16:46.380 ⇒ 00:16:48.829 Samuel Roberts: Maybe fill all this in.
175 00:17:07.450 ⇒ 00:17:08.569 Samuel Roberts: some more bumps.
176 00:17:09.140 ⇒ 00:17:09.710 Casie Aviles: Oh.
177 00:17:11.210 ⇒ 00:17:15.370 Samuel Roberts: Yeah, just let it… whatever. Okay. I don’t think this is really relevant to too much, but…
178 00:17:15.990 ⇒ 00:17:17.880 Samuel Roberts: Yeah, skip that for now, we can do it later.
179 00:17:32.330 ⇒ 00:17:33.739 Samuel Roberts: Tools, maybe?
180 00:17:43.660 ⇒ 00:17:45.290 Casie Aviles: Oh, yeah, great to open.
181 00:17:47.060 ⇒ 00:17:47.940 Samuel Roberts: Okay, cool.
182 00:17:48.520 ⇒ 00:17:49.160 Pranav Narahari: Nice.
183 00:18:27.800 ⇒ 00:18:31.850 Pranav Narahari: And Sam, you said Mother Duck is, like… Cloud-hosted, right?
184 00:18:32.500 ⇒ 00:18:33.710 Samuel Roberts: Yeah, so, I mean…
185 00:18:33.820 ⇒ 00:18:42.269 Samuel Roberts: My understanding of DuckDB is somewhat limited, but, it seems to be, like, a file, like SQLite, or SQLite.
186 00:18:43.470 ⇒ 00:18:50.189 Samuel Roberts: And I think this is… yeah, we’ve used this for some other clients. I ran some queries and stuff and dumped some data in there, so…
187 00:18:50.340 ⇒ 00:18:54.180 Samuel Roberts: We should be able to point our instance to it.
188 00:18:54.570 ⇒ 00:18:57.210 Samuel Roberts: And things should be pretty good.
189 00:18:57.470 ⇒ 00:18:59.280 Samuel Roberts: Maybe a little bit of setup there.
190 00:19:00.160 ⇒ 00:19:01.830 Pranav Narahari: Just config and railway.
191 00:19:02.440 ⇒ 00:19:04.020 Samuel Roberts: Yeah, I think so.
192 00:19:10.530 ⇒ 00:19:16.600 Samuel Roberts: Okay, cool. Configure connection, inspection to schema… yeah, so I think…
193 00:19:16.600 ⇒ 00:19:17.670 Pranav Narahari: Oh, nice, okay.
194 00:19:18.070 ⇒ 00:19:22.720 Samuel Roberts: I think this is gonna work. The question is long-term, like…
195 00:19:22.870 ⇒ 00:19:26.439 Samuel Roberts: Maybe we need to stop and think about this for a second, where this is about to dump the data.
196 00:19:26.670 ⇒ 00:19:27.510 Samuel Roberts: Right.
197 00:19:28.320 ⇒ 00:19:42.839 Samuel Roberts: so we’re gonna want… Some kind of separation with… The brands… and the… Sources, right?
198 00:19:47.240 ⇒ 00:19:47.680 Pranav Narahari: Right.
199 00:19:47.680 ⇒ 00:19:53.230 Samuel Roberts: So, I’m trying to think, like, is it every brand will add a new table to DuckDB, I guess?
200 00:19:53.740 ⇒ 00:19:54.779 Samuel Roberts: Or a new…
201 00:19:55.170 ⇒ 00:20:06.980 Samuel Roberts: series of tables? I’m not sure. This is where… this is where I need to do a little more research on my own to figure this out, but we can just play around and see what works, because we’re not… this isn’t live yet, you know, so… Right. Just a few things.
202 00:20:15.210 ⇒ 00:20:21.360 Samuel Roberts: The other question is, we can always ask someone to use more of this and just get a suggestion, maybe, but…
203 00:20:25.840 ⇒ 00:20:29.249 Pranav Narahari: Yeah, maybe this is where we can go back to OASH and talk about, like.
204 00:20:29.520 ⇒ 00:20:34.150 Pranav Narahari: data schema… data schema? For…
205 00:20:34.630 ⇒ 00:20:39.890 Pranav Narahari: for Mother Duck or DuckDB, how they’ve done it for past projects.
206 00:20:40.620 ⇒ 00:20:44.350 Samuel Roberts: Right, right, and specifically knowing that we’re adding more brands.
207 00:20:45.660 ⇒ 00:20:46.340 Pranav Narahari: Yeah.
208 00:20:47.130 ⇒ 00:20:49.589 Samuel Roberts: That will have several sources each over time.
209 00:20:49.590 ⇒ 00:20:50.200 Pranav Narahari: Yup.
210 00:20:51.370 ⇒ 00:20:51.960 Samuel Roberts: Okay.
211 00:20:52.150 ⇒ 00:20:58.179 Samuel Roberts: Could be worth also exploring the, like, API for AirByte.
212 00:20:59.430 ⇒ 00:21:05.429 Samuel Roberts: Sometime while this is ongoing, just to make sure where there is, like, a path forward for automating it a little bit.
213 00:21:05.770 ⇒ 00:21:06.410 Pranav Narahari: Right.
214 00:21:09.300 ⇒ 00:21:16.960 Samuel Roberts: But, I think, yeah, I think this is… Definitely, like… The best thing…
215 00:21:17.110 ⇒ 00:21:19.590 Samuel Roberts: to do right now. I’m just trying to think, like…
216 00:21:22.160 ⇒ 00:21:26.580 Samuel Roberts: We have a table per brand, probably, and then we have a… well, no, because we got… we need a… what is…
217 00:21:27.860 ⇒ 00:21:29.959 Samuel Roberts: Every brand will get a new…
218 00:21:30.860 ⇒ 00:21:34.060 Samuel Roberts: maybe? Or not schema, what’s the… yeah, schema.
219 00:21:34.250 ⇒ 00:21:37.279 Samuel Roberts: But that’s the PostgreSQL term, I don’t know what the term was here.
220 00:21:39.890 ⇒ 00:21:44.769 Samuel Roberts: Syncing from multiple sources, sync all streams and schema defined in the destination settings.
221 00:21:45.270 ⇒ 00:21:48.059 Samuel Roberts: Source defined. I don’t know there.
222 00:21:55.710 ⇒ 00:21:56.090 Casie Aviles: Yeah.
223 00:21:56.090 ⇒ 00:21:59.629 Samuel Roberts: Yeah, I might just have to be just digging in and playing around a little bit and seeing how things work.
224 00:22:00.410 ⇒ 00:22:02.510 Casie Aviles: Should I just, do a test sync?
225 00:22:04.700 ⇒ 00:22:07.810 Samuel Roberts: Yeah, see how that goes, because you can always just crash it if it’s…
226 00:22:11.990 ⇒ 00:22:13.030 Casie Aviles: Underwriting.
227 00:22:13.190 ⇒ 00:22:14.600 Casie Aviles: Looks like, you know…
228 00:22:20.420 ⇒ 00:22:25.439 Pranav Narahari: And did you specify what data, like, timeframe you’ll be pre-populating it with?
229 00:22:28.070 ⇒ 00:22:29.989 Casie Aviles: Oh, yeah, I don’t think I…
230 00:22:30.850 ⇒ 00:22:31.830 Samuel Roberts: Oh, yeah, it probably has a.
231 00:22:31.830 ⇒ 00:22:32.380 Casie Aviles: dear.
232 00:22:32.910 ⇒ 00:22:35.770 Pranav Narahari: I’m wondering what it’s even gonna… Pull in.
233 00:22:35.770 ⇒ 00:22:37.270 Samuel Roberts: How far back is it gonna go?
234 00:22:37.420 ⇒ 00:22:37.820 Pranav Narahari: Yeah.
235 00:22:37.820 ⇒ 00:22:39.330 Samuel Roberts: Go back to.
236 00:22:40.020 ⇒ 00:22:43.610 Pranav Narahari: I guess maybe every 24 hours, since we have it set for every 24 hours?
237 00:22:44.100 ⇒ 00:22:49.800 Samuel Roberts: Yeah, but what’s the kind of fill for the, the backfill user or something.
238 00:22:50.500 ⇒ 00:22:56.139 Pranav Narahari: Yeah, it’s probably a place to specify. I’m wondering, since we didn’t specify, it’s just gonna give the last 24 hours, but…
239 00:22:56.140 ⇒ 00:22:58.009 Samuel Roberts: That might be… that’s fine for now, we can…
240 00:22:58.010 ⇒ 00:22:58.549 Pranav Narahari: You can click on it.
241 00:22:58.550 ⇒ 00:23:00.440 Samuel Roberts: I just want to see it happen. Yeah, exactly.
242 00:23:00.440 ⇒ 00:23:00.990 Pranav Narahari: Yeah.
243 00:23:01.880 ⇒ 00:23:03.359 Pranav Narahari: Let’s see…
244 00:23:03.490 ⇒ 00:23:05.870 Samuel Roberts: Back so many…
245 00:23:08.560 ⇒ 00:23:09.160 Pranav Narahari: Yeah.
246 00:23:09.780 ⇒ 00:23:11.939 Pranav Narahari: Honestly, we’ll just see.
247 00:23:11.940 ⇒ 00:23:16.730 Samuel Roberts: Yeah, exactly, that’s kind of… this is kind of my thought, is let’s just… let’s just test it out, like, and see what’s gonna happen.
248 00:23:16.960 ⇒ 00:23:17.580 Pranav Narahari: Yep.
249 00:23:19.910 ⇒ 00:23:25.849 Pranav Narahari: Because even if there isn’t a way to specify this, there’s probably a way to just manually upload,
250 00:23:26.000 ⇒ 00:23:27.390 Pranav Narahari: Data as well.
251 00:23:28.380 ⇒ 00:23:34.819 Pranav Narahari: And so, we probably just need to run something that runs that per brand in the beginning, and then after that…
252 00:23:35.690 ⇒ 00:23:39.520 Pranav Narahari: Just having it set up where it pulls in data every 24 hours will be fine.
253 00:23:40.390 ⇒ 00:23:41.930 Samuel Roberts: Yeah, yeah, I think that’s perfect.
254 00:23:46.000 ⇒ 00:23:46.590 Pranav Narahari: One straight peak.
255 00:23:46.590 ⇒ 00:23:47.550 Casie Aviles: sometime.
256 00:23:47.580 ⇒ 00:23:48.140 Pranav Narahari: Yeah.
257 00:23:48.140 ⇒ 00:23:49.700 Samuel Roberts: Yeah, you may just have to let it go.
258 00:23:50.080 ⇒ 00:23:56.380 Samuel Roberts: And, and let it… It’s weird not knowing what it’s thinking.
259 00:23:56.550 ⇒ 00:24:02.990 Pranav Narahari: So how this works is it basically just pulls in all the data. Like, we didn’t… really specify…
260 00:24:02.990 ⇒ 00:24:05.399 Samuel Roberts: Yeah, there was a whole schema, if you click on the schema…
261 00:24:05.400 ⇒ 00:24:07.119 Pranav Narahari: Oh yeah, we just didn’t uncheck anything, right?
262 00:24:07.120 ⇒ 00:24:12.189 Samuel Roberts: We just said, took everything, so… Yeah. Which I think is fine for now.
263 00:24:12.190 ⇒ 00:24:15.310 Pranav Narahari: Yeah, we’ll change in the future, there’s probably a lot of stuff here that we don’t need, but…
264 00:24:15.310 ⇒ 00:24:21.590 Samuel Roberts: Yeah, exactly. You know, it depends, but then if they, you know, want more stuff, who knows what…
265 00:24:22.100 ⇒ 00:24:25.210 Samuel Roberts: Yeah, like, that’s pro- I think the defaults here are probably okay.
266 00:24:25.410 ⇒ 00:24:30.949 Samuel Roberts: Yeah. That’s fine for now. Like I said, we’re just trying to make sure that this works the way we think it does. Yeah.
267 00:24:33.420 ⇒ 00:24:34.160 Samuel Roberts: Cool.
268 00:24:35.450 ⇒ 00:24:35.910 Pranav Narahari: Okay.
269 00:24:35.910 ⇒ 00:24:39.180 Samuel Roberts: Yeah, I would say let this run and see what shows up in Mother Duck.
270 00:24:39.760 ⇒ 00:24:40.350 Pranav Narahari: Yeah.
271 00:24:40.460 ⇒ 00:24:45.450 Pranav Narahari: Nice. Okay. This looks like it’s gonna, fingers crossed, be a lot smoother than we thought.
272 00:24:47.310 ⇒ 00:24:51.270 Samuel Roberts: Yeah, yeah. Alright, cool. Yeah, keep… keep us…
273 00:24:51.860 ⇒ 00:24:56.360 Samuel Roberts: Up to date with that, when that’s done syncing, or when there’s progress shown on a mother duck?
274 00:24:57.820 ⇒ 00:24:59.200 Samuel Roberts: And then…
275 00:25:00.480 ⇒ 00:25:06.190 Samuel Roberts: Yeah, we can run from there with getting it into the actual warehouse, and running some models on it, and getting the…
276 00:25:07.610 ⇒ 00:25:10.390 Samuel Roberts: The cohorts and everything run.
277 00:25:11.080 ⇒ 00:25:14.089 Samuel Roberts: I think probably all just doing SQL on top of it.
278 00:25:14.600 ⇒ 00:25:15.400 Pranav Narahari: Dope.
279 00:25:15.910 ⇒ 00:25:16.550 Samuel Roberts: Okay.
280 00:25:16.950 ⇒ 00:25:21.450 Samuel Roberts: I think this is good progress for now. Again, we may have to loop in Someone more data-y.
281 00:25:21.580 ⇒ 00:25:22.490 Samuel Roberts: 2…
282 00:25:22.770 ⇒ 00:25:31.149 Samuel Roberts: answer some clarifying questions, or point us in another direction, or not another direction, that came out wrong. Point us in the right direction when we hit a certain point, but
283 00:25:31.470 ⇒ 00:25:32.690 Samuel Roberts: I’m feeling pretty good about that.
284 00:25:34.640 ⇒ 00:25:41.729 Pranav Narahari: Casey, I’m gonna add you to just the ETL tool connector setup. I feel like you’re kind of already working on that.
285 00:25:41.730 ⇒ 00:25:42.310 Casie Aviles: Sure.
286 00:25:42.560 ⇒ 00:25:46.700 Pranav Narahari: Yeah. So, yeah, just let me know if, you need any help on that.
287 00:25:46.860 ⇒ 00:25:48.929 Pranav Narahari: Yeah, for me,
288 00:25:49.610 ⇒ 00:26:04.889 Pranav Narahari: I’m gonna look into those two other PRs that we talked, or, Vibe coding branches that, Bobby set up last week. The meta dashboard one, maybe we can even get a head start on that.
289 00:26:05.170 ⇒ 00:26:05.890 Samuel Roberts: Yeah, maybe.
290 00:26:05.890 ⇒ 00:26:09.210 Pranav Narahari: I kind of want to push that, because that’s more of, like, a…
291 00:26:09.610 ⇒ 00:26:19.979 Pranav Narahari: that’s not really part of this POC, so actually, instead, what I’m gonna just focus on after I finish up this reporting thing is just, just a basic forecast and pacing model.
292 00:26:20.570 ⇒ 00:26:21.120 Samuel Roberts: Okay.
293 00:26:21.410 ⇒ 00:26:22.020 Pranav Narahari: Yeah.
294 00:26:22.270 ⇒ 00:26:23.950 Pranav Narahari: I’ll let you know if I find something in there.
295 00:26:23.950 ⇒ 00:26:28.539 Samuel Roberts: Yeah, take… and take a look at that PR, just so we can get the clock scales in, and make sure
296 00:26:28.870 ⇒ 00:26:34.530 Samuel Roberts: until it works. I’m a little worried, because it added this… or it was using code execution a lot, and I want to make sure it doesn’t…
297 00:26:34.990 ⇒ 00:26:36.080 Samuel Roberts: go crazy.
298 00:26:36.500 ⇒ 00:26:39.509 Pranav Narahari: Right, okay, okay. I’ll, yeah, I’ll test locally, I’ll do…
299 00:26:39.690 ⇒ 00:26:42.970 Pranav Narahari: Just, like, some QA stuff, and then, let you know how it goes.
300 00:26:43.350 ⇒ 00:26:44.380 Samuel Roberts: Great, thank you.
301 00:26:44.910 ⇒ 00:26:45.570 Pranav Narahari: Totally.
302 00:26:47.100 ⇒ 00:26:48.729 Samuel Roberts: Alright, anything else?
303 00:26:52.820 ⇒ 00:26:53.610 Samuel Roberts: Alright.
304 00:26:53.800 ⇒ 00:26:54.240 Pranav Narahari: I think we’re good.
305 00:26:54.650 ⇒ 00:27:01.660 Samuel Roberts: Yeah, alright, just keep us updated, Casey, when that finishes or makes some progress, and we’ll run from there. The mother duck is the Lilo…
306 00:27:01.950 ⇒ 00:27:08.679 Samuel Roberts: the Stitch at Lilo app, or stitch at lilosocial.com, and then for the AirByte, what is that set up under?
307 00:27:10.130 ⇒ 00:27:12.029 Casie Aviles: Oh, that was… that was my account.
308 00:27:12.560 ⇒ 00:27:14.519 Samuel Roberts: That was a free trial account.
309 00:27:14.770 ⇒ 00:27:20.699 Samuel Roberts: Okay, that’s fine. I just didn’t know, yeah, then that’s fine for now. We’ll figure it out when we need to. If we need to move it or something, it shouldn’t be a problem.
310 00:27:21.000 ⇒ 00:27:25.650 Pranav Narahari: Yeah, basically if this goes smoothly, I think we’re… we’re good to go to just kind of…
311 00:27:25.790 ⇒ 00:27:33.370 Pranav Narahari: probably bring it up with them, and then just talk about, like, yeah, we’ve kind of decided on this, we’ve set up, like, a POC,
312 00:27:33.650 ⇒ 00:27:41.340 Pranav Narahari: On what this whole connection end-to-end would look like, and so now, it’s, like, balls in their court to just, like, get the accounts.
313 00:27:42.580 ⇒ 00:27:43.430 Casie Aviles: Okay.
314 00:27:45.810 ⇒ 00:27:46.570 Casie Aviles: Cool, guys.
315 00:27:46.570 ⇒ 00:27:48.680 Samuel Roberts: Alrighty. Sounds good. Thank you guys.
316 00:27:48.950 ⇒ 00:27:49.460 Pranav Narahari: Yup.
317 00:27:49.980 ⇒ 00:27:51.160 Samuel Roberts: Alrighty, bye.
318 00:27:51.160 ⇒ 00:27:51.910 Casie Aviles: Thank you.