Meeting Title: DE-AE Standup Date: 2025-12-23 Meeting participants: Awaish Kumar, Mustafa Raja, Elizah Joy, Demilade, Uttam Kumaran, Ashwini Sharma
WEBVTT
1 00:01:43.300 ⇒ 00:01:44.120 Awaish Kumar: Hello?
2 00:01:45.700 ⇒ 00:01:47.050 Mustafa Raja: Hey, how are you?
3 00:01:48.130 ⇒ 00:01:49.220 Awaish Kumar: Good, how about you?
4 00:01:49.610 ⇒ 00:01:50.709 Mustafa Raja: Yeah, doing good.
5 00:01:53.000 ⇒ 00:01:54.480 Awaish Kumar: Do they have a camera?
6 00:01:55.680 ⇒ 00:01:57.749 Mustafa Raja: Oh, yeah, I got a new laptop.
7 00:01:58.000 ⇒ 00:01:59.550 Mustafa Raja: So, now I do have…
8 00:02:08.190 ⇒ 00:02:12.419 Mustafa Raja: Yeah, I asked Tim to, you know, go ahead, create a new…
9 00:02:12.650 ⇒ 00:02:16.660 Mustafa Raja: service account, and then, add the BigQuery roles.
10 00:02:17.830 ⇒ 00:02:20.810 Mustafa Raja: I thought we had that service account created somewhere.
11 00:02:21.550 ⇒ 00:02:24.749 Awaish Kumar: So we didn’t have the permission to create this account?
12 00:02:25.160 ⇒ 00:02:32.069 Mustafa Raja: Yeah, so, the Brain Forge, at GoAntETA only has editor access with the project.
13 00:02:32.380 ⇒ 00:02:38.320 Mustafa Raja: So yeah, this is going to be, like, we’ll have to go and ask Tim for anything we want to add.
14 00:02:39.060 ⇒ 00:02:39.760 Awaish Kumar: Okay.
15 00:02:43.020 ⇒ 00:02:45.599 Awaish Kumar: Okay, we got this…
16 00:02:45.910 ⇒ 00:02:55.980 Awaish Kumar: Yeah, BigQuery setup, I don’t know what the next step is, like, we… this is the basic, like, account setup. Now we need to do… maybe we want to configure BigQuery for dbt.
17 00:02:56.240 ⇒ 00:02:57.300 Awaish Kumar: Or…
18 00:02:57.300 ⇒ 00:03:08.120 Mustafa Raja: Yeah, yeah, yeah, yeah. Yeah, that is in the… that is in the Gantt chart. Now, once this is good, what we need to do is we need to set up a… set up dbt with BigQuery.
19 00:03:08.560 ⇒ 00:03:12.999 Mustafa Raja: Let me… let me open up and see what else is related to BigQuery.
20 00:03:16.040 ⇒ 00:03:19.040 Awaish Kumar: Basically, what do we need to do for… for…
21 00:03:19.960 ⇒ 00:03:26.439 Awaish Kumar: We see is that we, like, and that document is, like, just for account setting up, setup.
22 00:03:26.830 ⇒ 00:03:27.440 Mustafa Raja: Yeah.
23 00:03:27.440 ⇒ 00:03:38.680 Awaish Kumar: Another one which basically can help you create, like, the databases, service count, These different schemas.
24 00:03:41.150 ⇒ 00:03:44.259 Mustafa Raja: Yeah, so… So yeah, this is almost done.
25 00:03:44.460 ⇒ 00:03:46.580 Mustafa Raja: In a BigQuery.
26 00:03:46.710 ⇒ 00:03:53.040 Mustafa Raja: And then, we need to initialize dbt on BigQuery, and then these two migrations is what we need to do.
27 00:03:56.840 ⇒ 00:03:57.670 Awaish Kumar: Okay.
28 00:03:58.310 ⇒ 00:04:00.990 Awaish Kumar: And is that plan? Like…
29 00:04:04.470 ⇒ 00:04:08.359 Mustafa Raja: Yeah, this is… this is the… this is the… this is the plan. I’ll just need some…
30 00:04:09.350 ⇒ 00:04:13.859 Awaish Kumar: But this has to be done by, like, mid-January or end of January.
31 00:04:14.630 ⇒ 00:04:15.260 Mustafa Raja: Yep.
32 00:04:15.680 ⇒ 00:04:16.740 Mustafa Raja: Mid-January.
33 00:04:23.820 ⇒ 00:04:29.799 Uttam Kumaran: Yeah, so I need dbt, and BigQuery for some of the analysis I’m doing.
34 00:04:29.920 ⇒ 00:04:32.979 Uttam Kumaran: for the discovery work stream, so that’s mainly what I need.
35 00:04:33.510 ⇒ 00:04:35.829 Uttam Kumaran: The warehouse stood up for.
36 00:04:37.530 ⇒ 00:04:39.459 Uttam Kumaran: So I’m gonna be dropping some data in there.
37 00:04:39.680 ⇒ 00:04:43.869 Awaish Kumar: who, like, the… like, who set up the account in BigQuery?
38 00:04:44.090 ⇒ 00:04:48.540 Awaish Kumar: account yesterday, and I asked for a service account key as well.
39 00:04:48.820 ⇒ 00:04:54.980 Awaish Kumar: So… After that, we can start with the initialization, opportunity.
40 00:04:56.950 ⇒ 00:04:59.699 Awaish Kumar: Okay. We don’t have a service count yet, right?
41 00:05:00.190 ⇒ 00:05:08.659 Mustafa Raja: Yeah, yeah. We’re still waiting on the service account, but that should be… that should be done today. I’ve pinged him regarding that.
42 00:05:08.870 ⇒ 00:05:11.669 Mustafa Raja: So, once we have that, we can move over to this.
43 00:05:13.010 ⇒ 00:05:15.720 Mustafa Raja: in initializing dbt on BigQuery.
44 00:05:16.800 ⇒ 00:05:17.490 Awaish Kumar: Okay.
45 00:05:18.340 ⇒ 00:05:23.269 Awaish Kumar: Okay, once you get that, like, service count key, let me know, and we can.
46 00:05:23.270 ⇒ 00:05:23.620 Mustafa Raja: Yeah.
47 00:05:23.850 ⇒ 00:05:24.540 Awaish Kumar: Yeah.
48 00:05:24.540 ⇒ 00:05:25.980 Mustafa Raja: Yeah, of course.
49 00:05:26.930 ⇒ 00:05:32.520 Awaish Kumar: Yeah, I can… Okay, that’s for ABC, and I can share my screen. Yeah.
50 00:05:33.270 ⇒ 00:05:36.450 Awaish Kumar: And we can start with… Matrix Boom.
51 00:05:36.930 ⇒ 00:05:40.290 Awaish Kumar: So, is there anything we wanna share by the…
52 00:05:40.830 ⇒ 00:05:47.890 Awaish Kumar: by tomorrow, end of day, like, what we have done so far on discovering Office Spins API,
53 00:05:50.930 ⇒ 00:05:51.960 Awaish Kumar: Ashwini?
54 00:05:52.810 ⇒ 00:05:59.610 Ashwini Sharma: No, I don’t really have an update on that. I still could not see their code in the repo.
55 00:06:00.780 ⇒ 00:06:03.259 Ashwini Sharma: I don’t know if you guys can see it, because…
56 00:06:05.890 ⇒ 00:06:09.200 Awaish Kumar: No, yeah, yeah, the repo that he mentioned, where…
57 00:06:09.200 ⇒ 00:06:13.740 Ashwini Sharma: That contained, those, prefect files, they are not there.
58 00:06:14.240 ⇒ 00:06:18.729 Ashwini Sharma: And I’ve gone through the GraphQL API, right?
59 00:06:18.880 ⇒ 00:06:24.100 Ashwini Sharma: But yeah, until we start writing pipelines, I don’t think I can comment anything on that.
60 00:06:26.230 ⇒ 00:06:27.050 Awaish Kumar: Okay.
61 00:06:28.550 ⇒ 00:06:34.080 Awaish Kumar: Okay, yeah, we don’t know, like, we haven’t seen the… Prefect.
62 00:06:34.220 ⇒ 00:06:43.070 Awaish Kumar: But, like, the… like, you can work on discovery of API, like, what the plan for how we are going to implement the…
63 00:06:43.270 ⇒ 00:06:49.850 Awaish Kumar: extraction from Spins API. Like, their preferred code does not have anything for spins.
64 00:06:51.130 ⇒ 00:06:52.030 Awaish Kumar: Once you.
65 00:06:52.030 ⇒ 00:06:55.849 Ashwini Sharma: That’s fine. I just want to see the structure that they are following in Prefecture.
66 00:06:56.110 ⇒ 00:07:08.719 Awaish Kumar: That is not a blocker for disk… for understanding this PINS API. You can go in and read the documentation and figure out what API endpoints to hit, and how the presentation works, and things like that.
67 00:07:09.730 ⇒ 00:07:11.960 Ashwini Sharma: Yeah, sure, I think I can do that.
68 00:07:15.290 ⇒ 00:07:18.679 Awaish Kumar: Yes, Navalare, you wanna say something?
69 00:07:18.910 ⇒ 00:07:25.050 Demilade: Nothing, at least from my end, like, the perspective of, like,
70 00:07:25.270 ⇒ 00:07:32.849 Demilade: the audit. I still will need access to the GitHub, and I guess that has still not been done, so we’ll just follow up on that.
71 00:07:33.070 ⇒ 00:07:37.910 Demilade: By the time you’d be able to see what was done, the different models that exist.
72 00:07:39.580 ⇒ 00:07:52.730 Uttam Kumaran: Yeah, I wouldn’t… I wouldn’t, like… if they don’t give us access until later today, I wouldn’t worry too much about sending, like, frantically trying to get an update out. This is on them. They’re kind of screwing up this, like, access thing.
73 00:07:52.810 ⇒ 00:08:04.579 Uttam Kumaran: I kind of… I think I know why, but regardless, it’s okay. So, if they don’t give us access to GitHub, we can’t look at any of the prefect stuff, we can’t even look at any of the dbt stuff.
74 00:08:04.780 ⇒ 00:08:10.390 Uttam Kumaran: The only thing I think probably we can do is just investigate the Spins API.
75 00:08:10.690 ⇒ 00:08:13.680 Uttam Kumaran: Further, but yeah.
76 00:08:14.360 ⇒ 00:08:22.639 Demilade: Yeah, I guess the two main things to look at, even Redshift, I’ll need to confirm, we were having issues looking at the Redshift cluster information as well.
77 00:08:24.290 ⇒ 00:08:24.810 Awaish Kumar: Love it, too.
78 00:08:24.810 ⇒ 00:08:28.009 Demilade: Michael said he was gonna work on it. I don’t know if he’s fixed that.
79 00:08:28.390 ⇒ 00:08:33.020 Demilade: So, again, it’s hard to look. Redshift, dbt.
80 00:08:33.450 ⇒ 00:08:35.529 Demilade: I mean, we have dbt access, then.
81 00:08:35.760 ⇒ 00:08:38.499 Demilade: But we don’t have, like, the GitHub access.
82 00:08:39.049 ⇒ 00:08:42.949 Demilade: So, our plan is to just poke around in dbt today, and just…
83 00:08:42.950 ⇒ 00:08:45.839 Uttam Kumaran: Poke around, and then if you find it, if you see anything.
84 00:08:45.940 ⇒ 00:08:49.440 Uttam Kumaran: Just send some questions, I think that’s more than reasonable, yeah.
85 00:08:49.670 ⇒ 00:08:54.110 Demilade: Okay, so yeah, so I’ll look through dbt today and send some questions to the team.
86 00:08:54.520 ⇒ 00:08:55.790 Demilade: for EOT.
87 00:08:58.500 ⇒ 00:09:04.380 Awaish Kumar: Okay, then, yeah, that’s the plan. Like, we will just look into dbt and the Swins API,
88 00:09:04.480 ⇒ 00:09:08.450 Awaish Kumar: And ask some questions, or whatever we figure out.
89 00:09:08.970 ⇒ 00:09:11.940 Awaish Kumar: I’ll buy tomorrow. Yeah, that’s all.
90 00:09:12.290 ⇒ 00:09:19.589 Awaish Kumar: Then for Hydra, I… yesterday, I worked mostly on Element.
91 00:09:19.730 ⇒ 00:09:29.520 Awaish Kumar: So today, I will be focusing on Hydra. For Hydra, what she wants. Basically, we are still having sync… sync issues from Polyatomic for Stripe.
92 00:09:29.820 ⇒ 00:09:32.719 Awaish Kumar: So I don’t know why it’s taking so long.
93 00:09:32.950 ⇒ 00:09:34.860 Awaish Kumar: Basically, to get all the data.
94 00:09:35.440 ⇒ 00:09:37.330 Uttam Kumaran: I mean, the other thing you could do…
95 00:09:37.730 ⇒ 00:09:41.970 Uttam Kumaran: I mean, is you could go try to run a manual export from Stripe.
96 00:09:44.150 ⇒ 00:09:45.080 Awaish Kumar: Yep.
97 00:09:45.810 ⇒ 00:09:46.619 Uttam Kumaran: I don’t know.
98 00:09:47.260 ⇒ 00:09:49.950 Awaish Kumar: I can do that, but then, like, it will just…
99 00:09:50.320 ⇒ 00:09:56.659 Awaish Kumar: solves the current problem, but, like, a few days, she will say again that the data is still…
100 00:09:58.500 ⇒ 00:10:01.760 Uttam Kumaran: Yeah, this is such a weird thing, I don’t even… I’ve never seen this before.
101 00:10:02.170 ⇒ 00:10:03.329 Demilade: Sorry, what’s the issue?
102 00:10:04.790 ⇒ 00:10:07.750 Awaish Kumar: Yeah, data is… is… is, like, kind of…
103 00:10:09.050 ⇒ 00:10:23.410 Awaish Kumar: step API is basically slow, and Polytomics is… because of that, we are not able to sync all the historical stuff. So now that we have synced some of the historical data, now the current data is, like, coming with a delay of, like.
104 00:10:23.840 ⇒ 00:10:26.679 Awaish Kumar: At least 10 to 15 days.
105 00:10:28.190 ⇒ 00:10:30.810 Demilade: Dancing Infinity’s just crazy.
106 00:10:35.320 ⇒ 00:10:42.379 Awaish Kumar: So, and… but this issue is for invoice table, mostly, so what I will be trying to do today is…
107 00:10:42.720 ⇒ 00:10:46.780 Awaish Kumar: Rewrite some of the code to point to invoice line item.
108 00:10:46.880 ⇒ 00:10:52.640 Awaish Kumar: For… For, like, as a… as a… Badge?
109 00:10:53.330 ⇒ 00:10:58.999 Awaish Kumar: That will cause, like, missing out on some fields, which are related to invoices.
110 00:11:00.040 ⇒ 00:11:12.040 Awaish Kumar: But that’s what she wants, because she needs to do some analysis. For that, she needs product category and stuff like that. We are not able to do that without full invoices data not present.
111 00:11:12.130 ⇒ 00:11:21.220 Awaish Kumar: So, I will just move those chunks to use invoice line items for now, and then we might have to move back once the data is synced completely.
112 00:11:21.520 ⇒ 00:11:23.399 Awaish Kumar: So that’s the plan for today.
113 00:11:23.530 ⇒ 00:11:26.560 Awaish Kumar: And I will sync with Sandra also.
114 00:11:26.810 ⇒ 00:11:30.810 Awaish Kumar: And share the updates, whatever, by end of day today.
115 00:11:32.400 ⇒ 00:11:33.000 Uttam Kumaran: Okay.
116 00:11:36.190 ⇒ 00:11:47.930 Awaish Kumar: yeah, these are the two tasks which are basically here, and I have to finalize them today. For element, all… everything… for all the other setup is done.
117 00:11:48.970 ⇒ 00:11:53.740 Awaish Kumar: CICD and deployment part is pending, so I…
118 00:11:53.990 ⇒ 00:12:01.780 Awaish Kumar: try to integrate GitHub with, within, like, Snowflake for dbt, But, like, there are…
119 00:12:04.050 ⇒ 00:12:10.550 Awaish Kumar: few things, like, we… now that we… if we use Snowflake in dbt, the only…
120 00:12:11.010 ⇒ 00:12:15.749 Awaish Kumar: benefit we get is that for running the jobs, we can use Snowflake compute.
121 00:12:15.980 ⇒ 00:12:23.330 Awaish Kumar: But then, to trigger those, we have to trigger those tasks from… GitHub.
122 00:12:25.280 ⇒ 00:12:34.529 Awaish Kumar: So, like, right now, for example, for Eden, if we run a GitHub action, we just run the dbt… install the dbt and run it there as a dbt core.
123 00:12:34.720 ⇒ 00:12:38.430 Awaish Kumar: Instead, what we can do, we have to still have to have that GitHub action.
124 00:12:38.720 ⇒ 00:12:45.459 Awaish Kumar: But that is going to trigger the Snowflake job. Snowflake does not have direct… integration like dbt Cloud.
125 00:12:46.190 ⇒ 00:12:46.990 Awaish Kumar: To run the job.
126 00:12:46.990 ⇒ 00:12:52.189 Uttam Kumaran: Yeah, Srini, what did you find for CTA? Because we’re doing the same thing there. I thought you could do this via tasks.
127 00:12:52.190 ⇒ 00:12:58.240 Ashwini Sharma: So, this is how it is going to work, right? Task will not be a good approach.
128 00:12:58.480 ⇒ 00:13:06.630 Ashwini Sharma: And, like, why I’ll tell that, right? So, when we are using CICD, right, the recommended approach by Snowflake.
129 00:13:06.760 ⇒ 00:13:14.629 Ashwini Sharma: is to use the authentication mechanism called OIDC, and that didn’t work. I tried yesterday, entire day, trying to get it to work. It didn’t.
130 00:13:14.880 ⇒ 00:13:27.490 Ashwini Sharma: So, now we’ll have to go with a service account and a access token of that service account to do the authentication. Now, once we are authenticated, we, the recommended approach from Snowflake is to use Snow CLI,
131 00:13:27.770 ⇒ 00:13:42.129 Ashwini Sharma: And what Nose DLI does is, whatever code is there in GitHub, it’ll check it out, and then it’ll deploy that entire codebase as a dbt project into Snowflake, and then it will execute that dbt project. So dbt project is a…
132 00:13:42.800 ⇒ 00:13:45.599 Ashwini Sharma: Like, like a first-class object, right?
133 00:13:45.600 ⇒ 00:13:47.210 Uttam Kumaran: run, like, dbt build.
134 00:13:48.080 ⇒ 00:13:48.860 Ashwini Sharma: It won’t.
135 00:13:50.320 ⇒ 00:13:56.340 Uttam Kumaran: you can… you can specify, like, the dbt… Run command syntax?
136 00:13:56.340 ⇒ 00:14:04.409 Ashwini Sharma: Yeah, we can do that, but after it has deployed as an object, right? So, first thing is to deploy it as an object, dbt object, and then we can run it.
137 00:14:05.820 ⇒ 00:14:11.769 Uttam Kumaran: Okay. I mean, what do you guys… what do you guys think? Do you think it’s worth the hassle over just doing cloud?
138 00:14:12.430 ⇒ 00:14:20.679 Awaish Kumar: Yeah, that’s… that’s what I’m… I’m saying, that, like, now we have to do… maintain to… maintain the pipelines at two different places.
139 00:14:21.000 ⇒ 00:14:24.660 Awaish Kumar: We have to maintain it in… like…
140 00:14:24.940 ⇒ 00:14:30.869 Awaish Kumar: GitHub, and then we have to maintain the Snowflake as well for dbt.
141 00:14:31.010 ⇒ 00:14:32.710 Awaish Kumar: And I don’t think we…
142 00:14:33.280 ⇒ 00:14:41.629 Awaish Kumar: We need to do that. And although it can… we can get benefit from Snowflake Compute in terms of, like, when the…
143 00:14:41.890 ⇒ 00:14:48.719 Awaish Kumar: like, there are a lot of models and take time. We can basically can increase the warehouse and use
144 00:14:48.850 ⇒ 00:15:04.790 Awaish Kumar: like, the medium or whatever, and then it can to decrease the processing time and things like that. Like, it’s easy to scale when we are… we will be using Snowflake, but I think if we are using dbt Cloud, that resolves the problem.
145 00:15:06.090 ⇒ 00:15:19.319 Ashwini Sharma: Well, no, this is basically, like, choosing between where do you want to run the dbt command, right? So, I mean, the Eden way, you’re running the dbt command on a GitHub container.
146 00:15:19.950 ⇒ 00:15:26.459 Ashwini Sharma: And in, in Snowflake, We are doing it… we are running the dbt within Snowflake.
147 00:15:26.940 ⇒ 00:15:31.560 Ashwini Sharma: Actually, all the queries and models are going to run on Snowflake only, right?
148 00:15:31.710 ⇒ 00:15:40.989 Awaish Kumar: is that when we run, for example, a job from, like, when I executed a job from GitHub Action for BigQuery, if it
149 00:15:41.180 ⇒ 00:15:48.930 Awaish Kumar: If it, for example, takes a lot of memory to execute that job. Basically, the GitHub action fails.
150 00:15:49.220 ⇒ 00:15:50.060 Awaish Kumar: Right?
151 00:15:50.730 ⇒ 00:15:54.819 Awaish Kumar: And it was failing for Eden, and we need to increase that.
152 00:15:55.650 ⇒ 00:16:03.119 Awaish Kumar: The… basically… the memory, or things like that. So, basically.
153 00:16:03.240 ⇒ 00:16:08.260 Awaish Kumar: That’s what I’m saying. We, like, if we go directly with dbt Cloud.
154 00:16:08.450 ⇒ 00:16:11.550 Awaish Kumar: That is… will be the ideal situation here.
155 00:16:12.980 ⇒ 00:16:16.199 Ashwini Sharma: But dbt Cloud is not involved in this case at all, right?
156 00:16:17.080 ⇒ 00:16:20.059 Uttam Kumaran: No, no, no, like, I guess the point is, like, should we just…
157 00:16:20.940 ⇒ 00:16:24.540 Uttam Kumaran: ditch doing this in Snowflake and just… and purchase cloud.
158 00:16:24.680 ⇒ 00:16:25.330 Awaish Kumar: Yeah.
159 00:16:27.140 ⇒ 00:16:29.060 Uttam Kumaran: Like, would it be a better use of time?
160 00:16:33.360 ⇒ 00:16:35.929 Ashwini Sharma: Well, I don’t have an answer to that.
161 00:16:36.190 ⇒ 00:16:38.830 Awaish Kumar: Yeah, I think so, like… What’s the…
162 00:16:38.830 ⇒ 00:16:42.530 Demilade: upside to, like, the… donating Snowflake?
163 00:16:43.310 ⇒ 00:16:44.900 Uttam Kumaran: Just one last tool.
164 00:16:46.320 ⇒ 00:16:46.870 Awaish Kumar: Yeah.
165 00:16:46.870 ⇒ 00:16:54.260 Demilade: Yeah… But… I mean, in terms of the infrastructure of running dbt.
166 00:16:54.560 ⇒ 00:16:57.679 Demilade: Between, like, dbt Cloud and…
167 00:16:58.230 ⇒ 00:16:59.999 Demilade: Do you run into a snowflake.
168 00:17:01.160 ⇒ 00:17:07.500 Demilade: potentially is… easier to, like, maintain and hand off in dbt Cloud.
169 00:17:09.099 ⇒ 00:17:17.709 Awaish Kumar: Yes, yes, that’s what I’m saying. It’s easier to just maintain it in dbt Cloud. We can, if we want to do the semantic layers or the stuff like that, we can do that.
170 00:17:18.059 ⇒ 00:17:19.829 Awaish Kumar: Indivity Cloud.
171 00:17:20.379 ⇒ 00:17:22.229 Awaish Kumar: And we can’t do that in Snowflake.
172 00:17:25.109 ⇒ 00:17:30.219 Awaish Kumar: Snowflake is basically just using dbt core, and it provides us the way to execute.
173 00:17:30.549 ⇒ 00:17:32.259 Awaish Kumar: As we run in GitHub Action.
174 00:17:36.300 ⇒ 00:17:38.760 Ashwini Sharma: What is the advantage of dbt Cloud?
175 00:17:41.000 ⇒ 00:17:41.560 Awaish Kumar: Honey?
176 00:17:42.360 ⇒ 00:17:45.069 Ashwini Sharma: What is the advantage of having dbt Cloud?
177 00:17:45.630 ⇒ 00:17:50.940 Awaish Kumar: Yeah, for DBD Cloud, we can basically directly connect on… connect it with PRs.
178 00:17:51.120 ⇒ 00:17:53.419 Awaish Kumar: So we don’t have to do any extra…
179 00:17:53.670 ⇒ 00:17:57.600 Awaish Kumar: like, things like GitHub Actions, it can work,
180 00:17:58.660 ⇒ 00:18:02.249 Awaish Kumar: Basically, it can have… we can have, like, semantic layer.
181 00:18:04.990 ⇒ 00:18:06.749 Awaish Kumar: If we are using dbt Cloud.
182 00:18:08.170 ⇒ 00:18:14.359 Awaish Kumar: Option, and then also, like, yeah, like, that’s mainly it.
183 00:18:20.020 ⇒ 00:18:26.930 Demilade: I think in terms of being able to hand over to the teams, It’s less, yeah.
184 00:18:26.930 ⇒ 00:18:30.819 Awaish Kumar: Like, I’ve seen it in the open stems, like, there are…
185 00:18:31.350 ⇒ 00:18:48.669 Awaish Kumar: Cloud, we can have production jobs and multiple production jobs. If the, like, data volume grows, separate them by tags, so we have to basically build all those things ourselves in a GitHub action or something, that we just get it from dbt Cloud, and
186 00:18:49.500 ⇒ 00:18:56.800 Awaish Kumar: it’s worth it. Like, we don’t have to spend time then maintaining it in a… a negative actions pipeline.
187 00:19:05.270 ⇒ 00:19:07.819 Uttam Kumaran: Yeah, I don’t know, it’s tough because…
188 00:19:08.620 ⇒ 00:19:13.310 Uttam Kumaran: I… yeah, it’s… it’s just, like… it’s just expensive, and…
189 00:19:14.680 ⇒ 00:19:17.009 Awaish Kumar: Is it? Isn’t it hundreds?
190 00:19:17.010 ⇒ 00:19:19.659 Uttam Kumaran: I think it’s… I think it’s 100 bucks a user a month.
191 00:19:20.460 ⇒ 00:19:22.660 Awaish Kumar: We can use a common user.
192 00:19:29.000 ⇒ 00:19:39.079 Awaish Kumar: Like, we can develop, like, locally, and we don’t need it for development, right? We just can have a single user, and it will just run the production jobs.
193 00:19:40.310 ⇒ 00:19:41.010 Awaish Kumar: Oh, and the.
194 00:19:41.010 ⇒ 00:19:41.780 Uttam Kumaran: Okay.
195 00:19:41.780 ⇒ 00:19:45.410 Awaish Kumar: in the CICD part, so it can basically do that.
196 00:19:47.050 ⇒ 00:19:54.549 Ashwini Sharma: What about for a client like CTA, where she’s expecting, like, some developers from her team to do the modeling as well?
197 00:19:56.900 ⇒ 00:19:59.639 Uttam Kumaran: Yeah, they can, I mean, dbt Cloud is fine for that, too.
198 00:20:01.510 ⇒ 00:20:02.540 Demilade: countries have…
199 00:20:05.340 ⇒ 00:20:14.839 Demilade: You can do, like, you can have dbt Core on your system for development, and then when you want to test, it invokes dbt, like, the dbt Cloud account.
200 00:20:15.540 ⇒ 00:20:20.509 Demilade: It runs it in development, and then you can push and test, like.
201 00:20:21.150 ⇒ 00:20:26.050 Demilade: people can have individual, dbt core setups on their computers.
202 00:20:32.010 ⇒ 00:20:39.739 Uttam Kumaran: Yeah, this is a tough question. I’m not sure what the right answer is. Because on one hand, yes, like, it’s easy to use dbt Cloud and, like.
203 00:20:39.950 ⇒ 00:20:43.290 Uttam Kumaran: Set up all the jobs, but then it’s another tool
204 00:20:43.710 ⇒ 00:20:49.970 Uttam Kumaran: Whereas, like, for Eden, for example, we’re running everything on GitHub Actions, And…
205 00:20:51.430 ⇒ 00:20:58.360 Uttam Kumaran: Yeah, I mean, it’s… it sort of consolidates everything into just one tool. Like, we’re just running open source GitHub.
206 00:20:58.590 ⇒ 00:21:00.750 Uttam Kumaran: We’re running at just a few jobs.
207 00:21:01.030 ⇒ 00:21:05.790 Uttam Kumaran: Right? Like, why go and get dbt Cloud when nobody’s gonna use the IDE there?
208 00:21:06.060 ⇒ 00:21:06.930 Uttam Kumaran: Either.
209 00:21:07.120 ⇒ 00:21:11.799 Uttam Kumaran: So, for Aiden, it’s a simple setup, right? We are running all the jobs.
210 00:21:12.280 ⇒ 00:21:13.210 Awaish Kumar: Every time.
211 00:21:13.210 ⇒ 00:21:16.179 Uttam Kumaran: Element is not gonna be a more complicated setup, you know?
212 00:21:17.870 ⇒ 00:21:19.489 Awaish Kumar: Yeah, could be, like…
213 00:21:21.050 ⇒ 00:21:25.859 Uttam Kumaran: Same with CTA, it’s not gonna be, like… I mean, but what is complexity here? It’s like…
214 00:21:26.380 ⇒ 00:21:28.670 Uttam Kumaran: Just adding another job, you know?
215 00:21:28.870 ⇒ 00:21:34.779 Awaish Kumar: Do we need to get a whole other tool? Like, what else does dbt Cloud give us that’s, like, worth it?
216 00:21:34.920 ⇒ 00:21:36.049 Uttam Kumaran: I don’t know.
217 00:21:41.810 ⇒ 00:21:42.580 Uttam Kumaran: Right.
218 00:21:47.220 ⇒ 00:21:50.999 Uttam Kumaran: I mean, so the alternatives we’re weighing is, like, we do it all in Snowflake.
219 00:21:51.230 ⇒ 00:21:53.779 Uttam Kumaran: Alternatively, we just do the Eden setup.
220 00:21:56.290 ⇒ 00:22:09.030 Awaish Kumar: Yeah, I’m okay, like, I’m… I’m okay with two, like, ways. One is either use dbt Code and GitHub Actions, as we do for Eden, or use dbt Cloud. Like, adding a layer in Snowflake.
221 00:22:09.260 ⇒ 00:22:10.200 Awaish Kumar: Okay.
222 00:22:10.430 ⇒ 00:22:12.140 Awaish Kumar: You have to run GitHub Actions.
223 00:22:12.270 ⇒ 00:22:14.600 Awaish Kumar: There’s no, like, benefit of that.
224 00:22:17.030 ⇒ 00:22:23.609 Demilade: So, with the Snowflake method, we would have both Snowflake setup and a GitHub setup, or would it only be Snowflake?
225 00:22:23.900 ⇒ 00:22:25.150 Awaish Kumar: I need both setups.
226 00:22:26.570 ⇒ 00:22:28.550 Demilade: Yeah, I think we should just, like…
227 00:22:28.770 ⇒ 00:22:31.770 Demilade: doing a GitHub sect… a GitHub alone, because…
228 00:22:31.770 ⇒ 00:22:38.000 Uttam Kumaran: I mean, but the… again, the nice thing about the Snowflake setup is you just see the dbt project, like, within Snowflake.
229 00:22:38.380 ⇒ 00:22:39.900 Uttam Kumaran: So you’re basically…
230 00:22:39.900 ⇒ 00:22:41.400 Ashwini Sharma: Thanks over there, yeah.
231 00:22:41.640 ⇒ 00:22:47.810 Uttam Kumaran: Yeah, basically, Demolati, they just, like, gave… they just, like, integrated… they did deeper integration with dbt Core.
232 00:22:48.120 ⇒ 00:22:52.830 Uttam Kumaran: So you can see your project and the details all within Snowflake Interface.
233 00:22:53.880 ⇒ 00:23:00.889 Awaish Kumar: like, we can connect GitHub, you know, like, with GitHub, right? We don’t have to basically utilize it for…
234 00:23:01.080 ⇒ 00:23:10.689 Awaish Kumar: for CICD or stuff, like, it can be connected, people can just see that, all the code there, everything is there, but we don’t have to run it there.
235 00:23:19.630 ⇒ 00:23:24.659 Demilade: I think, potentially, what we might just do is maybe we could try this as a use case.
236 00:23:24.800 ⇒ 00:23:30.310 Demilade: See how, like, I think off the top of my head, it does feel a bit more, like…
237 00:23:32.410 ⇒ 00:23:38.900 Demilade: it feels like there’s a bit more to it, like, a bit more complexity to it, than just having everything in GitHub.
238 00:23:39.220 ⇒ 00:23:45.709 Demilade: But, like, we could try it and actually find out that it’s really not as, stressful, quote-unquote.
239 00:23:46.320 ⇒ 00:23:51.239 Demilade: But actually, we might also find that there are, you know, benefits we can get.
240 00:23:51.350 ⇒ 00:23:52.219 Demilade: from it.
241 00:23:52.610 ⇒ 00:23:56.260 Demilade: But… Yeah, I, I, like…
242 00:23:56.490 ⇒ 00:23:59.970 Demilade: Personally, I think I’m leaning towards just the GitHub alone, like the Eden.
243 00:23:59.970 ⇒ 00:24:00.610 Uttam Kumaran: Okay.
244 00:24:01.060 ⇒ 00:24:06.670 Demilade: But, again, it’s a new client, if we feel like we want to try something new, And…
245 00:24:06.880 ⇒ 00:24:14.420 Demilade: maybe over time, we can remove that aspect and go back to just GitHub alone, that’s possible. Versus, like, if we just do…
246 00:24:15.200 ⇒ 00:24:21.599 Demilade: If we do… if we just do GitHub, it might be harder to ramp up to Snowflake as well.
247 00:24:23.730 ⇒ 00:24:33.189 Uttam Kumaran: Yeah, the other reason… the other reason why GitHub Actions is nice, guys, is, like, once we… like, the error logging, we can start to… if we get an error, we can use AI.
248 00:24:33.490 ⇒ 00:24:37.049 Uttam Kumaran: And AI can take the first pass at, like, a PR eventually.
249 00:24:37.350 ⇒ 00:24:38.460 Uttam Kumaran: So…
250 00:24:39.880 ⇒ 00:24:42.200 Demilade: So, with Snowflake?
251 00:24:42.200 ⇒ 00:24:50.490 Awaish Kumar: Yeah, like, my point is that, like, in the snowflake, we can basically connect, and we can showcase that to the client, that the data is
252 00:24:50.610 ⇒ 00:24:54.350 Awaish Kumar: Like, you can see the data and dbt models all at one place.
253 00:24:54.560 ⇒ 00:25:00.739 Awaish Kumar: But then, we can run our jobs in GitHub Actions. Otherwise, it’s just, like, from GitHub Actions…
254 00:25:00.740 ⇒ 00:25:03.789 Uttam Kumaran: I see. So you’re saying don’t do the jobs in Snowflake?
255 00:25:04.100 ⇒ 00:25:08.629 Uttam Kumaran: But do… do the deployment in Snowflake, just to see it.
256 00:25:08.900 ⇒ 00:25:12.300 Awaish Kumar: It’s just the integration. You can connect and it will show up, that’s all.
257 00:25:12.520 ⇒ 00:25:16.829 Awaish Kumar: It’s simple to… to connect GitHub and Snowflake.
258 00:25:18.780 ⇒ 00:25:20.470 Uttam Kumaran: I think that’s a good compromise.
259 00:25:22.180 ⇒ 00:25:25.109 Uttam Kumaran: leverage this. Don’t leverage the snowflake jobs.
260 00:25:25.320 ⇒ 00:25:31.550 Uttam Kumaran: But leverage the UI for, like, deploying the dbt project onto Snowflake.
261 00:25:32.990 ⇒ 00:25:38.879 Uttam Kumaran: But apart from that, like, don’t use Snowflake tasks to orchestrate the jobs or anything like that, right?
262 00:25:44.000 ⇒ 00:25:51.539 Awaish Kumar: Basically, you can do, like… basically, when it is connected, you can do the custom runs in the UI. Like, if somebody…
263 00:25:51.930 ⇒ 00:25:52.850 Uttam Kumaran: I see.
264 00:25:53.390 ⇒ 00:25:57.759 Awaish Kumar: they can custom… want to run something, from UI,
265 00:25:57.850 ⇒ 00:26:13.399 Awaish Kumar: they can just try, test it, do it, but they will not be the one who are deploying, so we don’t need to have deployment part in Snowflake. We can just keep it in GitHub. Simple, like, the way we do it for EDIM.
266 00:26:13.750 ⇒ 00:26:24.140 Ashwini Sharma: No, there are certain advantages of using the deployment within Snowflake as a dbt project, because Snowflake provides additional monitoring tools on top of the dbt runs, right?
267 00:26:24.300 ⇒ 00:26:30.170 Ashwini Sharma: For example, right, if the customer wants to see what went wrong with .
268 00:26:30.170 ⇒ 00:26:40.259 Uttam Kumaran: Yeah, but that’s what I’m saying, you can also… you can also do that in GitHub, so it’s not a… that’s not a… I don’t see that as an advantage. You can also do that in GitHub Actions.
269 00:26:40.260 ⇒ 00:26:42.589 Ashwini Sharma: In GitHub, you can’t see a compiled code, right?
270 00:26:42.780 ⇒ 00:26:45.609 Uttam Kumaran: No, you can. Oh. Yeah, you… well…
271 00:26:46.300 ⇒ 00:26:52.189 Uttam Kumaran: You can… you can download, basically, the… Runtimes, assets, yeah.
272 00:26:52.360 ⇒ 00:26:57.569 Demilade: Can you say… well, a compilation error, is that the question? Like, can you say compilation error?
273 00:26:58.470 ⇒ 00:26:58.840 Ashwini Sharma: No.
274 00:26:58.840 ⇒ 00:26:59.779 Uttam Kumaran: Well, like, this…
275 00:27:00.280 ⇒ 00:27:00.969 Ashwini Sharma: Yeah, if a lot of…
276 00:27:00.970 ⇒ 00:27:04.350 Uttam Kumaran: Even in dbt Cloud, you can’t see the compiled
277 00:27:04.990 ⇒ 00:27:07.960 Uttam Kumaran: Like, for the most part, when a model errors out.
278 00:27:08.180 ⇒ 00:27:14.679 Uttam Kumaran: it’s gonna sit… like, and… but that’s also something that we… you’re gonna catch in PR, it’s not gonna make it in, you know?
279 00:27:15.220 ⇒ 00:27:22.910 Uttam Kumaran: like, what are the odds that a model compil… like, a model compilation error makes it in? For the most part, it’s gonna be a model run error, right?
280 00:27:23.070 ⇒ 00:27:30.579 Ashwini Sharma: Yeah, run error, right. So, basically, when you’re exploring a run error, sometimes you would like to see what code got executed, right?
281 00:27:35.500 ⇒ 00:27:38.049 Awaish Kumar: Mmm, I think it… it does show.
282 00:27:39.300 ⇒ 00:27:39.930 Ashwini Sharma: And good luck.
283 00:27:39.930 ⇒ 00:27:41.340 Uttam Kumaran: Also, like, you could… you can just rock…
284 00:27:41.340 ⇒ 00:27:41.850 Demilade: I’m dbt.
285 00:27:41.850 ⇒ 00:27:43.860 Uttam Kumaran: Decompile on your machine.
286 00:27:44.020 ⇒ 00:27:44.860 Uttam Kumaran: You know?
287 00:27:45.190 ⇒ 00:27:46.499 Uttam Kumaran: Like, that’s what, like…
288 00:27:46.500 ⇒ 00:27:46.830 Ashwini Sharma: Yeah.
289 00:27:46.830 ⇒ 00:27:47.480 Uttam Kumaran: Yeah.
290 00:27:47.790 ⇒ 00:27:53.730 Demilade: dbt run errors do show in GitHub Actions, if you have the, like, if you have it as, like, individual…
291 00:27:54.340 ⇒ 00:28:00.429 Demilade: Like, if you have a dbt run and it fails, it will put the error in, like, what happened.
292 00:28:02.500 ⇒ 00:28:05.169 Demilade: That’s kind of how we investigate, like, eating errors.
293 00:28:10.970 ⇒ 00:28:18.480 Demilade: Unless maybe there’s a, like, more detailed thing that Snowflake does, but, like, if it’s the dbt run error, we do see it with GitHub.
294 00:28:26.870 ⇒ 00:28:30.300 Uttam Kumaran: That just seems like one use case that, like, I don’t think…
295 00:28:31.880 ⇒ 00:28:36.770 Uttam Kumaran: I don’t think really sells it for me. Like, we can… in GitHub Actions, you can still see the error.
296 00:28:37.020 ⇒ 00:28:39.299 Uttam Kumaran: And you can replicate the error locally.
297 00:28:39.810 ⇒ 00:28:41.350 Uttam Kumaran: You can rerun the job.
298 00:28:42.210 ⇒ 00:28:43.010 Uttam Kumaran: You know?
299 00:28:45.670 ⇒ 00:28:46.310 Awaish Kumar: Okay.
300 00:28:46.310 ⇒ 00:28:51.129 Uttam Kumaran: For me, the bigger thing is, like, I think, guys, we’re gonna get into a future where…
301 00:28:51.260 ⇒ 00:29:00.329 Uttam Kumaran: when a dbt error happens, most likely we’ll have AI do the first pass investigation. And so, keeping all of that in GitHub.
302 00:29:01.010 ⇒ 00:29:03.189 Uttam Kumaran: Seems like the best path.
303 00:29:03.700 ⇒ 00:29:04.530 Uttam Kumaran: You know?
304 00:29:04.780 ⇒ 00:29:05.840 Awaish Kumar: Net-net.
305 00:29:05.860 ⇒ 00:29:25.010 Uttam Kumaran: Like, I feel like if we were to talk about, okay, what is a potential advantage versus just cost savings, I think that is it, where when a dbt job error happens, we take the logs, we pass it to AI to do the first investigation on, like, what it could be, and potentially ship the patch.
306 00:29:25.280 ⇒ 00:29:31.039 Uttam Kumaran: that’s, like… for me, out of everything, like, that sells me on GitHub more than anything.
307 00:29:32.420 ⇒ 00:29:34.510 Uttam Kumaran: Okay, because you can’t do that in dbt right now.
308 00:29:38.300 ⇒ 00:29:42.240 Awaish Kumar: Okay, yeah, so yeah, we’ll keep it… With…
309 00:29:42.240 ⇒ 00:29:44.779 Uttam Kumaran: Does that make sense, guys? Like, do we feel good about that?
310 00:29:47.220 ⇒ 00:29:58.690 Uttam Kumaran: I used to be more… I’m more of a fan of dbt Cloud, but it’s kind of like… even the four of us have a hard time articulating any benefit apart from just, like, a UI to click and create jobs, like…
311 00:29:59.040 ⇒ 00:29:59.909 Uttam Kumaran: You know?
312 00:30:00.110 ⇒ 00:30:08.580 Uttam Kumaran: So, for non-technical stakeholders, maybe it’s fine, like, for Hedra, they only have, like, one job, and… Whatever.
313 00:30:08.900 ⇒ 00:30:13.480 Uttam Kumaran: But for an element and this, I don’t know, I feel like it’s just better, but…
314 00:30:25.540 ⇒ 00:30:30.449 Awaish Kumar: Yeah, I’m okay with, like, keeping it simple, that’s… that’s what I was saying.
315 00:30:39.920 ⇒ 00:30:48.230 Awaish Kumar: Okay, then, like, what about, like, for Element, like, we are starting, and I don’t think they have any strong preferences, we can go with GitHub Actions.
316 00:30:48.480 ⇒ 00:30:52.770 Awaish Kumar: But what about CTA? Do they want… Like, there was…
317 00:30:52.770 ⇒ 00:30:56.169 Uttam Kumaran: Yeah, she’s the same thing. I think she’d prefer GitHub Actions.
318 00:30:56.170 ⇒ 00:30:56.780 Awaish Kumar: Okay.
319 00:30:57.090 ⇒ 00:30:59.909 Uttam Kumaran: She’s technical. Their whole team will be technical.
320 00:31:00.990 ⇒ 00:31:04.199 Uttam Kumaran: So maybe we can brief… we can,
321 00:31:04.440 ⇒ 00:31:10.080 Uttam Kumaran: I think, OH, the best thing is if we can just make sure our template repo has the GitHub action set up.
322 00:31:10.400 ⇒ 00:31:11.070 Awaish Kumar: Okay.
323 00:31:11.930 ⇒ 00:31:15.109 Uttam Kumaran: like, you can just… maybe you and Ashwini can take a look at that.
324 00:31:15.400 ⇒ 00:31:18.839 Uttam Kumaran: And then, that way we can copy-paste that for every client, no problem.
325 00:31:21.530 ⇒ 00:31:23.939 Awaish Kumar: Wow. Okay, yeah, sure.
326 00:31:26.600 ⇒ 00:31:27.180 Uttam Kumaran: Yeah.
327 00:31:28.560 ⇒ 00:31:33.499 Awaish Kumar: like, template, like, we are talking about brain-first data platforms, right? Or…
328 00:31:35.130 ⇒ 00:31:37.729 Uttam Kumaran: I think it’s just a new client template.
329 00:31:39.240 ⇒ 00:31:46.409 Awaish Kumar: There are multiple of GitHub repos now. We have demo, which, like, working demo, PR repo, and .
330 00:31:46.920 ⇒ 00:31:54.109 Uttam Kumaran: Yeah, I think there should just be a one that’s, like, new client template. That’s gonna be the one where, like, every new client repo would just copy that over.
331 00:31:54.110 ⇒ 00:31:55.360 Awaish Kumar: Okay, yeah.
332 00:31:56.090 ⇒ 00:31:57.939 Uttam Kumaran: But maybe you guys can have a look at…
333 00:31:58.240 ⇒ 00:32:03.760 Uttam Kumaran: the way we do it for Eden, and then… Create a clean version.
334 00:32:06.230 ⇒ 00:32:09.379 Awaish Kumar: Yeah, basically, okay, I will start on that today.
335 00:32:09.380 ⇒ 00:32:12.710 Uttam Kumaran: Like, CTA will be happy because they don’t want to get more tools.
336 00:32:12.820 ⇒ 00:32:18.770 Uttam Kumaran: And, like, I think they’re comfortable with everything on GitHub. Element, they don’t care much.
337 00:32:19.350 ⇒ 00:32:21.330 Awaish Kumar: Hedra is fine.
338 00:32:21.960 ⇒ 00:32:23.360 Uttam Kumaran: You know, so…
339 00:32:23.740 ⇒ 00:32:30.140 Uttam Kumaran: Yeah, and then… and then, basically, Awish, what you can do is you can work with AI team to see if maybe they can help on the,
340 00:32:30.700 ⇒ 00:32:34.910 Uttam Kumaran: having AI investigate the errors. That would make this, like, way worth it.
341 00:32:35.390 ⇒ 00:32:36.340 Awaish Kumar: Okay, yeah.
342 00:32:39.920 ⇒ 00:32:45.530 Awaish Kumar: Okay, I can add that in the plan to meet with Sam and figure out how can we
343 00:32:47.150 ⇒ 00:32:49.039 Awaish Kumar: Like, scope that out.
344 00:32:49.430 ⇒ 00:32:54.200 Awaish Kumar: From that, we are on CTA.
345 00:32:56.290 ⇒ 00:32:58.310 Awaish Kumar: Yes, actually.
346 00:33:00.110 ⇒ 00:33:03.950 Ashwini Sharma: No progress on CTA, except for the CICD thing.
347 00:33:04.520 ⇒ 00:33:09.090 Ashwini Sharma: Yeah, that’s alright.
348 00:33:10.080 ⇒ 00:33:14.249 Uttam Kumaran: Ashwin, do you want me to… I can send a note this morning, with a couple things.
349 00:33:16.100 ⇒ 00:33:16.980 Ashwini Sharma: for CDA?
350 00:33:17.370 ⇒ 00:33:18.080 Uttam Kumaran: Yeah.
351 00:33:20.360 ⇒ 00:33:20.890 Ashwini Sharma: Okay.
352 00:33:20.890 ⇒ 00:33:26.710 Uttam Kumaran: Like, I’m gonna… I’m gonna send over a little bit… we finished our scope for the proposal,
353 00:33:27.580 ⇒ 00:33:31.880 Uttam Kumaran: And then, yeah, if you want to tag along and share any progress,
354 00:33:32.510 ⇒ 00:33:34.320 Uttam Kumaran: On your side, that would be good.
355 00:33:37.130 ⇒ 00:33:39.960 Ashwini Sharma: Yeah, I’ll, I’ll talk with, that guy.
356 00:33:40.370 ⇒ 00:33:40.909 Ashwini Sharma: Go ahead.
357 00:33:41.660 ⇒ 00:33:42.570 Uttam Kumaran: Yeah. Okay.
358 00:33:42.570 ⇒ 00:33:47.860 Ashwini Sharma: I’ll do that today, and get, start working on the new report, right?
359 00:33:49.030 ⇒ 00:33:52.820 Uttam Kumaran: Yeah, and I wish for Element to send… you can send a message whenever you’re comfortable.
360 00:33:54.290 ⇒ 00:34:01.370 Awaish Kumar: Yes, I will be just adding GitHub Actions, and I’m, like, that will be the end of it. Alright, cool.
361 00:34:01.550 ⇒ 00:34:02.700 Awaish Kumar: And sent it.
362 00:34:03.890 ⇒ 00:34:04.660 Uttam Kumaran: Great, perfect.
363 00:34:07.550 ⇒ 00:34:14.029 Awaish Kumar: Yeah, for CICD, I think we… for production, like, do we have any cadence requirement?
364 00:34:16.650 ⇒ 00:34:19.060 Ashwini Sharma: So once a day, until they change it.
365 00:34:20.980 ⇒ 00:34:22.199 Awaish Kumar: Daily, right? Yeah.
366 00:34:24.880 ⇒ 00:34:25.710 Awaish Kumar: Okay.
367 00:34:25.710 ⇒ 00:34:26.330 Uttam Kumaran: Duck.
368 00:34:28.170 ⇒ 00:34:32.400 Awaish Kumar: Okay, so I think that’s it for CT as well, right? We will be just…
369 00:34:32.570 ⇒ 00:34:33.100 Ashwini Sharma: Yep.
370 00:34:33.100 ⇒ 00:34:36.340 Awaish Kumar: Mapping up the CICD and maybe this new report.
371 00:34:36.719 ⇒ 00:34:37.689 Awaish Kumar: And that’s all.
372 00:34:38.840 ⇒ 00:34:39.420 Ashwini Sharma: Right.
373 00:34:43.429 ⇒ 00:34:49.760 Awaish Kumar: Okay, then for… to default…
374 00:34:52.370 ⇒ 00:34:54.709 Awaish Kumar: Tamilare, do you want to give updates?
375 00:34:57.630 ⇒ 00:35:08.939 Demilade: So for default, I’ll be honest, I haven’t gotten to… I had a meeting with Laura, so that was done. I was able to get some context on what was important to her. So for her, a lot of things were, like, GTM metrics.
376 00:35:10.890 ⇒ 00:35:22.719 Demilade: a marketer’s done, so maybe that’s why it’s not here. But the important things were GTM metrics and some financial metrics, so she just cares about, you know, revenue,
377 00:35:23.720 ⇒ 00:35:25.990 Demilade: And all of those kind of things, so…
378 00:35:26.450 ⇒ 00:35:39.779 Demilade: Yeah, that’s what she cares about. For the other part, the dashboard, I wasn’t able to get to it. I had an issue, like, there’s an issue with Eden and ad spend, so I was kind of switched into that.
379 00:35:40.000 ⇒ 00:35:42.589 Demilade: So that’s what put a pause on this.
380 00:35:42.750 ⇒ 00:35:43.590 Awaish Kumar: So, thank you.
381 00:35:44.160 ⇒ 00:35:47.969 Awaish Kumar: Whatever she said, like, did you document them somewhere?
382 00:35:48.250 ⇒ 00:35:58.860 Demilade: Yeah, so I’ve put the gran… in the ticket, I’ve put the granola of the… I gave a high level, I put the granola of the meeting, but I would always… I would just make a doc. Appreni, they’re also making a doc
383 00:35:59.130 ⇒ 00:36:07.819 Demilade: In the default team, where they are gathering their requirements together, but it’s a bit… it’s going to be a bit slow, because most of the team is out of office.
384 00:36:07.990 ⇒ 00:36:09.879 Demilade: Until next year.
385 00:36:10.080 ⇒ 00:36:18.159 Demilade: But there is an existing doc that is work in progress, but I will start the… I will start our own version of it.
386 00:36:18.300 ⇒ 00:36:21.520 Demilade: Based off the calls, so in case, you know.
387 00:36:22.040 ⇒ 00:36:29.600 Demilade: so if there’s a case where I have some stuff, and they have, they forget to share it in the doc, we would have it documented.
388 00:36:30.060 ⇒ 00:36:31.700 Demilade: So yeah.
389 00:36:31.700 ⇒ 00:36:35.349 Awaish Kumar: So, for 191, is it… Still in progress.
390 00:36:35.350 ⇒ 00:36:40.859 Demilade: It’s still in progress, like I said, I wasn’t able to get to you yesterday, what I was handling was the Eden work.
391 00:36:41.040 ⇒ 00:36:41.650 Demilade: Understood.
392 00:36:41.650 ⇒ 00:36:42.200 Awaish Kumar: Are you planning.
393 00:36:42.200 ⇒ 00:36:42.660 Demilade: Literally.
394 00:36:42.680 ⇒ 00:36:45.740 Awaish Kumar: Finalize it today, or, like, tomorrow, or…
395 00:36:46.100 ⇒ 00:36:47.230 Demilade: Oh, no, no, today.
396 00:36:47.570 ⇒ 00:36:48.250 Awaish Kumar: Okay.
397 00:36:48.250 ⇒ 00:36:56.610 Demilade: the eating stuff just came up. I had to help, Sesan get… try and get started with Cursor and, GitHub.
398 00:36:56.800 ⇒ 00:37:01.209 Demilade: that was… there’s an issue with that, and then also, I had to work on…
399 00:37:02.350 ⇒ 00:37:07.179 Demilade: Just trying to figure out why ad spend was $20K lower, but yeah.
400 00:37:07.560 ⇒ 00:37:09.830 Demilade: That was kind of what was happening yesterday.
401 00:37:12.250 ⇒ 00:37:19.090 Awaish Kumar: Yeah, but I just mean… I just want to confirm, like, so this is the plan for this week, right? By tomorrow, you will close these two.
402 00:37:19.820 ⇒ 00:37:21.289 Demilade: Yes, yes, yes, yes, yes, yes.
403 00:37:22.550 ⇒ 00:37:24.039 Demilade: That is definitely the plan.
404 00:37:25.040 ⇒ 00:37:26.920 Awaish Kumar: Okay, yeah, whistle.
405 00:37:28.820 ⇒ 00:37:36.409 Mustafa Raja: Yeah, we received this note, sorry, it’s not supposed data, and I wrote it down. So, 202…
406 00:37:36.830 ⇒ 00:37:39.080 Mustafa Raja: Instrumental Review.
407 00:37:39.420 ⇒ 00:37:44.409 Mustafa Raja: And then… yeah. And then for 2 and 3R,
408 00:37:44.770 ⇒ 00:37:47.180 Mustafa Raja: I’ll either work today or tomorrow.
409 00:37:48.630 ⇒ 00:37:49.340 Awaish Kumar: Okay.
410 00:37:49.340 ⇒ 00:37:50.050 Mustafa Raja: Yeah.
411 00:37:51.730 ⇒ 00:37:53.330 Awaish Kumar: This is in review.
412 00:37:54.240 ⇒ 00:37:57.319 Mustafa Raja: Yeah, yeah, the data is in Motherdeck now, yeah.
413 00:37:57.640 ⇒ 00:37:59.039 Mustafa Raja: In the meme schema.
414 00:37:59.800 ⇒ 00:38:05.000 Awaish Kumar: Okay, so data is uploaded to MotherRap, it’s… Yeah. Receiving the data, okay.
415 00:38:05.000 ⇒ 00:38:05.819 Mustafa Raja: Yeah, yeah.
416 00:38:07.560 ⇒ 00:38:08.320 Demilade: Yes.
417 00:38:08.320 ⇒ 00:38:10.690 Mustafa Raja: I just need Devinadi to confirm if it looks good.
418 00:38:11.330 ⇒ 00:38:14.380 Demilade: Yeah, so I started looking at it, I’m poking around at it,
419 00:38:14.840 ⇒ 00:38:22.260 Demilade: It does look good, I mean, obviously, you pointed out there are some null tables, yeah, so that’s fine as well.
420 00:38:22.460 ⇒ 00:38:30.560 Demilade: I will just look through the rest, and I’ve marked the ones I know, and I’ll look through the rest and just kind of ensure that
421 00:38:31.210 ⇒ 00:38:33.480 Demilade: We’re able to get all the information we need from that.
422 00:38:33.890 ⇒ 00:38:34.530 Mustafa Raja: Okay.
423 00:38:35.810 ⇒ 00:38:36.640 Awaish Kumar: Okay.
424 00:38:38.000 ⇒ 00:38:41.149 Awaish Kumar: Okay, for default, then that’s the plan.
425 00:38:41.820 ⇒ 00:38:49.509 Awaish Kumar: We will be finalizing some data architecture diagram, data platform documentation, And some ETL plan.
426 00:38:52.730 ⇒ 00:38:53.260 Uttam Kumaran: Okay.
427 00:38:53.540 ⇒ 00:38:56.650 Awaish Kumar: And then for Eden,
428 00:38:58.920 ⇒ 00:39:04.360 Awaish Kumar: Yes, Ashwini, we… I think we just merged that PR for…
429 00:39:04.870 ⇒ 00:39:09.310 Awaish Kumar: Or the pipelines. Yeah. So, was that working? Did you try running it?
430 00:39:09.310 ⇒ 00:39:15.179 Ashwini Sharma: Yeah, yeah, it is working now. That is working. It is sending out, data to Northbeam.
431 00:39:15.390 ⇒ 00:39:20.799 Ashwini Sharma: The 1275 is in review by Henry.
432 00:39:21.300 ⇒ 00:39:23.590 Ashwini Sharma: And so is 1240.
433 00:39:24.910 ⇒ 00:39:26.099 Awaish Kumar: This is in review, right?
434 00:39:26.560 ⇒ 00:39:27.770 Ashwini Sharma: Yes.
435 00:39:27.770 ⇒ 00:39:28.550 Awaish Kumar: into host.
436 00:39:30.310 ⇒ 00:39:31.080 Ashwini Sharma: Okay.
437 00:39:31.480 ⇒ 00:39:38.180 Awaish Kumar: So, I think we are good here, like, once you review it, we can just move it to done, and that’s basically it.
438 00:39:38.980 ⇒ 00:39:39.620 Uttam Kumaran: Okay.
439 00:39:40.290 ⇒ 00:39:43.400 Awaish Kumar: Yeah, demo, like, you already given up with that.
440 00:39:44.600 ⇒ 00:39:46.469 Awaish Kumar: You worked on it already, right?
441 00:39:48.730 ⇒ 00:39:53.060 Demilade: Yes. I’ve started working it, and…
442 00:39:53.710 ⇒ 00:40:00.579 Demilade: I haven’t finally gotten the final answer. I was… but I’m almost there, and I’ll send a…
443 00:40:00.770 ⇒ 00:40:02.260 Demilade: A message to the team.
444 00:40:02.570 ⇒ 00:40:05.130 Demilade: Also the, like, the…
445 00:40:05.320 ⇒ 00:40:13.540 Demilade: Because it was pointed out by Stuart, so… I’m sending a message to the Eden team today as to what it was and the fix that has been done.
446 00:40:18.530 ⇒ 00:40:19.230 Awaish Kumar: Okay, so…
447 00:40:19.230 ⇒ 00:40:22.109 Uttam Kumaran: Yeah, let me know where I can… let me know if I can plug in anywhere.
448 00:40:23.190 ⇒ 00:40:24.479 Demilade: Okay, sounds good.
449 00:40:24.720 ⇒ 00:40:30.450 Awaish Kumar: So for this, like, for Eden, we will be just closing 1 to 9.0 today.
450 00:40:30.630 ⇒ 00:40:35.619 Awaish Kumar: And then, 1282… Are you still working on this, or…
451 00:40:37.760 ⇒ 00:40:38.430 Demilade: Excuse me.
452 00:40:39.260 ⇒ 00:40:40.340 Awaish Kumar: for next…
453 00:40:41.360 ⇒ 00:40:49.570 Demilade: I haven’t started work on this, to be honest, so… I… I don’t know.
454 00:40:49.730 ⇒ 00:40:58.819 Demilade: I don’t know the priority of it, I think that’s the… they say it’s high, but, like, I’m not sure if it’s actually that high. But I will talk to, because, like.
455 00:40:58.950 ⇒ 00:41:01.760 Demilade: ad spend being office, I would assume, higher.
456 00:41:01.940 ⇒ 00:41:02.450 Awaish Kumar: Yes.
457 00:41:05.960 ⇒ 00:41:08.840 Awaish Kumar: Yeah, that one is… Yeah, like…
458 00:41:10.110 ⇒ 00:41:12.690 Awaish Kumar: Third one is high, and we can…
459 00:41:13.460 ⇒ 00:41:19.589 Awaish Kumar: I don’t know if you, like, do you have time to work on this tomorrow and close it, or are we, like, otherwise we can just push it?
460 00:41:19.590 ⇒ 00:41:32.540 Demilade: I think tomorrow will be… I think tomorrow might be my ideal. Today, I’m not sure I necessarily have the time to close… to get to this today, but tomorrow I should be able to have the time, if I, like, once I close default and this.
461 00:41:32.980 ⇒ 00:41:34.940 Demilade: I would… I should have time tomorrow.
462 00:41:36.410 ⇒ 00:41:37.150 Awaish Kumar: Okay.
463 00:41:37.380 ⇒ 00:41:45.040 Awaish Kumar: So yeah, that’s the plan before we go to the vacation, right? We will be closing these two tickets for…
464 00:41:45.440 ⇒ 00:41:50.100 Awaish Kumar: Eden and these two are basically… Almost done from Ashwini.
465 00:41:56.530 ⇒ 00:41:59.890 Awaish Kumar: Okay, I think we are good here, and…
466 00:42:03.300 ⇒ 00:42:04.919 Awaish Kumar: Basically, that’s it, so…
467 00:42:05.180 ⇒ 00:42:05.790 Uttam Kumaran: Okay.
468 00:42:06.000 ⇒ 00:42:08.800 Awaish Kumar: Does that sound good, like, before going into the vacation?
469 00:42:09.680 ⇒ 00:42:16.590 Uttam Kumaran: Yeah, I think, I think, as much as I can get some help on getting BigQuery set up for ABC, that would be helpful.
470 00:42:18.320 ⇒ 00:42:21.169 Uttam Kumaran: I’ll need to do some analysis work for them.
471 00:42:22.000 ⇒ 00:42:22.430 Awaish Kumar: So, service.
472 00:42:22.430 ⇒ 00:42:23.460 Uttam Kumaran: Probably this…
473 00:42:23.460 ⇒ 00:42:24.120 Awaish Kumar: The blotto.
474 00:42:24.120 ⇒ 00:42:24.880 Uttam Kumaran: Yeah.
475 00:42:25.040 ⇒ 00:42:25.730 Uttam Kumaran: Okay.
476 00:42:26.310 ⇒ 00:42:29.980 Awaish Kumar: We can work on, like, the structuring the biker itself.
477 00:42:30.330 ⇒ 00:42:34.709 Awaish Kumar: But, like, from… to connect it with the dbt core, you need to source count key.
478 00:42:35.480 ⇒ 00:42:38.180 Uttam Kumaran: Okay, yeah, if we can at least do the structure.
479 00:42:38.720 ⇒ 00:42:39.220 Awaish Kumar: Okay.
480 00:42:39.220 ⇒ 00:42:41.730 Uttam Kumaran: I can start to dump things in.
481 00:42:41.730 ⇒ 00:42:45.399 Awaish Kumar: Do we have already have GitHub repo, and we want to create one, or how?
482 00:42:47.110 ⇒ 00:42:49.340 Uttam Kumaran: We do have a GitHub repo.
483 00:42:49.450 ⇒ 00:42:52.920 Uttam Kumaran: for ABC, we don’t have one for data, though.
484 00:42:54.230 ⇒ 00:42:57.530 Uttam Kumaran: for, like… Their data work.
485 00:42:57.880 ⇒ 00:43:02.190 Uttam Kumaran: It’s all in one, so… I’ll leave it to…
486 00:43:03.740 ⇒ 00:43:08.240 Uttam Kumaran: Yeah, I mean, up to you, if we want to put it all in the same thing, or create a new one.
487 00:43:08.370 ⇒ 00:43:09.879 Uttam Kumaran: Maybe we’ll create a new one.
488 00:43:10.120 ⇒ 00:43:16.150 Awaish Kumar: But is it, like, if it is internal, like, we can create a new one, but if it is external, we can just put it in there.
489 00:43:16.780 ⇒ 00:43:21.529 Uttam Kumaran: No, it’s inter… it’s… it’s in… it’s… the codebase is in our organization right now.
490 00:43:22.120 ⇒ 00:43:23.949 Awaish Kumar: Okay, then we can keep it separate.
491 00:43:24.870 ⇒ 00:43:30.560 Uttam Kumaran: Okay. Yeah, for me, I just… I have, like, tons of Excel files that I need to write to the…
492 00:43:30.670 ⇒ 00:43:31.930 Uttam Kumaran: to BigQuery.
493 00:43:32.190 ⇒ 00:43:37.300 Uttam Kumaran: This week, and I’m getting some SQL DB access.
494 00:43:37.860 ⇒ 00:43:41.300 Uttam Kumaran: To a few other systems that I’m gonna just…
495 00:43:41.490 ⇒ 00:43:44.659 Uttam Kumaran: Try to pipe in one, like, one-time export.
496 00:43:44.780 ⇒ 00:43:48.339 Uttam Kumaran: So, as long as there’s a area for me to drop those.
497 00:43:48.500 ⇒ 00:43:52.890 Uttam Kumaran: That’s what’s most important, and then… Also, like,
498 00:43:53.950 ⇒ 00:44:02.550 Uttam Kumaran: like, for example, Amber is gonna… Amber is gonna help with some analysis, so is Zoran, so I just want to make sure they have an environment where they can run queries.
499 00:44:03.850 ⇒ 00:44:04.370 Uttam Kumaran: Okay.
500 00:44:05.800 ⇒ 00:44:09.969 Awaish Kumar: So, I think I will work with Mustafa, and we can just…
501 00:44:10.310 ⇒ 00:44:24.520 Awaish Kumar: initialize the dbt project, and and then set up the structure in BigQuery, but the only thing we’ll be missing is the connection between those two ones, until and unless we have the service company.
502 00:44:24.970 ⇒ 00:44:25.650 Uttam Kumaran: Okay.
503 00:44:25.970 ⇒ 00:44:26.820 Awaish Kumar: Okay.
504 00:44:28.870 ⇒ 00:44:32.590 Awaish Kumar: Yeah, but we can leave the template profiles.yml once…
505 00:44:33.510 ⇒ 00:44:37.250 Awaish Kumar: We have the key, how it ought to set up and… and run it.
506 00:44:39.600 ⇒ 00:44:40.150 Uttam Kumaran: Okay.
507 00:44:41.550 ⇒ 00:44:43.279 Awaish Kumar: Okay, yeah, that’s all.
508 00:44:47.300 ⇒ 00:44:47.930 Uttam Kumaran: Okay.
509 00:44:48.200 ⇒ 00:44:49.790 Uttam Kumaran: Alright, thank you guys.
510 00:44:51.010 ⇒ 00:44:52.030 Uttam Kumaran: Talk to you soon.
511 00:44:52.460 ⇒ 00:44:52.950 Ashwini Sharma: Thank you.
512 00:44:54.070 ⇒ 00:44:54.840 Demilade: Thank you.