Meeting Title: Brainforge Interview w- Demilade Date: 2026-04-21 Meeting participants: Harsh Punjabi, Demilade Agboola
WEBVTT
1 00:00:54.150 ⇒ 00:00:55.550 Harsh Punjabi: Hey, hey Timmy, how are you?
2 00:00:56.400 ⇒ 00:01:04.600 Demilade Agboola: Hi, hi, my name’s Dimladeh, I’m a Senior Analytics Engineer with Brainforge. Nice to meet you, Hash.
3 00:01:04.910 ⇒ 00:01:06.730 Demilade Agboola: How’s your daughter, by the way?
4 00:01:06.730 ⇒ 00:01:10.960 Harsh Punjabi: She’s much better now, thanks for asking. And so, so sorry for the delay earlier.
5 00:01:11.160 ⇒ 00:01:19.689 Demilade Agboola: Oh, no, no, no, that’s all good, it’s fine. I mean, health is, in fact, the most important thing, so I have no issues rescheduling for those reasons.
6 00:01:22.080 ⇒ 00:01:22.515 Demilade Agboola: Long…
7 00:01:23.160 ⇒ 00:01:42.390 Demilade Agboola: Alright, so the context of this interview will just be, discussing your experience, as well as trying to walk us, like, we’ll walk through, like, scenarios. I just would like to learn, how you, you know, solve these sort of issues, what
8 00:01:42.740 ⇒ 00:01:48.260 Demilade Agboola: Assumptions and questions you have as you’re walking through these scenarios and how your mind thinks about this.
9 00:01:50.540 ⇒ 00:01:59.009 Demilade Agboola: So there won’t be any, like, live coding, but it would just be more of, like, okay, so how would you handle this scenario, if we build out this scenario?
10 00:01:59.520 ⇒ 00:02:05.909 Demilade Agboola: and you have any issues, how would you handle that? So that’s kind of what we’ll be doing today. Do you have any questions about that?
11 00:02:06.150 ⇒ 00:02:23.820 Harsh Punjabi: I think, no, I’m so far so good. You know, I think if I have any questions, maybe I can ask you around the time, because I also need to understand the structure of it a little bit better, so once we jump into it, maybe if I have any process-related questions, I’ll confirm those at that time. Let me know if that works.
12 00:02:24.170 ⇒ 00:02:27.829 Demilade Agboola: Okay, sounds good then. I think in that case, we can just kind of get started.
13 00:02:28.210 ⇒ 00:02:33.410 Demilade Agboola: Alright, so first thing would just be, like, please can you walk me through, like, your…
14 00:02:33.740 ⇒ 00:02:38.540 Demilade Agboola: Your history, your job history, like, what you’ve worked on, as well as your technical stack.
15 00:02:38.720 ⇒ 00:02:42.049 Demilade Agboola: So at least I have an understanding of, like, what you’ve done.
16 00:02:42.170 ⇒ 00:02:43.460 Demilade Agboola: How you would fit in.
17 00:02:43.950 ⇒ 00:02:57.890 Harsh Punjabi: Sure. So, as of now, I am, leading the hospitality analytics engineering initiatives at Lightspeed. So, in hospitality, we are building… we’re actually working on a migration project to migrate their data warehouse to the new hub-and-spoke model.
18 00:02:57.890 ⇒ 00:03:10.220 Harsh Punjabi: The tech stack that we primarily use is, earlier it was dbt and BigQuery, but now it’s, they’ve moved to DataForm, they are, fully, you know, adapted to the GCP stack.
19 00:03:10.220 ⇒ 00:03:26.499 Harsh Punjabi: So, we’ve, recently migrated to DataForm, and now we’re building out the DataForm projects, within the hub-and-spoke model for, specifically for the hospitality vertical. So, the, the day-to-day basically entails, first of all.
20 00:03:26.500 ⇒ 00:03:38.480 Harsh Punjabi: Identifying the use cases for the migration, what current reports they have which needs migrating, building the data models behind them, implementing the medallion architecture for them, building out all the SQLX files and data models.
21 00:03:38.480 ⇒ 00:03:48.850 Harsh Punjabi: understanding how to optimize the costs, on BigQuery, understanding what would be the best approach to handle, carry forward or, you know, data refresh.
22 00:03:48.850 ⇒ 00:03:55.989 Harsh Punjabi: So we’re basically working on that. I am using this approach called business event analysis modeling to kind of design my data models.
23 00:03:55.990 ⇒ 00:04:09.760 Harsh Punjabi: For this, so that’s what we’ve been doing here. Before hospitality within Lightspeed, I was working with the product squads. I was working on the GTV, which is Gross Transaction Value, metric. Also did a small project on payments, NRR.
24 00:04:09.770 ⇒ 00:04:13.999 Harsh Punjabi: So, again, this is a very similar thing working on data models end-to-end.
25 00:04:14.330 ⇒ 00:04:31.850 Harsh Punjabi: Prior to my time at Lightspeed, I was with a consulting company which was doing a project for TD Bank. So there, we were building their cost allocation warehouse. So, within a bank, there are multiple different verticals, right? Some of them are revenue generating, some of them are not, some of them are support.
26 00:04:31.850 ⇒ 00:04:35.609 Harsh Punjabi: So, there is cost allocated between the supporting,
27 00:04:35.610 ⇒ 00:04:47.320 Harsh Punjabi: verticals versus the revenue-generating verticals. So, it was kind of manual there, and they had an on-prem, database which they were using. But eventually, as they started to migrate, to,
28 00:04:47.470 ⇒ 00:04:50.819 Harsh Punjabi: Snowflake, so we kind of built out the dbt models.
29 00:04:50.820 ⇒ 00:05:10.720 Harsh Punjabi: designed the data warehouse model for them, and there was a bunch of consulting work involved in that as well, where we had to kind of sit with all the VPs of different finance teams to understand how they allocate costs, in terms of cost allocation methodology, and define the metrics. We were primarily using Hyperion data. Hyperion was the tool that they were using for their accounting.
30 00:05:10.720 ⇒ 00:05:22.340 Harsh Punjabi: So, so that was one of the projects I did there. And before that, I did a similar project for, like, again, on cost-related piece itself, but that was for the stock trading platform.
31 00:05:22.340 ⇒ 00:05:38.780 Harsh Punjabi: So, they, they were, it was not cost allocation, but the expenses that they were making on their stock trading platform, for different services, we had to design, like, a data warehouse for them to be able to track their costs better. It was a relatively smaller project, but, you know, kind of fit into the same, their financial strategy.
32 00:05:38.780 ⇒ 00:05:51.669 Harsh Punjabi: Prior to that, I worked with a startup in Dubai, so I was the second member of the data team there. So that kind of, included setting up Fivetran connectors, bringing in data from the different sources.
33 00:05:51.670 ⇒ 00:05:56.910 Harsh Punjabi: A lot of… they had a lot of funnel analysis, because it was e-commerce.
34 00:05:56.910 ⇒ 00:06:16.750 Harsh Punjabi: So, you know, building out all the reporting, building the investor reporting, so I was working on their entire tech stack there. I was doing everything there, I mean, you know, defining the database tables, you know, bringing out the data from the API integrations, cleaning up the data, creating the final gold layers, and eventually building dashboards as well.
35 00:06:16.750 ⇒ 00:06:28.350 Harsh Punjabi: And before that, I was, in a business intelligence-oriented role, in American Express and in ZS. So ZS was a consulting firm, again, an American pharma client, so we were…
36 00:06:28.350 ⇒ 00:06:37.799 Harsh Punjabi: basically helping them, you know… and in both those roles, I was mainly working with the business teams to gather requirements, build dashboards, build,
37 00:06:38.490 ⇒ 00:06:49.880 Harsh Punjabi: work on, you know, like, more analytical projects, like market basket analysis, product bundling. For ZS, I did a pro… I worked on a project for a product launch.
38 00:06:50.110 ⇒ 00:07:04.479 Harsh Punjabi: So those were more analytics-oriented projects, which was methodology plus dashboarding. So yeah, I mean, that’s the progression in my career. I started off as a business analyst, you know, grew into a data analyst role, and eventually into an analytics engineering role.
39 00:07:04.780 ⇒ 00:07:06.950 Demilade Agboola: That’s pretty good, that’s pretty good.
40 00:07:07.280 ⇒ 00:07:14.650 Demilade Agboola: Okay, so let’s kind of start, like, the scenario questions, and,
41 00:07:15.630 ⇒ 00:07:19.980 Demilade Agboola: So, if we had a client that wanted to…
42 00:07:20.590 ⇒ 00:07:24.410 Demilade Agboola: I wanted to build out, like, a daily reporting revenue marked.
43 00:07:24.560 ⇒ 00:07:30.379 Demilade Agboola: Yep. From 3 different sources, that was Stripe, Salesforce, on Google Ads.
44 00:07:32.910 ⇒ 00:07:40.059 Demilade Agboola: how would you design a solution? What assumptions would you make? What questions would you have? And how would you go about going from
45 00:07:40.350 ⇒ 00:07:45.009 Demilade Agboola: These are the sources to our mind.
46 00:07:45.260 ⇒ 00:07:57.250 Harsh Punjabi: Okay. So, first thing to understand here is the business process around it. So, the… I understand there are… these are three sources, like, we have Salesforce, we have, Google Ads, and I think you mentioned Stripe, or…
47 00:07:57.250 ⇒ 00:07:59.129 Demilade Agboola: Yes, Salesforce, Stripe, and Google Ads, yeah.
48 00:07:59.130 ⇒ 00:08:16.120 Harsh Punjabi: Yeah, so we have these three sources. First, I would try to understand their business process, how is it fitting into their entire workflow, right? Based on that workflow, you know, normally, like, I really appreciate the BEAM methodology, like Business Event Analysis Modeling, because what that does is.
49 00:08:16.220 ⇒ 00:08:33.169 Harsh Punjabi: based on the business process, it helps me identify the seven W’s. So, what, where, when, etc, and then how much is the final piece. So, how much becomes my fact, the others become my dimensions. I would structure the… so structuring the dimensions, kind of, this is the first step.
50 00:08:33.190 ⇒ 00:08:56.999 Harsh Punjabi: Right? And after that, before I structure my dimensional tables properly, I would actually go and see what metrics they’re trying to report. Like, based on revenue, there could be one or there could be multiple metrics, so I would first align on them on the definition of what they want to see. And based on those definitions and based on their use case scenarios, I would structure my dimensional tables to understand if a start schema is the best, or if maybe a snowflake schema would be better based on how the
51 00:08:57.000 ⇒ 00:08:58.589 Harsh Punjabi: Dimensions are structured.
52 00:08:58.600 ⇒ 00:09:02.590 Harsh Punjabi: And that would help give me the final design of the data mart.
53 00:09:02.590 ⇒ 00:09:22.149 Harsh Punjabi: And, then I would, once I have the final design of the data mart, I would kind of reverse engineer it to see how can I build, like, from the staging layer, what business logic should live in the middle layer, silver or conformed, whatever, you know, we could call it. And, you know, what business logic could live there, and, yeah, how the final data marts would play out.
54 00:09:22.150 ⇒ 00:09:25.390 Harsh Punjabi: Again, at the last step, like, while I’m…
55 00:09:25.390 ⇒ 00:09:29.199 Harsh Punjabi: building out the final data marts, I would kind of also include
56 00:09:29.200 ⇒ 00:09:46.760 Harsh Punjabi: query optimization, cloud cost perspective there as well. What type of queries are the users, you know, doing most frequently? Depends on whether they’re using Snowflake, BigQuery. Every warehouse has its own different metric, but just try to understand what type of materialization, what type of incremental strategy would be best.
57 00:09:46.760 ⇒ 00:09:55.919 Harsh Punjabi: What type of… how can I structure the tables in such a way that the query performance is not impacted? Irrelevant dimensions for each queries are not pulled again and again.
58 00:09:55.920 ⇒ 00:10:01.659 Harsh Punjabi: So I would, like, kind of make sure all of those questions are also answered while building out the final layer, but…
59 00:10:02.110 ⇒ 00:10:04.620 Harsh Punjabi: Essentially, that’s the process I would follow, yeah.
60 00:10:04.950 ⇒ 00:10:21.539 Demilade Agboola: That’s pretty good, that’s pretty good. I’ll just be, like… some of your answers have already touched on some of the questions I plan to ask further on, and so we’ll go… when we get to that point, we’ll actually go into more depth. But yeah, I really like your answer, it’s very thorough, it’s… it speaks to the, like, the…
61 00:10:22.080 ⇒ 00:10:30.619 Demilade Agboola: the process of how you work, and I like that. So I think, like, just kind of following up on that question is…
62 00:10:31.580 ⇒ 00:10:36.219 Demilade Agboola: In our scenario, for instance, you did mention, like, the star schema.
63 00:10:36.320 ⇒ 00:10:43.839 Demilade Agboola: And the normalized schema, snowflake schema. So my question to you would be, when do you use when?
64 00:10:44.130 ⇒ 00:10:54.440 Demilade Agboola: like, when do you use what, and what determines… like, for this project or for this task, I would use, a snowflake schema, and for this task, I would use,
65 00:10:54.590 ⇒ 00:10:55.640 Demilade Agboola: Star schema.
66 00:10:56.280 ⇒ 00:10:59.579 Harsh Punjabi: So, it really depends what our dimensions are based on the dataset.
67 00:10:59.600 ⇒ 00:11:19.100 Harsh Punjabi: on how this decision is made. If I have limited number of dimensions, and most of them have, like, a one-to-one relationship with the fact table, they do not overlap, and they will not impact performance too much, then it’s better to adapt a snowflake schema, but if there are scenarios where there are layers of dimensions involved.
68 00:11:19.100 ⇒ 00:11:37.010 Harsh Punjabi: and the requirement is more complex, and if we have a one-to-one rela… sorry. And specifically in scenarios where, if we try to build out one-on-one relationships between all the DIMS and facts, there could be redundancy and there could be wasted space, then I would opt for a snowflake schema.
69 00:11:37.010 ⇒ 00:11:43.820 Harsh Punjabi: But for the most part, you know, yeah, I mean, that’s what… that’s what my approach would be. The trade-off…
70 00:11:43.820 ⇒ 00:11:45.900 Demilade Agboola: Sorry, you said Snowflake schema twice.
71 00:11:45.900 ⇒ 00:12:05.020 Harsh Punjabi: Oh, sorry. Sorry, my bad. Sorry. Yeah, so if I have, like, if there are less number of dimensions, one-to-one relationship between different dimensions and facts is okay, there is no redundancy, then star schema would be the perfect scenario, but if I’m able to see there’s a lot of redundancy, there’s a lot of…
72 00:12:05.020 ⇒ 00:12:13.839 Harsh Punjabi: you know, multiple pulls that are happening. There are repeated information across two different dimensions. Or,
73 00:12:13.840 ⇒ 00:12:32.060 Harsh Punjabi: or it’s not possible to create a relationship between a dimension and a fact directly. So, in those scenarios, I would go for a snowflake schema, because that would simplify my process a little bit, and that would also ensure that the warehouse performs better, because, again, redundancy is something that I would focus on for the most part.
74 00:12:32.350 ⇒ 00:12:33.110 Demilade Agboola: Okay.
75 00:12:33.280 ⇒ 00:12:38.869 Demilade Agboola: Alright, so in this scenario, let’s say we’ve built out our models.
76 00:12:39.520 ⇒ 00:12:44.550 Demilade Agboola: of push everything into production, but now, over time.
77 00:12:45.660 ⇒ 00:12:50.220 Demilade Agboola: We now have a point where our tables are, like, 400 million rows.
78 00:12:50.370 ⇒ 00:12:50.830 Harsh Punjabi: Yep.
79 00:12:51.630 ⇒ 00:12:53.459 Demilade Agboola: And now it’s getting slow.
80 00:12:54.270 ⇒ 00:12:59.049 Demilade Agboola: Assuming the worst case scenario, how would you optimize this query?
81 00:12:59.850 ⇒ 00:13:00.860 Demilade Agboola: Okay.
82 00:13:01.540 ⇒ 00:13:02.929 Harsh Punjabi: So,
83 00:13:03.080 ⇒ 00:13:15.689 Harsh Punjabi: again, first thing that I would look at is how… I mean, I would first, depending on what the warehouse is, I would first go and see what is causing the inflation. I mean, is it the…
84 00:13:17.110 ⇒ 00:13:35.840 Harsh Punjabi: So first, okay, I’ll tackle it in two ways. One is the query performance per se, and second is the data volume. So first, I would go and check the query performance. How is it working, right? What are the bytes scanned? You know, if it’s Snowflake, then, you know, it’s pretty easy to check the, you know, partition pruning.
85 00:13:35.840 ⇒ 00:13:44.860 Harsh Punjabi: and check other parameters of how the query is being executed to understand where the exact inflation is happening. If it is BigQuery, the information schema is able to provide us enough
86 00:13:44.860 ⇒ 00:13:47.630 Harsh Punjabi: Information on where the inflation is happening.
87 00:13:47.630 ⇒ 00:14:12.290 Harsh Punjabi: And if I can implement something within my dbt or data form models in terms of how to cluster or partition the table for a better or more efficient performance, I would do that. If the problem is too extended, and I’m doing this more recently, if the problem is too extended, then I would try to see how… and if I have the freedom to kind of restructure the table to make sure that, you know, the data volume is not too huge and this problem doesn’t happen, because
88 00:14:12.290 ⇒ 00:14:18.449 Harsh Punjabi: Even the table storage cost is a factor in the long run, right? So, then I would go and restructure the tables.
89 00:14:18.450 ⇒ 00:14:30.759 Harsh Punjabi: Again, that depends on the business case. Sometimes a long table is helpful, so, you know, we would maybe stick with that and, reduce the number of columns and, you know, split those out into different models. Or if, you know, if,
90 00:14:30.760 ⇒ 00:14:39.690 Harsh Punjabi: a wide table is the requirement, you know, a wide table would be a better fit for our use cases. There is not a lot of downstream breakage that might happen. So, so, I mean.
91 00:14:39.690 ⇒ 00:14:57.110 Harsh Punjabi: I’ll just press upon one thing… one concept that I work on first. A long table is good if you have a lot of downstream dependencies which might break, because your schema remains consistent, but a wide table is better if you’re… you don’t have a lot of downstream dependencies which might break, but you want more modularity. So…
92 00:14:57.350 ⇒ 00:15:13.140 Harsh Punjabi: depending on the scenario, I would take a trade-off and pick a long or a wide table, and then, depending on what the scenario is, I might create… I might break down that long table into, different, you know, kind of like a different models which are related to each other.
93 00:15:13.140 ⇒ 00:15:18.019 Harsh Punjabi: So that we don’t have a single table which has too many columns and is very explosive.
94 00:15:18.080 ⇒ 00:15:36.200 Harsh Punjabi: And if that’s not the scenario, then I would try to create a wide table and try to make sure how I can aggregate the data and maybe materialize it as… materialize it as a view. Use aggregated data for, you know, specific downstream requirement where the warehouse doesn’t have to scan all 400 million rows again and again and again.
95 00:15:36.540 ⇒ 00:15:42.210 Demilade Agboola: Okay. What materialization would you use to prevent it from scarring that every single time?
96 00:15:43.120 ⇒ 00:15:46.080 Harsh Punjabi: So, you mean dbt materialization, right?
97 00:15:46.080 ⇒ 00:15:50.980 Demilade Agboola: Yeah. Or how would you materialize such? Like, you’re gonna have to, you know, reveal.
98 00:15:51.410 ⇒ 00:16:07.190 Harsh Punjabi: So, what I would do is I would kind of use SCD type 2, and what I do is, in SCD type 2, I kind of create a flag, current is equal to true, or, you know, maybe row effective and row ineffective timestamp, and then I would kind of use that as a… I mean, I would…
99 00:16:07.280 ⇒ 00:16:17.770 Harsh Punjabi: that would help me cluster my data better, that would help me filter out the information better, that would basically filter out a lot of redundant information in the past. So…
100 00:16:17.910 ⇒ 00:16:31.689 Harsh Punjabi: again, that’s also a trade-off. A full refresh also might work, but a full refresh is at the cost of not storing historized data. If the requirement is that we also have to have historized data, I would use SCD type 2.
101 00:16:31.690 ⇒ 00:16:41.229 Harsh Punjabi: And then, you know, I don’t exactly remember, there is an SCD type which stores historical data in a separate table, and only the latest data in the main table, I think it’s SCD type 4.
102 00:16:41.230 ⇒ 00:16:48.999 Harsh Punjabi: If the need be, I would, you know, kind of use that as well, but basically incremental, like, incremental materialization, and
103 00:16:49.290 ⇒ 00:17:00.289 Harsh Punjabi: you know, then that would define my increment… my incremental strategy. If it’s a CD type rule, then I’ll use merge. Otherwise, you know, there are other options based on what I finally decide, based on the situation, yeah.
104 00:17:00.590 ⇒ 00:17:02.190 Demilade Agboola: Okay, that’s fair, that’s fair.
105 00:17:02.660 ⇒ 00:17:18.439 Demilade Agboola: Alright, so… let’s step aside from the scenario for a quick second. So if we come… if we have a client, and the client says, hey, I want a dashboard.
106 00:17:19.349 ⇒ 00:17:22.260 Demilade Agboola: But they haven’t defined their metrics.
107 00:17:23.339 ⇒ 00:17:26.930 Demilade Agboola: How do you gain the clarity that you need?
108 00:17:27.240 ⇒ 00:17:36.479 Demilade Agboola: in terms of, like, building out the models that you would, need to meet whatever demands they have, because, you know, that happens a lot in consulting, you know? Yep, absolutely.
109 00:17:36.480 ⇒ 00:17:37.060 Harsh Punjabi: separately.
110 00:17:37.250 ⇒ 00:17:38.430 Demilade Agboola: How would you go about that?
111 00:17:38.950 ⇒ 00:17:48.119 Harsh Punjabi: Okay. So, now, so, like, first, approach that I’ve followed, I’ve been in these situations in the past, and the first approach that I’ve tried to follow is.
112 00:17:48.120 ⇒ 00:17:59.219 Harsh Punjabi: speak to the non-technical stakeholders, and try to gain clarity on what they’re exactly looking for. Even if they cannot tell me the exact metrics they want to build out, I would try to first understand their business,
113 00:17:59.450 ⇒ 00:18:07.180 Harsh Punjabi: what is the right word for it? Their rationale behind, you know, what they’re trying to build, what value they’re trying to generate out of it. Based on that.
114 00:18:07.210 ⇒ 00:18:20.069 Harsh Punjabi: normally, like, what I’ve earlier done is, based on that, I would actually suggest a few metrics myself, and then take their feedback, and, you know, obviously, some of the metrics land, some of them, you know, they’re not looking for, so…
115 00:18:20.070 ⇒ 00:18:32.699 Harsh Punjabi: open the conversation from that angle. Like, if I summarize it, understand their, understand their exact requirement, what they’re looking for, build out the metrics from my end, validate with them, and then take their perspective.
116 00:18:32.740 ⇒ 00:18:42.590 Harsh Punjabi: And, you know, then reverse engineer it, and then build out the data models. Otherwise, if that’s completely out of the picture, then focus on the business process, right?
117 00:18:42.870 ⇒ 00:18:50.179 Harsh Punjabi: Because if they want a dashboard, they would be looking for something. Even if they don’t have defined metrics, they would be measuring something, right? So…
118 00:18:50.450 ⇒ 00:19:03.930 Harsh Punjabi: Try to create as granular a model as possible in the first pass, so that most of their dashboard requirements are created, and once their metrics are stabilized, then we would create more aggregation layers to improve the performance.
119 00:19:03.930 ⇒ 00:19:12.009 Harsh Punjabi: If there is a lot of ambiguity. So, in consulting, there is one thing that delivery is very important. They, you know, the clients want to see something tangible.
120 00:19:12.010 ⇒ 00:19:26.540 Harsh Punjabi: So, you know, I would, like, take a more granular, more atomic data as the first pass, so that anything that they want built out, I can build out in the final dashboard layer, and then eventually, based on the clarity, I would create an aggregated layer and migrate the dashboard to that aggregated layer.
121 00:19:26.980 ⇒ 00:19:29.610 Demilade Agboola: Okay, alright, sounds good.
122 00:19:29.820 ⇒ 00:19:34.589 Demilade Agboola: And then this is my final question, and after this, if you have any questions, you can toss them my way.
123 00:19:34.900 ⇒ 00:19:41.559 Demilade Agboola: So that’s say the client that wants fast delivery. They want something, they want quick turnaround times.
124 00:19:41.920 ⇒ 00:19:53.590 Demilade Agboola: But you know that… What… to meet that, there would be a compromise on either… like, scalability…
125 00:19:53.800 ⇒ 00:19:55.980 Demilade Agboola: or quality control.
126 00:19:56.120 ⇒ 00:20:05.409 Demilade Agboola: Or just, basically, you know that technical excellence will be compromised for such a quick turnaround. How do you go about handling that scenario?
127 00:20:06.200 ⇒ 00:20:25.850 Harsh Punjabi: So, normally what I’ve done in such situations is I have, there is a little bit of pushback required in such situations for, you know, of course, because if you want to build out a scalable solution, then you have to do that. But again, you know, every organization is different, every client is different. Depending on how the response is, the first stage is to build an MVP.
128 00:20:25.850 ⇒ 00:20:37.289 Harsh Punjabi: Right? Because if there is a delivery, you have to make some trade-offs, you have to build an MVP. So, again, that’s… it’s pretty subjective in a lot of situations, depending on what the priority is. If the priority is only to save cost.
129 00:20:37.320 ⇒ 00:20:54.599 Harsh Punjabi: then I would compromise on scalability and, you know, optimize for cost and build something, build an MVP which would be good enough for them. If the priority is scalability, cost is not too much of an object, the data volume is not too high, they don’t care about 4,000, $5,000 of difference in expenses, then I would focus on scalability, compromise on cost.
130 00:20:54.600 ⇒ 00:21:07.399 Harsh Punjabi: essentially make a trade-off based on the subjectivity of the situation, but I would still call it an MVP and keep refining it, you know, once the main first draft is delivered and their initial requirement is done. I would…
131 00:21:07.400 ⇒ 00:21:16.740 Harsh Punjabi: like, optimize it for scalability, cost, and everything in the longer run, either way, right? Once the main, like, first draft is delivered, or the MVP is delivered.
132 00:21:17.420 ⇒ 00:21:26.249 Demilade Agboola: Okay, alright. Thank you, I don’t have any follow-up questions, so if you have any questions, feel free to send them my way. If not, we can call it a day.
133 00:21:26.250 ⇒ 00:21:38.569 Harsh Punjabi: Sure, absolutely. I have, I mean, I’ll go with more of an, you know, like, your opinion kind of question first, and then I’ll jump into, you know, like, brain forge or, you know, process-related questions. So,
134 00:21:38.620 ⇒ 00:21:52.690 Harsh Punjabi: in your opinion, and based on your experience with Brainforge, what does success in this role, you know, look like in the first 90 days, and then in the first year? Like, how do you… how would you define a successful candidate in this role?
135 00:21:53.380 ⇒ 00:22:09.149 Demilade Agboola: I think a successful candidate in this role, number one, is someone who has the technical expertise, obviously, who can come in, who can be able to deliver good quality code. Two.
136 00:22:09.400 ⇒ 00:22:28.969 Demilade Agboola: can utilize AI, you know? He knows how to leverage AI and knows how to be able to use that to be able to speed up their work without compromising on quality. And three would also be someone who can communicate with the client, right? So sometimes we want the gap between the technical team and the
137 00:22:28.970 ⇒ 00:22:34.099 Demilade Agboola: the people we’re delivering to, the business stakeholders, we want to reduce that gap.
138 00:22:34.100 ⇒ 00:22:49.879 Demilade Agboola: And sometimes we have found… we’ve tried different things, because I’ve been in Brainford for a little about one year. We’ve tried different things, and one of the things we’ve realized are engineers who are capable of interacting with the business stakeholders provide more impact for us, and also put less pressure
139 00:22:49.960 ⇒ 00:23:08.569 Demilade Agboola: on people having to be a bridge to the stakeholders. So, yes, success in this role would be, good technical quality, good ability with stakeholders, and I would say, like, finally, a problem-solving, like, mentality. Someone who…
140 00:23:08.660 ⇒ 00:23:14.390 Demilade Agboola: When the problems come, They are able to think through the problem.
141 00:23:15.260 ⇒ 00:23:17.800 Demilade Agboola: Understand what the problem actually is.
142 00:23:18.250 ⇒ 00:23:20.520 Demilade Agboola: Come up with a plan and a solution.
143 00:23:20.850 ⇒ 00:23:26.569 Demilade Agboola: communicate to the general team what they’re trying to do, and then attack it, right? So,
144 00:23:28.020 ⇒ 00:23:46.629 Demilade Agboola: there’s a level of, like, competence, you know? We don’t want people who necessarily were going to be hand-holding or, you know, asking questions about, hey, how about you solving this way? I mean, obviously, we’re always here to brainstorm together, but we also want people who are able to think by themselves and come up with solutions by themselves.
145 00:23:46.630 ⇒ 00:23:48.150 Demilade Agboola: And, you know.
146 00:23:48.510 ⇒ 00:24:01.749 Demilade Agboola: bring solutions to the common part where we’re all talking about how we want to solve the problem, you know? This isn’t a true role where we just want to be like, hey, this is your ticket, this is your ticket, we want someone who actually brings something to the table.
147 00:24:02.890 ⇒ 00:24:14.109 Harsh Punjabi: really helpful. Thank you so much. The next question that I have, again, but first of all, thank you so much for the previous answer. It clarified things a lot for me and, you know, gave me a lot of perspective.
148 00:24:14.110 ⇒ 00:24:27.910 Harsh Punjabi: My second question is, I know, like, Brainforge has a bunch of clients, and that they’re all from different industries, you know, I confirmed… I confirmed this with Avish in my previous, interview. What I’m… what I want to understand a bit more is,
149 00:24:28.150 ⇒ 00:24:45.020 Harsh Punjabi: for the analytics engineering role specifically, what tech stack is being used across different clients? I know dbt Snowflake, these are all pretty standard, but, you know, there could be some variability, so I’m just trying to understand if there are different tools which are being used, or is it pretty consistent across the board? How does that work?
150 00:24:45.600 ⇒ 00:24:47.890 Demilade Agboola: I would say we are…
151 00:24:48.300 ⇒ 00:25:05.280 Demilade Agboola: With dbt, I mean, transformation layer is pretty consistent. We use dbt for almost every scenario. There are some smaller clients that we’ve just used, like, the transformation happens in Omni. Like, they had really, really small data, and they just needed to have something out the door, so we did transformations within Omni. That’s…
152 00:25:05.280 ⇒ 00:25:16.970 Demilade Agboola: few and far between. Most times, we just use dbt for transformation. In terms of warehouses, yeah, that’s where the variability exists. So in some cases, we will use…
153 00:25:17.250 ⇒ 00:25:22.079 Demilade Agboola: what the client has already. So, some clients will come, and they already have BigQuery.
154 00:25:22.430 ⇒ 00:25:29.150 Demilade Agboola: or they’re already on Snowflake, or whatever. And in case… in situations where they don’t necessarily have any, like.
155 00:25:29.380 ⇒ 00:25:37.470 Demilade Agboola: warehouse already. We use… we try to look at what the entire infrastructure is, like, some people are already on, like,
156 00:25:37.880 ⇒ 00:25:57.219 Demilade Agboola: AWS, so it’s, like, easier to just, you know, since you’re in there, Redshift might work much better for you. But also, we recommend things, like, we try, like, Mother Doc. Mother Doc is pretty good. We recommend it to our clients. So yes, there is variability in terms of, like, the warehouse. In terms of transformation, it’s mostly dbt.
157 00:25:57.390 ⇒ 00:26:03.969 Demilade Agboola: And, one of the things we have been testing and trying to find, like, what we consistently will lean on.
158 00:26:04.250 ⇒ 00:26:19.009 Demilade Agboola: for observability is, like, we’ve tried different things, but right now, I think we’re leaning towards, like, metaplane consistently, a bit more consistently. But yes, there is a lot of, like, variability, but also some consistency, depending on the area.
159 00:26:19.540 ⇒ 00:26:34.989 Harsh Punjabi: That’s really helpful. And I think dbt being pretty universal is kind of helpful, because… I mean, I’ll be honest with you, I did not really… I mean, personally, with the data form migration that has been done, really, I feel dbt was much better data form
160 00:26:35.010 ⇒ 00:26:40.120 Harsh Punjabi: is very… like, it does… DataForm is not very good at giving lineage tracking.
161 00:26:40.120 ⇒ 00:26:56.669 Harsh Punjabi: it’s not very good at developing macros and, you know, reusing them. So, I personally feel dbt is a game changer in analytics engineering, and if it’s pretty consistent, then that’s great. And of course, like, warehouses… warehouse requirements vary from, you know, even
162 00:26:56.720 ⇒ 00:27:05.480 Harsh Punjabi: the type of client that you’re dealing with, so that’s understandable. I just have one final question. So, you did mention that, you know.
163 00:27:05.520 ⇒ 00:27:24.700 Harsh Punjabi: you know, there is a lot of AI usage, and using AI to, you know, be more efficient and, you know, ensuring accuracy and high-quality work is one part of the role here. So I was just curious, what type of AI integrations are, like, being used? Are you using Copilot with VS Code? Are you using Cloud Co-work?
164 00:27:24.700 ⇒ 00:27:31.009 Harsh Punjabi: Or open code, like, what, what, level of integration is being used as of now?
165 00:27:31.320 ⇒ 00:27:41.169 Demilade Agboola: So right now, as things currently stand, we are using Cursor, so we have Cursor set up across everyone, to be honest.
166 00:27:42.280 ⇒ 00:27:54.170 Demilade Agboola: So, I mean, I mean everyone, I mean everyone. So, the data infrastructure team, the data modeling team, the AI team, the sales team, the marketing team, and we try our best to also allow for
167 00:27:58.200 ⇒ 00:28:17.760 Demilade Agboola: like, we try our best to, like, store data that allows our AI to be the most efficient in terms of just, like, general usage. And in, like, this role in particular, yes, we also, like, have, tokens that the entire team has access to, and then you can use that in terms of, like, your day-to-day,
168 00:28:18.230 ⇒ 00:28:29.849 Demilade Agboola: as you’re working on clients, as you’re working on different projects, as you’re, you know, creating documentation, we strongly always, like, the team strongly recommends that you lean on AI.
169 00:28:29.900 ⇒ 00:28:43.190 Demilade Agboola: And we also, like, have integrations, like MCPs and stuff, so you’re able to have, like, things to Notion, you have MCPs to Linear, so Linear’s our project management tool.
170 00:28:43.270 ⇒ 00:28:56.370 Demilade Agboola: We set it up for, like, Mother Doc, so you can also use that to, like, query warehouses and use that to, validate, for instance, your, query. If you’ve done a dbt run, you can say, hey, look into.
171 00:28:56.460 ⇒ 00:29:05.189 Demilade Agboola: the RAW, compare it to the prod, and let me… sorry, not raw. Look into the dev, compare it to the prod, and show me if, like.
172 00:29:05.420 ⇒ 00:29:20.239 Demilade Agboola: we have a data explosion. Was there a bad join? Is the level of data still the same? And you can start doing all those quality checks before you finally, like, you know, send your PR in. So yeah, we have a bunch of that. We have, like, Slack integrations.
173 00:29:20.240 ⇒ 00:29:25.700 Demilade Agboola: As well, so we’ve built skills, like, for instance, one of the skills I built was, like, a…
174 00:29:26.310 ⇒ 00:29:44.759 Demilade Agboola: daily to-do, briefing scale, so if you run it every morning, it will go through, like, your Slack integration, go through your linear tickets, go through a bit of Notion, go through your Gmail, your calendar, and everything, and kind of just gives you a to-do list for what you have to do that day, and that allows you to be, like.
175 00:29:45.550 ⇒ 00:29:53.750 Demilade Agboola: clear on what you need to do every day. But yeah, we just, like, strongly, like, lean on that, and we strongly recommend it across the company, yeah.
176 00:29:54.160 ⇒ 00:30:11.489 Harsh Punjabi: That’s just really… that’s really great to know, actually, that’s very, very insightful. Again, based on this, I just had a follow-up question. I won’t take too much of your time, this is just one last question. So, I was, like, as a personal project, just for upskilling, I was tinkering with something, around conversational analytics, so…
177 00:30:11.490 ⇒ 00:30:17.109 Harsh Punjabi: you know, building out a semantic layer for Cortex Analyst in Snowflake, and then I kind of figured out how to do that for free.
178 00:30:17.110 ⇒ 00:30:41.830 Harsh Punjabi: by using, like, Gemini API and building out, like, a text-to-SQL kind of tool using Vanna AI. I was just trying to see what free tools are available out there, and then the basic, like, I think Flash 3.0 model is… the API usage is free for, you know, some specific usage amount in Gemini, and I kind of integrated that, and then, you know, created a semantic layer using dbt, and then built out, like, a text-to-SQL
179 00:30:42.070 ⇒ 00:30:59.549 Harsh Punjabi: tool out, and that will leverage the schema.yaml within dbt as the documentation. So is there any work going around in conversational analytics around? I am… I was very curious, because it’s very hard to implement that. Anybody else that I speak to, you know, who’s in the analytics engineering space.
180 00:30:59.550 ⇒ 00:31:09.170 Harsh Punjabi: They’re saying it’s not really taking off too much, like, very few startups kind of are doing it, but have you seen any traction on conversational analytics taking off anytime soon, or is there any kind of work there?
181 00:31:09.740 ⇒ 00:31:17.889 Demilade Agboola: So, it’s something that we’ve considered. Unfortunately, we’ve not necessarily had the bandwidth to be able to tackle that.
182 00:31:17.890 ⇒ 00:31:33.679 Demilade Agboola: So what we’ve kind of done is, in terms of, like, conversational analytics, we rely heavily on Omni, so I don’t know if you use Omni, the BI tool. So Omni allows you to create, like, topics, and you can provide AI context in there as well.
183 00:31:34.120 ⇒ 00:31:51.309 Demilade Agboola: And so now, we leverage on that when we give clients, like, their data within Omni, so it’s part of why we recommend Omni to clients, generally speaking. And so we can say, hey, there are certain questions, like, we’ve had clients come and meet us and ask us, what’s the total number of sales we had last year?
184 00:31:51.410 ⇒ 00:31:53.850 Demilade Agboola: This… is this not stuff that…
185 00:31:53.910 ⇒ 00:32:15.129 Demilade Agboola: you need to be messaging us about, or… and if the dashboard isn’t giving you the answers you need, just ask Omni, and Omni can go, hey, based off of, like, the topics that have been created, this is the number of sales you had last year. And then you can also ask for a lot of questions, or can you remove the pending orders, or abandoned orders, or can you ensure it’s only, like, the completed orders, things like that, and…
186 00:32:15.130 ⇒ 00:32:21.870 Demilade Agboola: That’s kind of what we lean on, in terms of that. In terms of, like, in-house, like, our own…
187 00:32:21.870 ⇒ 00:32:23.870 Demilade Agboola: tool we have when you have built one.
188 00:32:24.030 ⇒ 00:32:26.479 Demilade Agboola: Even though he has come up in conversations, yeah.
189 00:32:26.970 ⇒ 00:32:30.369 Harsh Punjabi: That’s just really helpful to know. And, yeah, because,
190 00:32:30.800 ⇒ 00:32:41.509 Harsh Punjabi: Actually, yeah, that’s really insightful, and one thing that I’m really happy about Omni, even though I’ve not used Omni before, I’ll be very honest with you, I’ve kind of just tinkered around with it, I’ve understood that a little bit.
191 00:32:41.510 ⇒ 00:32:54.470 Harsh Punjabi: But, what’s really interesting to me is, most of the tech stack in analytics engineering ends at Looker, and I find Looker to be very tedious. I have never… I’ve worked on Looker for more than 5 years now, and I’ve never liked it once.
192 00:32:54.470 ⇒ 00:33:07.730 Demilade Agboola: I’m also the same. I find it very… I find it very tedious. I think it’s very unnecessary, and this is just my own popular opinion, because people that love Looker love Looker, but I find it very tedious. Like, it feels like…
193 00:33:08.160 ⇒ 00:33:26.870 Demilade Agboola: a duplicate layer of dbt, because it’s like, we’re already doing so much of the transformation here, and then you come into Looker, and we’re doing so much transformation here, and now, when things break, you have to go through two different sources to kind of figure out where things are breaking. It just feels so unnecessary, and I think it’s…
194 00:33:27.150 ⇒ 00:33:32.299 Demilade Agboola: people that, again, people that love it, love it, but I’m not, I’m not the biggest fan of Looker.
195 00:33:33.200 ⇒ 00:33:45.460 Harsh Punjabi: me as well. I mean, I’ve worked on a PDT to DBT migration once, like, a year and a year… one year ago. Yeah, I mean, you can imagine, right?
196 00:33:45.460 ⇒ 00:33:57.979 Demilade Agboola: Yeah, I agree, I agree, yeah. I’m not, again, looker, it’s not, it’s not… I mean, you do what you have to do, because especially if people are on contracts and they can’t, like, leave their contracts, yeah, but I feel like…
197 00:33:58.220 ⇒ 00:34:10.500 Demilade Agboola: sometimes it’s just people rely on too much logic within, Looker, and then if you aren’t using Looker for the logic and, like, the standardization of how you want to put it there, then…
198 00:34:10.679 ⇒ 00:34:25.319 Demilade Agboola: what’s the point? So it’s kind of like… you… if you’re using Looker, you have to use it, because that’s part of why you’re paying so much money for it. If not, you would just use a regular, just, visualization tool. But when you then have it.
199 00:34:25.550 ⇒ 00:34:29.319 Demilade Agboola: it’s like, okay, so you have DBT, and you have this.
200 00:34:29.449 ⇒ 00:34:33.950 Demilade Agboola: It just feels like too much, and it feels like you need one single source of truth.
201 00:34:34.090 ⇒ 00:34:37.870 Demilade Agboola: And yeah, it’s just that, like, tricky balance to find.
202 00:34:37.870 ⇒ 00:34:44.190 Harsh Punjabi: Yeah, absolutely. No, but I’m happy to, you know, I’m happy to see, you know, there’s somebody else out there who’s resonating.
203 00:34:44.190 ⇒ 00:34:54.789 Demilade Agboola: Don’t worry, if you ever need… if you ever need someone who can back you up in a conversation, just tag me, I’ll come in and let them know.
204 00:34:54.790 ⇒ 00:34:55.610 Harsh Punjabi: Exactly, thank you so much.
205 00:34:56.889 ⇒ 00:34:58.809 Demilade Agboola: Alright, Ben, take care of yourself.
206 00:34:59.119 ⇒ 00:35:05.039 Demilade Agboola: Alright, thank you, Debbie. I’ll respond to Kayla, give her my thoughts on this call, and she’ll be in touch.
207 00:35:05.410 ⇒ 00:35:08.540 Harsh Punjabi: Alright, absolutely. Thank you so much. Had a great conversation today. Thank you.
208 00:35:08.760 ⇒ 00:35:09.569 Demilade Agboola: Alright, bye.
209 00:35:09.570 ⇒ 00:35:10.240 Harsh Punjabi: Babe.