Meeting Title: Brainforge Interview w- Demilade Date: 2026-04-15 Meeting participants: Yeshwanth C Mineni, Demilade Agboola
WEBVTT
1 00:02:41.640 ⇒ 00:02:53.880 Demilade Agboola: Hello, hello! Hi, my name is Demiladeh. Nice to meet you. Sorry about yesterday. I canceled because I had, quite… I had to see the doctor on very short notice.
2 00:02:54.250 ⇒ 00:03:03.400 Demilade Agboola: So I had to cancel the call. I believe you saw it, but I didn’t know you hadn’t seen it, so apologies for yesterday.
3 00:03:07.140 ⇒ 00:03:08.520 Yeshwanth C Mineni: No worries, actually.
4 00:03:09.230 ⇒ 00:03:09.920 Demilade Agboola: That’s fine.
5 00:03:10.200 ⇒ 00:03:14.909 Demilade Agboola: My name is Timadhi, I work with the Brain4 team. I’m a Senior Analytics Engineer on the team.
6 00:03:15.780 ⇒ 00:03:19.830 Demilade Agboola: I’m not exactly sure how I’ll pronounce your name, please can you help me with that?
7 00:03:20.260 ⇒ 00:03:22.310 Yeshwanth C Mineni: You can call me Yash, that’s fine.
8 00:03:22.940 ⇒ 00:03:23.590 Demilade Agboola: Okay.
9 00:03:23.730 ⇒ 00:03:25.970 Demilade Agboola: Nice to meet you, Yash.
10 00:03:25.970 ⇒ 00:03:26.510 Yeshwanth C Mineni: Thank you.
11 00:03:26.510 ⇒ 00:03:30.409 Demilade Agboola: So I see that you’re applying for the data engineer role?
12 00:03:31.130 ⇒ 00:03:31.690 Yeshwanth C Mineni: Yes.
13 00:03:31.690 ⇒ 00:03:40.330 Demilade Agboola: Until the call… this call will just be focused on basically understanding how you… Work.
14 00:03:40.980 ⇒ 00:03:46.680 Demilade Agboola: How you prototype and design systems from a, you know, Data engineering perspective.
15 00:03:46.970 ⇒ 00:03:59.340 Demilade Agboola: So there won’t be any live coding, but it’ll just be… I’ll be asking you to walk me through scenarios, how you’ll troubleshoot, how you’ll build, and how you’ll be able to, you know, design those systems and architect the systems.
16 00:04:00.190 ⇒ 00:04:00.879 Demilade Agboola: Oh, we haven’t.
17 00:04:00.880 ⇒ 00:04:01.610 Yeshwanth C Mineni: Makes sense.
18 00:04:01.960 ⇒ 00:04:03.030 Yeshwanth C Mineni: Yes, definitely.
19 00:04:03.030 ⇒ 00:04:03.610 Demilade Agboola: Okay.
20 00:04:03.730 ⇒ 00:04:14.319 Demilade Agboola: Alright, so let’s just start off simple with just, like, can you tell me about yourself and your experience, and just generally what kind of, like, what your tech stack is, and, you know, what you work with?
21 00:04:16.469 ⇒ 00:04:21.429 Yeshwanth C Mineni: My most recent role, I was working as a data engineer with Upstart.
22 00:04:21.829 ⇒ 00:04:24.769 Yeshwanth C Mineni: when I was, using, Python, Spark.
23 00:04:24.770 ⇒ 00:04:28.049 Demilade Agboola: Your audio sounds a bit off, I’m not sure why.
24 00:04:29.060 ⇒ 00:04:31.100 Yeshwanth C Mineni: Oh, okay, one more.
25 00:04:32.040 ⇒ 00:04:32.800 Demilade Agboola: Much better.
26 00:04:35.170 ⇒ 00:04:37.760 Yeshwanth C Mineni: Let me change my microphone one moment.
27 00:04:39.080 ⇒ 00:04:40.480 Yeshwanth C Mineni: Can you hear it now?
28 00:04:41.030 ⇒ 00:04:42.209 Demilade Agboola: Yes, yes, I can hear you.
29 00:04:43.430 ⇒ 00:04:47.770 Yeshwanth C Mineni: So, as I was saying, like, I was a data engineer with, Upstart.
30 00:04:48.140 ⇒ 00:04:52.940 Yeshwanth C Mineni: And, whereas I’m using Python, Spark, and also SQL.
31 00:04:53.300 ⇒ 00:04:59.940 Yeshwanth C Mineni: And Airflow as well, for building, like, pipelines and orchestrating data pipelines on a daily basis.
32 00:05:00.270 ⇒ 00:05:05.839 Yeshwanth C Mineni: And, I did use, cloud platforms such as Azure and AWS.
33 00:05:06.180 ⇒ 00:05:16.209 Yeshwanth C Mineni: Where I was using services such as ADF, which is Azure Data Factory, Databricks, Redshift, and Kafka for streaming.
34 00:05:18.070 ⇒ 00:05:29.639 Yeshwanth C Mineni: So, our team, basically, data engineering team works with several verticals that we have. Like, we have our data science teams, where we have several data scientists.
35 00:05:29.750 ⇒ 00:05:39.419 Yeshwanth C Mineni: And also, we have ML teams where we have machine learning engineers working on our ML models. So, basically, our firm, it’s a financial services firm.
36 00:05:39.550 ⇒ 00:05:45.729 Yeshwanth C Mineni: They do, credit lending based on customer portfolios, based on, machine learning algorithms.
37 00:05:45.930 ⇒ 00:05:47.780 Yeshwanth C Mineni: So, they…
38 00:05:47.890 ⇒ 00:05:54.320 Yeshwanth C Mineni: So, they have several data streams, and they do have several data vendors as well for their data needs.
39 00:05:54.810 ⇒ 00:05:59.719 Yeshwanth C Mineni: So, this is, the stack, typically, that I was using.
40 00:06:00.470 ⇒ 00:06:01.869 Yeshwanth C Mineni: Actually, it’s true.
41 00:06:02.930 ⇒ 00:06:04.899 Demilade Agboola: Just curious.
42 00:06:05.660 ⇒ 00:06:13.649 Demilade Agboola: Question about how deep into the data, like, you did the ingestion, Did you do the transformation?
43 00:06:14.160 ⇒ 00:06:19.329 Demilade Agboola: Like, at what point did you hand over to someone else? Like, how much did you handle in that process?
44 00:06:20.580 ⇒ 00:06:26.540 Yeshwanth C Mineni: So, I’ve handled, like, the full lifecycle, I would say, for the data pipeline management.
45 00:06:26.700 ⇒ 00:06:32.820 Yeshwanth C Mineni: Like, I designed, and I also deployed ETL pipelines and ELT pipelines as well.
46 00:06:33.140 ⇒ 00:06:35.929 Yeshwanth C Mineni: Depends on the particular, situation.
47 00:06:36.090 ⇒ 00:06:39.239 Yeshwanth C Mineni: And I’ve done data ingestion transformations.
48 00:06:39.370 ⇒ 00:06:41.350 Yeshwanth C Mineni: like, using Spark.
49 00:06:41.710 ⇒ 00:06:48.770 Yeshwanth C Mineni: In Databricks. And also, I’ve, I’ve worked on, non-relational databases as well.
50 00:06:49.370 ⇒ 00:06:57.729 Yeshwanth C Mineni: And, and, I’ve developed, like, ML-ready feature pipelines, I would say.
51 00:06:57.850 ⇒ 00:07:01.500 Yeshwanth C Mineni: That’s the most interesting aspect of work I’ve done.
52 00:07:01.980 ⇒ 00:07:09.620 Yeshwanth C Mineni: And I’ve, also, I’ve also done, like, automation in the delivery as well.
53 00:07:09.890 ⇒ 00:07:13.150 Yeshwanth C Mineni: like, I’ve,
54 00:07:13.800 ⇒ 00:07:20.960 Yeshwanth C Mineni: And I also have, implemented, monitoring also, like, for any event logs or error handling as well.
55 00:07:21.570 ⇒ 00:07:25.630 Yeshwanth C Mineni: So, this is something that I have done end-to-end.
56 00:07:26.750 ⇒ 00:07:27.890 Demilade Agboola: Okay, sounds good.
57 00:07:28.270 ⇒ 00:07:41.050 Demilade Agboola: Just a follow-up question on that. What would you say the most complex pipeline you’ve worked on? This is not, you know, the most complex pipeline you’ve worked on, and what made it complex and complicated?
58 00:07:43.270 ⇒ 00:07:49.809 Yeshwanth C Mineni: Okay, so… Coming to the most complex situation was with Upstarto.
59 00:07:49.920 ⇒ 00:07:52.070 Yeshwanth C Mineni: when I was working with Upstart only.
60 00:07:52.390 ⇒ 00:07:59.329 Yeshwanth C Mineni: Where I had to migrate, like, legacy batch pipelines to real-time streaming pipelines.
61 00:07:59.690 ⇒ 00:08:05.809 Yeshwanth C Mineni: Like, where I have to use Kafka and Spark.
62 00:08:06.080 ⇒ 00:08:12.479 Yeshwanth C Mineni: Initially, when I was onboarded, it was, pretty critical for me, because I never had such exposure.
63 00:08:12.850 ⇒ 00:08:17.869 Yeshwanth C Mineni: Like, I was trying to integrate through multiple data sources.
64 00:08:18.230 ⇒ 00:08:21.510 Yeshwanth C Mineni: like, including non-relational APIs as well.
65 00:08:22.000 ⇒ 00:08:33.970 Yeshwanth C Mineni: Because we have several financial data sets, like, and we had to, check for data consistency and synchronization across, like, all of our microservices that we have.
66 00:08:34.510 ⇒ 00:08:39.880 Yeshwanth C Mineni: And, we need to ensure that we have only around, like, 5 seconds left, so…
67 00:08:40.020 ⇒ 00:08:46.360 Yeshwanth C Mineni: That’s something complex that I’ve managed to achieve in the initial days, I would say.
68 00:08:46.960 ⇒ 00:08:57.759 Yeshwanth C Mineni: And, and, validation checks part also was something, complicated, because I was dealing with,
69 00:08:57.980 ⇒ 00:09:07.139 Yeshwanth C Mineni: a lot of latency requirements, and also, I have a legacy batch pipeline. That’s something complex to monitor, and
70 00:09:07.630 ⇒ 00:09:16.509 Yeshwanth C Mineni: And also, on top of it, as this includes, like, end-to-end lifecycle, because I was not managing a particular portion of it.
71 00:09:16.650 ⇒ 00:09:19.600 Yeshwanth C Mineni: I had to manage the CICD deployment as well.
72 00:09:20.330 ⇒ 00:09:27.580 Yeshwanth C Mineni: And, this… this, added, like, the operational complexity, I would say, but, like, Oh.
73 00:09:28.310 ⇒ 00:09:32.379 Yeshwanth C Mineni: But it ensured, like, smooth releases throughout our production.
74 00:09:34.060 ⇒ 00:09:35.360 Demilade Agboola: Okay, alright, fair enough.
75 00:09:35.360 ⇒ 00:09:35.820 Yeshwanth C Mineni: Yeah.
76 00:09:35.820 ⇒ 00:09:36.360 Demilade Agboola: Cheers.
77 00:09:36.790 ⇒ 00:09:39.060 Demilade Agboola: One of those things that can be very tricky.
78 00:09:39.510 ⇒ 00:09:44.379 Demilade Agboola: Switching from just batch stream… batch to stream, to streaming data.
79 00:09:45.430 ⇒ 00:09:45.810 Yeshwanth C Mineni: Yeah.
80 00:09:47.490 ⇒ 00:09:52.659 Demilade Agboola: Alright, so this is, like, our first, like… Scenario, system-building question.
81 00:09:53.150 ⇒ 00:09:58.880 Demilade Agboola: You can take, like, a minute or two if you need to think about it, but basically.
82 00:09:59.210 ⇒ 00:10:04.730 Demilade Agboola: So if we have a client that wants a daily revenue, Reporting.
83 00:10:04.840 ⇒ 00:10:06.230 Demilade Agboola: Mott, right?
84 00:10:06.860 ⇒ 00:10:11.910 Demilade Agboola: And they have 3 different data sources. They have Salesforce, they have Stripe.
85 00:10:12.440 ⇒ 00:10:14.689 Demilade Agboola: And then they also have Google Ads.
86 00:10:15.600 ⇒ 00:10:19.279 Demilade Agboola: And you can do it into any cloud warehouse of your choice.
87 00:10:19.630 ⇒ 00:10:28.040 Demilade Agboola: how would you go about designing it? So, walk me through your considerations, walk me through whatever questions you would want to ask.
88 00:10:28.420 ⇒ 00:10:31.940 Demilade Agboola: And whatever strategies you go about to,
89 00:10:33.670 ⇒ 00:10:35.730 Demilade Agboola: Get the data that you want.
90 00:10:36.820 ⇒ 00:10:39.270 Yeshwanth C Mineni: Yes, let me think about it.
91 00:10:39.640 ⇒ 00:10:43.590 Demilade Agboola: No, that’s fine, like I said, you have, like, a minute or two. You don’t have to respond immediately.
92 00:10:45.740 ⇒ 00:10:48.119 Yeshwanth C Mineni: I would need some,
93 00:10:48.920 ⇒ 00:10:52.810 Yeshwanth C Mineni: Reporting requirements, like, exact reporting requirements, like…
94 00:10:53.240 ⇒ 00:10:58.310 Yeshwanth C Mineni: Any dimensions, or, like, any revenue metrics that you have, or, like.
95 00:10:59.010 ⇒ 00:11:04.639 Yeshwanth C Mineni: Or any latency requirements that they have, or… I would go by…
96 00:11:04.770 ⇒ 00:11:09.350 Yeshwanth C Mineni: Are there any, data quality or compliance requirements?
97 00:11:09.830 ⇒ 00:11:16.029 Yeshwanth C Mineni: And, I would check for data volume and growth. So these are the questions I would typically…
98 00:11:16.180 ⇒ 00:11:17.600 Yeshwanth C Mineni: look around for…
99 00:11:18.660 ⇒ 00:11:20.040 Demilade Agboola: Okay,
100 00:11:20.610 ⇒ 00:11:27.930 Demilade Agboola: Alright, so let’s start knocking off latency. So, for instance, latency, they need it every morning, right? So, you don’t need it to be real time.
101 00:11:28.500 ⇒ 00:11:30.830 Demilade Agboola: They’re fine with the 24-hour cycle.
102 00:11:31.430 ⇒ 00:11:36.060 Demilade Agboola: Sorry, I didn’t… I wasn’t… like, what other questions did you…
103 00:11:37.430 ⇒ 00:11:42.099 Yeshwanth C Mineni: Like, what kind of dimensions that they need, or any compliance requirements.
104 00:11:42.660 ⇒ 00:11:45.639 Demilade Agboola: So compliance, we just need to…
105 00:11:45.910 ⇒ 00:11:48.750 Demilade Agboola: obviously not have PII in the data.
106 00:11:49.630 ⇒ 00:11:58.470 Demilade Agboola: So… any, like, Anything around things about the emails, or…
107 00:11:59.360 ⇒ 00:12:04.319 Demilade Agboola: Yeah, anything around, like, emails and anything PII related, we need to get them out.
108 00:12:05.020 ⇒ 00:12:07.940 Demilade Agboola: Phone numbers as well. Let’s get that out.
109 00:12:08.170 ⇒ 00:12:18.349 Demilade Agboola: And then, in terms of, like, dimensions, well, like, ultimately, they just want to be able to see the reporting data, so they want to be able to see, like, how much we sold.
110 00:12:18.620 ⇒ 00:12:28.140 Demilade Agboola: Like, yesterday, we want to see how many, like, how the ads led to conversion, things around that sort of nature, so…
111 00:12:28.300 ⇒ 00:12:29.960 Demilade Agboola: Yeah.
112 00:12:31.690 ⇒ 00:12:33.270 Yeshwanth C Mineni: Sounds good.
113 00:12:34.330 ⇒ 00:12:46.600 Yeshwanth C Mineni: So… For, P2 removal, I would, implement, like, data masking or, Anonymization techniques.
114 00:12:47.060 ⇒ 00:12:56.549 Yeshwanth C Mineni: like, during the ETL itself, like, I would exclude any, emails or phone numbers or any other fields in that particular section.
115 00:12:56.820 ⇒ 00:13:00.479 Yeshwanth C Mineni: I would use, hashing or tokenization.
116 00:13:00.660 ⇒ 00:13:03.590 Yeshwanth C Mineni: If any identifiers are needed for joints.
117 00:13:03.760 ⇒ 00:13:07.969 Yeshwanth C Mineni: But, must remain on… if… I mean, like.
118 00:13:08.090 ⇒ 00:13:13.510 Yeshwanth C Mineni: if they must remain, like, non-identifiable, I would use hashing or tokenization.
119 00:13:13.890 ⇒ 00:13:22.730 Yeshwanth C Mineni: And, validate… I would do validation checks that, no, P2 level fields are stored in the data mart, or exposed in the reports.
120 00:13:23.310 ⇒ 00:13:29.619 Yeshwanth C Mineni: And I would apply, like, any access controls, and also encryption, like, in the transit.
121 00:13:30.150 ⇒ 00:13:33.719 Yeshwanth C Mineni: And, addressed as well, to ensure compliance.
122 00:13:34.020 ⇒ 00:13:42.560 Yeshwanth C Mineni: For dimensions and metrics, I would, create, like, fact tables, capturing, like, daily, sales revenues and conversion events.
123 00:13:43.000 ⇒ 00:13:54.660 Yeshwanth C Mineni: And for… to build the dimension, I would go by the date, the product, the campaign, and the channel to enable the slicing and dicing of the particular data that we need.
124 00:13:55.380 ⇒ 00:13:59.640 Yeshwanth C Mineni: And also, I would, include campaign attribution fields.
125 00:14:00.190 ⇒ 00:14:03.860 Yeshwanth C Mineni: To link and spend on, conversions.
126 00:14:04.180 ⇒ 00:14:07.930 Yeshwanth C Mineni: And, I would do a schema validation checks as well.
127 00:14:08.120 ⇒ 00:14:15.219 Yeshwanth C Mineni: Which will, create, like, easy aggregations of revenue by day, or,
128 00:14:15.790 ⇒ 00:14:20.410 Yeshwanth C Mineni: We can, we can, we can go by campaign as well, and also…
129 00:14:20.750 ⇒ 00:14:23.460 Yeshwanth C Mineni: Also, like, any other relevant dimensions.
130 00:14:23.760 ⇒ 00:14:28.780 Yeshwanth C Mineni: And I would also develop a data model to support, like, quick queries.
131 00:14:28.930 ⇒ 00:14:34.100 Yeshwanth C Mineni: For, like, yesterday’s sales, or conversion metrics, something like that.
132 00:14:34.420 ⇒ 00:14:39.560 Yeshwanth C Mineni: And, I would, I would,
133 00:14:40.040 ⇒ 00:14:45.260 Yeshwanth C Mineni: Ensure that the pipeline includes, like, the validation checks on a regular basis.
134 00:14:46.160 ⇒ 00:14:46.620 Demilade Agboola: Okay.
135 00:14:46.620 ⇒ 00:14:56.389 Yeshwanth C Mineni: And… and, and, just to confirm the… it’s P2 compliant, you know, I mean, like…
136 00:14:56.640 ⇒ 00:15:06.270 Yeshwanth C Mineni: And remove the not-needed data, and, like, I would, check on the data accuracy as well, before loading to the data mart.
137 00:15:07.220 ⇒ 00:15:12.430 Yeshwanth C Mineni: I don’t know how good this is, but this is the approach that I have in mind.
138 00:15:13.400 ⇒ 00:15:19.170 Demilade Agboola: Just final questions on that would just be along the lines of, how would you want to handle the ingestion?
139 00:15:21.390 ⇒ 00:15:27.650 Yeshwanth C Mineni: So, for the ignition part, like,
140 00:15:34.190 ⇒ 00:15:37.179 Yeshwanth C Mineni: So, it depends on,
141 00:15:48.010 ⇒ 00:15:53.070 Yeshwanth C Mineni: Just give me one moment, let me think about it. I didn’t have an approach on that.
142 00:16:04.190 ⇒ 00:16:13.090 Yeshwanth C Mineni: So, previously, my intention was something like this. I would go by that, like, that same structure, like,
143 00:16:13.600 ⇒ 00:16:15.979 Yeshwanth C Mineni: I used to have, like,
144 00:16:16.250 ⇒ 00:16:21.319 Yeshwanth C Mineni: Kafka, then Sparks Structure Streaming, and then ADF.
145 00:16:22.220 ⇒ 00:16:25.440 Yeshwanth C Mineni: So… like,
146 00:16:28.440 ⇒ 00:16:38.820 Yeshwanth C Mineni: I mean, like, with Kafka and Spark, like, I was having a severe reduction in the lag, like, to under 5 seconds.
147 00:16:39.290 ⇒ 00:16:40.330 Yeshwanth C Mineni: So…
148 00:16:40.580 ⇒ 00:16:47.289 Yeshwanth C Mineni: that’s the usual Indian part, like, it depends on the… I mean, like, can you give me, like, what…
149 00:16:47.550 ⇒ 00:16:49.909 Yeshwanth C Mineni: stack that we have, so that I can…
150 00:16:50.270 ⇒ 00:16:55.409 Demilade Agboola: It’s… we’re designing this for the client, so it’s up to us. That’s why this is a freeform question.
151 00:16:57.960 ⇒ 00:16:58.509 Demilade Agboola: So we don’t.
152 00:16:58.510 ⇒ 00:16:59.350 Yeshwanth C Mineni: Oh.
153 00:16:59.610 ⇒ 00:17:01.120 Demilade Agboola: We’re just designing for them.
154 00:17:04.290 ⇒ 00:17:05.369 Yeshwanth C Mineni: So…
155 00:17:10.569 ⇒ 00:17:14.939 Yeshwanth C Mineni: basically, I would go by the process of collecting, like.
156 00:17:15.290 ⇒ 00:17:23.190 Yeshwanth C Mineni: and, importing from various sources. For example, let’s say we have, Kafka for streaming, right? So…
157 00:17:24.490 ⇒ 00:17:26.650 Yeshwanth C Mineni: I would, go by,
158 00:17:27.579 ⇒ 00:17:32.550 Yeshwanth C Mineni: Kafka for streaming, and ADF for a batch, and hybrid workflows.
159 00:17:32.900 ⇒ 00:17:34.259 Yeshwanth C Mineni: And,
160 00:17:35.020 ⇒ 00:17:46.900 Yeshwanth C Mineni: My pipelines would have typically data such as, for example, if it has data, like, structured or semi-structured or unstructured data.
161 00:17:47.130 ⇒ 00:17:48.880 Yeshwanth C Mineni: Like, I would,
162 00:17:49.460 ⇒ 00:17:58.290 Yeshwanth C Mineni: optimize it for, low latency and also high throughput, and enabling the real-time analytics. And also.
163 00:17:58.470 ⇒ 00:18:07.570 Yeshwanth C Mineni: I would, I would, monitor the error handling as well in the key parts of my engine workflows to maintain, like, reliability.
164 00:18:13.970 ⇒ 00:18:17.240 Yeshwanth C Mineni: And, and, and I would check for,
165 00:18:17.610 ⇒ 00:18:27.390 Yeshwanth C Mineni: The data types that we have, typically, and then, and focus more on,
166 00:18:28.830 ⇒ 00:18:36.390 Yeshwanth C Mineni: Focus more on the… Particular data types, if they are P2 compliant or not.
167 00:18:37.320 ⇒ 00:18:44.660 Yeshwanth C Mineni: And then, proceed further. And, for the logs and the metrics, I would, check for, several measures.
168 00:18:44.930 ⇒ 00:18:50.970 Yeshwanth C Mineni: Such as, so, while I was working on this.
169 00:18:52.270 ⇒ 00:19:03.479 Yeshwanth C Mineni: At Upstart, like, I had several issues, like, where we have a particular set of data that was missing. But, we have, we had, dashboards.
170 00:19:03.680 ⇒ 00:19:06.530 Yeshwanth C Mineni: that have several KPIs to measure this.
171 00:19:07.190 ⇒ 00:19:15.460 Yeshwanth C Mineni: So, I had to, drill down to those measures to handle those errors, typically, so…
172 00:19:16.060 ⇒ 00:19:18.019 Yeshwanth C Mineni: This is the reason I would,
173 00:19:18.300 ⇒ 00:19:24.899 Yeshwanth C Mineni: Have severe validation checks to ensure that The data types are met.
174 00:19:25.050 ⇒ 00:19:27.000 Yeshwanth C Mineni: With whatever the requirements are.
175 00:19:27.480 ⇒ 00:19:30.120 Yeshwanth C Mineni: And, yeah.
176 00:19:31.900 ⇒ 00:19:32.580 Demilade Agboola: Okay.
177 00:19:33.090 ⇒ 00:19:38.659 Demilade Agboola: Alright, let’s, let’s, let’s build on what we’ve done so far. So, we have our system built out.
178 00:19:39.220 ⇒ 00:19:43.640 Demilade Agboola: Over time, we’ve discovered that, hey, we have a large volume.
179 00:19:44.580 ⇒ 00:19:46.950 Demilade Agboola: of this. Have you ever used dbt before?
180 00:19:48.700 ⇒ 00:19:51.179 Yeshwanth C Mineni: I did in my previous projects.
181 00:19:51.180 ⇒ 00:20:00.460 Demilade Agboola: Okay, that’s fine. If you haven’t used dbt, I was going to say we could think of it as a stop procedure, but if you have used dbt, we can also just go to dbt and use that as an example.
182 00:20:00.890 ⇒ 00:20:05.669 Demilade Agboola: So we have… a dbt model that…
183 00:20:05.780 ⇒ 00:20:09.959 Demilade Agboola: now it’s taking a very long time to run. They have 400 million on rows.
184 00:20:10.230 ⇒ 00:20:15.419 Demilade Agboola: So this model has gone from running in, like, 20 minutes to running in 2 hours.
185 00:20:16.770 ⇒ 00:20:27.889 Demilade Agboola: how would you go about getting our time back down? What are the steps you’ll take? What are the things you will look out for? And consider this the worst possible scenario, so…
186 00:20:28.530 ⇒ 00:20:33.309 Demilade Agboola: there were no good techniques applied, what are the things you’ll be looking out for to try to…
187 00:20:33.630 ⇒ 00:20:36.680 Demilade Agboola: You know, bring down the time it takes for the model to run.
188 00:20:42.550 ⇒ 00:20:44.200 Yeshwanth C Mineni: So,
189 00:20:47.930 ⇒ 00:20:50.730 Yeshwanth C Mineni: I would, I would, handle the…
190 00:20:55.910 ⇒ 00:21:03.280 Yeshwanth C Mineni: Usually, like, I would go for the models that we have.
191 00:21:03.800 ⇒ 00:21:09.819 Yeshwanth C Mineni: Because, usually the latency issues often arise from the inefficient SQL models.
192 00:21:10.370 ⇒ 00:21:19.199 Yeshwanth C Mineni: When we have large data volumes or resource constraints on the data warehouse, this is what, usually occurs in dbt.
193 00:21:19.400 ⇒ 00:21:27.179 Yeshwanth C Mineni: to troubleshoot, I would, optimize the SQL queries by minimizing, like, the joins or the sub-queries that we have.
194 00:21:27.550 ⇒ 00:21:37.779 Yeshwanth C Mineni: or, use incremental models in dbt to process only, change data instead of, the full refreshes.
195 00:21:37.970 ⇒ 00:21:43.799 Yeshwanth C Mineni: Or, I would leverage the indexes and also clustering in the data warehouse.
196 00:21:44.200 ⇒ 00:21:46.480 Yeshwanth C Mineni: To speed up the query execution.
197 00:21:46.820 ⇒ 00:21:55.500 Yeshwanth C Mineni: And also, I would, monitor and profile, dbt runs to identify, like, the slow models that we have, or the steps.
198 00:21:55.930 ⇒ 00:21:59.899 Yeshwanth C Mineni: And I would, consider partitioning the large tables.
199 00:22:00.090 ⇒ 00:22:08.900 Yeshwanth C Mineni: Using, like, or tables, or snapshots, where… like, wherever it’s appropriate.
200 00:22:09.450 ⇒ 00:22:15.459 Yeshwanth C Mineni: And, and also the compute is the main, key part where I would,
201 00:22:15.900 ⇒ 00:22:22.569 Yeshwanth C Mineni: Scale it, adequately during the runs, based on the data size or data volume that we have.
202 00:22:24.810 ⇒ 00:22:25.440 Demilade Agboola: Okay.
203 00:22:25.870 ⇒ 00:22:28.519 Demilade Agboola: Alright, that’s… that’s… that’s fair, that’s fine.
204 00:22:29.270 ⇒ 00:22:33.939 Demilade Agboola: Just a follow-up to that now.
205 00:22:34.210 ⇒ 00:22:36.510 Demilade Agboola: And this… So…
206 00:22:38.800 ⇒ 00:22:46.410 Demilade Agboola: if we’re… because you know what Brainforge does? We’re a consulting company. If a client comes in and says, hey, we need the dashboard.
207 00:22:46.810 ⇒ 00:22:50.219 Demilade Agboola: But doesn’t define the metrics. How do you go from…
208 00:22:52.080 ⇒ 00:22:58.540 Demilade Agboola: Just that request, to having very clear instructions that you can then execute on to work on what you need to work on.
209 00:23:00.710 ⇒ 00:23:04.699 Yeshwanth C Mineni: Sorry, I… I missed your… What? Can you repeat?
210 00:23:05.310 ⇒ 00:23:11.510 Demilade Agboola: I said, if a client comes in and says they have a request for a dashboard, But…
211 00:23:11.910 ⇒ 00:23:15.549 Demilade Agboola: they don’t… they’re not giving you client instructions. How do you go from…
212 00:23:16.090 ⇒ 00:23:25.569 Demilade Agboola: That big request, that unclear request, to having very clear instructions that you can take and work on what you need to work on on the back end.
213 00:23:27.670 ⇒ 00:23:33.730 Demilade Agboola: what’s your process like to go from unclear, I need a dashboard, to very clear instructions?
214 00:23:35.850 ⇒ 00:23:36.630 Yeshwanth C Mineni: Oh…
215 00:23:41.710 ⇒ 00:23:50.870 Yeshwanth C Mineni: like, usually, like, while building, I will go for the type of the tool, so…
216 00:23:51.340 ⇒ 00:23:54.739 Yeshwanth C Mineni: For example, I’ve used Tableau and Power BI as well.
217 00:23:55.200 ⇒ 00:24:01.979 Yeshwanth C Mineni: And, to ensure that the performance of these dashboards are, efficient, like.
218 00:24:02.390 ⇒ 00:24:06.649 Yeshwanth C Mineni: I would go by, designing the data models initially.
219 00:24:06.950 ⇒ 00:24:10.480 Yeshwanth C Mineni: And, use the aggregated data sets.
220 00:24:10.930 ⇒ 00:24:15.430 Yeshwanth C Mineni: And also, just to reduce the load times for them.
221 00:24:15.790 ⇒ 00:24:17.040 Yeshwanth C Mineni: And also…
222 00:24:17.210 ⇒ 00:24:18.290 Demilade Agboola: Sorry, sorry to interrupt.
223 00:24:18.620 ⇒ 00:24:22.000 Demilade Agboola: What do you, what do you think… what’s… what… what do you understand by the question?
224 00:24:24.520 ⇒ 00:24:25.170 Yeshwanth C Mineni: You’d like.
225 00:24:26.750 ⇒ 00:24:29.359 Yeshwanth C Mineni: I’m so sorry, actually, I didn’t get it, cause…
226 00:24:29.550 ⇒ 00:24:34.749 Yeshwanth C Mineni: Your voice was breaking in between. Can you repeat it again? Yeah, I’m so sorry about it.
227 00:24:35.490 ⇒ 00:24:39.379 Demilade Agboola: So, if a client comes and asks you for a dashboard.
228 00:24:40.700 ⇒ 00:24:47.320 Demilade Agboola: Okay. But they do not give clear instructions on what they need in the dashboard, or what is in the dashboard.
229 00:24:47.810 ⇒ 00:24:48.130 Yeshwanth C Mineni: Okay.
230 00:24:48.130 ⇒ 00:24:51.090 Demilade Agboola: You take that question, that request.
231 00:24:51.760 ⇒ 00:24:55.579 Demilade Agboola: And find a way to make it clear to you on what you need to do.
232 00:24:56.010 ⇒ 00:24:57.240 Demilade Agboola: For the dashboard.
233 00:24:59.810 ⇒ 00:25:07.950 Yeshwanth C Mineni: So… I would first engage with the client to clarify their goals and key metrics.
234 00:25:08.480 ⇒ 00:25:16.899 Yeshwanth C Mineni: But, I would, I would ask, like, targeted questions, like the data sources or frequency of updates.
235 00:25:17.270 ⇒ 00:25:19.369 Yeshwanth C Mineni: And the users for the dashboard.
236 00:25:20.890 ⇒ 00:25:30.180 Yeshwanth C Mineni: But initially, my… my way of handling it is, like, to create a prototype or, like, MVP dashboard quickly to gather, like.
237 00:25:30.650 ⇒ 00:25:32.270 Yeshwanth C Mineni: Client feedback early.
238 00:25:32.730 ⇒ 00:25:37.020 Yeshwanth C Mineni: And, iterated based on their… input, like.
239 00:25:37.330 ⇒ 00:25:40.509 Yeshwanth C Mineni: Just to make sure that the client’s happy about it.
240 00:25:42.930 ⇒ 00:25:50.480 Yeshwanth C Mineni: And, then I would document, like, the assumptions or auditions, just to be transparent with the client.
241 00:25:51.700 ⇒ 00:25:52.370 Demilade Agboola: Okay.
242 00:25:52.960 ⇒ 00:25:55.880 Demilade Agboola: Alright, so this will be my final question.
243 00:25:56.130 ⇒ 00:25:58.630 Demilade Agboola: Yeah. If a client wants
244 00:25:58.840 ⇒ 00:26:02.370 Demilade Agboola: Fast delivery, so they need something by tomorrow, right?
245 00:26:02.540 ⇒ 00:26:04.999 Demilade Agboola: But, you know, whatever way you would do.
246 00:26:05.300 ⇒ 00:26:10.249 Demilade Agboola: Or you go about to get that done by tomorrow, is not a scalable way.
247 00:26:10.770 ⇒ 00:26:15.860 Demilade Agboola: So potentially, you know, you can have errors, potentially.
248 00:26:16.030 ⇒ 00:26:23.309 Demilade Agboola: It’s just… it’s just there’s a lot of patches in it, the code isn’t the best, whatever, like, scalable means to you.
249 00:26:24.290 ⇒ 00:26:26.420 Demilade Agboola: How would you go about balancing?
250 00:26:26.720 ⇒ 00:26:27.920 Demilade Agboola: Delivery.
251 00:26:28.270 ⇒ 00:26:30.639 Demilade Agboola: As well as scalability.
252 00:26:31.000 ⇒ 00:26:32.520 Demilade Agboola: on technical excellence.
253 00:26:33.780 ⇒ 00:26:36.720 Yeshwanth C Mineni: Coming to that part,
254 00:26:41.340 ⇒ 00:26:46.370 Yeshwanth C Mineni: Usually, like, I would,
255 00:26:47.100 ⇒ 00:26:52.979 Yeshwanth C Mineni: Prioritize that particular client immediately, because, they have an impact.
256 00:26:53.120 ⇒ 00:26:55.380 Yeshwanth C Mineni: The delivery asphalt.
257 00:26:57.120 ⇒ 00:27:02.889 Yeshwanth C Mineni: like, to deliver a functional patch, or, like, MVP, just to meet the urgent deadline.
258 00:27:03.430 ⇒ 00:27:15.670 Yeshwanth C Mineni: But we don’t have a clear or concise, concise amount of time, so… I would, I would,
259 00:27:16.460 ⇒ 00:27:24.329 Yeshwanth C Mineni: I would communicate that to… The client, like, what kind of technical debt or limitations that are involved?
260 00:27:24.590 ⇒ 00:27:31.590 Yeshwanth C Mineni: And also, I would plan up an iteration to refactor and optimize the code just to scale and maintain.
261 00:27:32.730 ⇒ 00:27:36.769 Yeshwanth C Mineni: Just to meet the urgent necessity of the client.
262 00:27:37.290 ⇒ 00:27:42.369 Yeshwanth C Mineni: Or I would, I would, leverage automation for this.
263 00:27:43.260 ⇒ 00:27:49.360 Yeshwanth C Mineni: Immediately, just to make, things faster, more, and… Make them smaller.
264 00:27:49.700 ⇒ 00:27:54.000 Yeshwanth C Mineni: And also, I would, rely more on documentation as well.
265 00:27:54.240 ⇒ 00:27:59.709 Yeshwanth C Mineni: Of the temporary fixes that we do, or, that we can address them later.
266 00:28:01.440 ⇒ 00:28:07.800 Yeshwanth C Mineni: I’m in, I would follow this pattern, typically, because just to,
267 00:28:08.490 ⇒ 00:28:16.420 Yeshwanth C Mineni: Ensure that the client’s expectations are met, as well as, The robustness of our…
268 00:28:16.920 ⇒ 00:28:20.060 Yeshwanth C Mineni: Technical building is also good on our end.
269 00:28:21.880 ⇒ 00:28:23.500 Demilade Agboola: Okay, alright, sounds good.
270 00:28:24.040 ⇒ 00:28:29.469 Demilade Agboola: So that’s my final question. I don’t know if you have any questions for me.
271 00:28:33.070 ⇒ 00:28:40.390 Yeshwanth C Mineni: Actually, I do not have any questions, but I just want to know, like, what would be the next steps, if I don’t have any idea.
272 00:28:41.480 ⇒ 00:28:49.030 Demilade Agboola: Okay, so next steps, if you move forward, will be, take-home assignment.
273 00:28:49.550 ⇒ 00:28:54.270 Demilade Agboola: And usually, that stops real within, like, 5-6 hours.
274 00:28:54.700 ⇒ 00:29:02.840 Demilade Agboola: And then… Once that’s done, we’ll have a presentation with myself, I wish I knew Tom.
275 00:29:03.220 ⇒ 00:29:04.919 Demilade Agboola: 40, 45 minutes.
276 00:29:05.110 ⇒ 00:29:12.499 Demilade Agboola: And the idea of that is you just show us what you’ve done, what you used to decide what you wanted to go about.
277 00:29:13.230 ⇒ 00:29:19.720 Demilade Agboola: And how you went about different things, and we can just ask questions and get your idea on how you handle these things.
278 00:29:20.390 ⇒ 00:29:23.520 Demilade Agboola: And that’ll just be the final stage, basically, so…
279 00:29:24.650 ⇒ 00:29:31.069 Demilade Agboola: Yeah, that’s just the process. So this final, take home, final stage, interview.
280 00:29:31.180 ⇒ 00:29:33.559 Demilade Agboola: Presentation and interview. That’s it.
281 00:29:34.860 ⇒ 00:29:35.620 Yeshwanth C Mineni: That’s good.
282 00:29:36.130 ⇒ 00:29:37.900 Yeshwanth C Mineni: Thank you so much for your time.
283 00:29:38.270 ⇒ 00:29:42.360 Demilade Agboola: Alright, it’s no problem. And again, apologies again once again for yesterday.
284 00:29:42.790 ⇒ 00:29:45.400 Yeshwanth C Mineni: No, no, no, I’m so sorry, like,
285 00:29:45.720 ⇒ 00:29:51.590 Yeshwanth C Mineni: I did check my calendar, but then again, it was… stating that.
286 00:29:51.860 ⇒ 00:29:55.700 Yeshwanth C Mineni: the meeting is still active, so I joined and I was waiting, but, like.
287 00:29:55.820 ⇒ 00:30:00.870 Yeshwanth C Mineni: There was no update, so that’s the reason I was waiting in the call, and I did, let
288 00:30:01.240 ⇒ 00:30:02.799 Yeshwanth C Mineni: Okay, I know about this.
289 00:30:03.430 ⇒ 00:30:09.090 Yeshwanth C Mineni: So, just to get an update, yeah. Don’t apologize on that, I understand. There might be a lot of…
290 00:30:09.210 ⇒ 00:30:16.620 Yeshwanth C Mineni: lot of fixes or a lot of issues in the team that you need to handle and address immediately, right? I get that part. I’ve been there.
291 00:30:17.450 ⇒ 00:30:19.429 Demilade Agboola: Okay, alright, sounds good.
292 00:30:19.910 ⇒ 00:30:26.579 Demilade Agboola: Alright, thank you very much. I will reach out to Kayla and let her know, feedback on this call.
293 00:30:27.340 ⇒ 00:30:29.579 Yeshwanth C Mineni: Sure. Thank you, sir. Have a nice day.
294 00:30:29.890 ⇒ 00:30:30.470 Demilade Agboola: Bye.