Meeting Title: Brainforge Interview w- Awaish Date: 2026-04-10 Meeting participants: Awaish Kumar, Al
WEBVTT
1 00:00:32.020 ⇒ 00:00:32.930 Al: Hey!
2 00:00:33.790 ⇒ 00:00:34.450 Awaish Kumar: No.
3 00:00:36.390 ⇒ 00:00:37.370 Awaish Kumar: How you doing?
4 00:00:37.730 ⇒ 00:00:39.089 Al: Doing fine.
5 00:00:40.580 ⇒ 00:00:41.040 Awaish Kumar: Sorry?
6 00:00:41.040 ⇒ 00:00:43.139 Al: Doing fine, and yourself?
7 00:00:44.060 ⇒ 00:00:45.540 Awaish Kumar: I’m good, I’m good.
8 00:00:46.110 ⇒ 00:00:47.320 Al: That’s good to hear.
9 00:00:48.450 ⇒ 00:00:49.450 Awaish Kumar: Cool.
10 00:00:50.420 ⇒ 00:00:57.889 Awaish Kumar: Yeah, so in this introduction, we are just going to briefly talk about your experiences and what you’ve been working so far.
11 00:00:58.280 ⇒ 00:01:01.950 Awaish Kumar: And, answer any questions if you have regarding Brain Forge.
12 00:01:01.950 ⇒ 00:01:04.200 Al: Absolutely. Okay.
13 00:01:04.209 ⇒ 00:01:06.939 Awaish Kumar: So, we can start with your introduction.
14 00:01:06.939 ⇒ 00:01:12.969 Al: Did you… did you… I’m sorry to cut you off. Did you watch my boom video?
15 00:01:13.990 ⇒ 00:01:17.600 Awaish Kumar: I mean, video, is… Like…
16 00:01:17.600 ⇒ 00:01:20.789 Al: I think this is the first step. It was a Loom video.
17 00:01:21.240 ⇒ 00:01:26.139 Awaish Kumar: I have a summary of the Zoom video. Basically, we have people in our.
18 00:01:27.150 ⇒ 00:01:27.939 Al: Okay, I got it.
19 00:01:27.940 ⇒ 00:01:35.909 Awaish Kumar: I see your video, and they have a summary for me, ready to… basically Talk about it, and then…
20 00:01:37.010 ⇒ 00:01:41.840 Al: Okay, I’m just gonna repeat whatever I said there.
21 00:01:41.860 ⇒ 00:01:45.709 Awaish Kumar: Yeah, you don’t have to. I have a summary, you can introduce yourself.
22 00:01:46.570 ⇒ 00:01:54.790 Al: So, my name is Al, I’m in the data engineer field for, like, 7 years.
23 00:01:54.970 ⇒ 00:01:58.690 Al: Currently I’m working at New York Life, and
24 00:01:59.060 ⇒ 00:02:04.660 Al: You know, with a team called Bali. Bali mean fast, and I believe it’s…
25 00:02:05.050 ⇒ 00:02:09.739 Al: It’s my lesion, so, Bali mean fast, fast.
26 00:02:10.030 ⇒ 00:02:19.679 Al: So over there, me and my team, we build end-to-end data pipelines that focus on insurance policies, administration, and operational data.
27 00:02:19.760 ⇒ 00:02:36.260 Al: And, we’re working with different tools, and, but, of course, the data comes from different sources, system, and we, of course, are designing instigations, pipelines into, AWS S3 bucket, basically.
28 00:02:36.290 ⇒ 00:02:45.890 Al: We’re using batch and streaming. I use, we use basically different languages, Scala, Python, but I’m focusing on Python.
29 00:02:46.010 ⇒ 00:02:54.070 Al: And, we, of course, owned by Spark, and, we’re using Datapricks, and the EMR.
30 00:02:54.230 ⇒ 00:03:10.530 Al: To clean duplications and, applying the business, logic on, on the, across the policies and, the operational data sets. Okay. I, I, I can go, more, to give you more idea.
31 00:03:10.530 ⇒ 00:03:18.250 Awaish Kumar: I was hoping to more hear about yourself, your background, where you come from, how you entered into this field, not just tools and technologies that you’re using.
32 00:03:18.250 ⇒ 00:03:22.390 Al: Okay, so you don’t want technical, you wanna… Just talk.
33 00:03:24.590 ⇒ 00:03:25.580 Awaish Kumar: It’s a journey, right?
34 00:03:25.580 ⇒ 00:03:32.750 Al: Yeah, so… I was on the business side, earlier.
35 00:03:33.100 ⇒ 00:03:38.709 Al: I came to New York, about, like, 15 years ago,
36 00:03:39.290 ⇒ 00:03:44.289 Al: It was, an interesting, adventure for me.
37 00:03:44.780 ⇒ 00:03:52.370 Al: So, my whole family lives here from the 60s. I decided to come here.
38 00:03:53.390 ⇒ 00:03:59.350 Al: that I decided to come here, enjoy the family, and enjoy the social life with them.
39 00:03:59.350 ⇒ 00:04:03.160 Awaish Kumar: I’ll do… came into this field, you were.
40 00:04:03.160 ⇒ 00:04:13.640 Al: But yes, yes, I start… I came here, when I came here, I started my MBA with, with my… one of my friends, in LIU, Brooklyn.
41 00:04:14.030 ⇒ 00:04:25.679 Al: And, from there, we… I… it was my first time seeing the, you know, this field, the IT field in general. I started, like, I explored the code.
42 00:04:26.020 ⇒ 00:04:28.979 Al: I learned, the first thing I learned is Scala.
43 00:04:29.550 ⇒ 00:04:36.820 Al: And they still didn’t know nothing about the data engineer or the data developer, position is nothing.
44 00:04:38.340 ⇒ 00:04:47.610 Al: So, after I’m done, I, I start with Santander Bank. My, my, one of my, family members got me the…
45 00:04:47.880 ⇒ 00:04:55.170 Al: this position. I got lucky enough to get it. I learned a lot, based on…
46 00:04:55.300 ⇒ 00:04:58.999 Al: Data engineer, big data developing,
47 00:04:59.880 ⇒ 00:05:06.989 Al: And from there, I started my career, learning different tools, different languages, different cultures.
48 00:05:07.840 ⇒ 00:05:08.540 Awaish Kumar: Okay.
49 00:05:09.090 ⇒ 00:05:10.300 Al: Yeah.
50 00:05:11.300 ⇒ 00:05:15.169 Awaish Kumar: Okay, so… You mentioned that you have been working with
51 00:05:16.220 ⇒ 00:05:21.180 Awaish Kumar: In your current job, you work with Database, and you also work with the Spark.
52 00:05:21.370 ⇒ 00:05:28.140 Awaish Kumar: And then you work… and you do, right, SPAR jobs on EMR, right?
53 00:05:28.390 ⇒ 00:05:30.109 Al: Yeah, yeah, correct.
54 00:05:30.660 ⇒ 00:05:35.090 Awaish Kumar: So, like, what are… why… Quiet.
55 00:05:35.090 ⇒ 00:05:37.780 Al: It’s not only EMR, as an example.
56 00:05:38.280 ⇒ 00:05:45.379 Al: We have, so, we’re using different types of sources, right? So, in Nepali.
57 00:05:46.000 ⇒ 00:05:51.689 Al: We have incidents that’s, like, as an example, from ServiceNow and Jira and,
58 00:05:52.330 ⇒ 00:05:57.100 Al: So, we get updated daily from it, and
59 00:05:57.670 ⇒ 00:06:05.670 Al: how can I put it? So, we start doing it, believe it or not, we start taking the data from the PG admin, the Bostcursus SQL,
60 00:06:07.030 ⇒ 00:06:10.500 Al: I’m adding the data to the Databricks.
61 00:06:10.810 ⇒ 00:06:14.029 Al: We have, a category of volume over there.
62 00:06:14.470 ⇒ 00:06:17.540 Al: Daily. We’ve taken it, that’s how we started.
63 00:06:17.650 ⇒ 00:06:23.369 Al: We… we put the data in the bronze layer, like raw data, nothing.
64 00:06:23.740 ⇒ 00:06:25.280 Al: from Jira and ServiceNow.
65 00:06:25.430 ⇒ 00:06:31.939 Al: And, we was, you know, optimizing the data and, you know, cleaning the data.
66 00:06:32.190 ⇒ 00:06:38.240 Al: And, you know, we make sure there’s no duplicates, the policy number, the incident numbers are all good.
67 00:06:38.440 ⇒ 00:06:45.780 Al: After that, we start using EMR to clean duplications, applying business rules.
68 00:06:45.780 ⇒ 00:06:46.719 Awaish Kumar: Yeah, I’m good.
69 00:06:46.950 ⇒ 00:06:49.979 Awaish Kumar: Just… I have just one question. Yeah.
70 00:06:50.620 ⇒ 00:06:51.560 Al: Yeah, tell me.
71 00:06:51.560 ⇒ 00:06:57.270 Awaish Kumar: Question is, once you’re ingesting data from any of these resources to DataBricks.
72 00:06:57.740 ⇒ 00:06:58.090 Al: Okay.
73 00:06:58.090 ⇒ 00:07:03.939 Awaish Kumar: You can run your… by Spark in the Databricks as well. Correct.
74 00:07:04.060 ⇒ 00:07:07.509 Awaish Kumar: Right? Like, why you chose to use EMR then?
75 00:07:09.440 ⇒ 00:07:18.820 Al: So basically, AMR is from AWS, right? So we have to use AMR basically on, like.
76 00:07:19.070 ⇒ 00:07:22.450 Al: Databricks is our primary platform, I’m gonna put it this way.
77 00:07:22.820 ⇒ 00:07:33.710 Al: So, if we need… if we need complex transformation and, and Delta Lake, operations, and if we have, like, that’s… I’m talking about Delta Bricks, by the way.
78 00:07:33.990 ⇒ 00:07:40.490 Al: Okay, and we have, like, heavy batch jobs, especially, like, larger-scale ones.
79 00:07:40.870 ⇒ 00:07:59.859 Al: We have to use EMR, right? I don’t know, like, we’re not gonna use serviceless, clusters. We have to use, like, we have to play with the cluster configurations. That’s especially, like, if, like, the data is huge, so we have to, to play with the configurations.
80 00:07:59.860 ⇒ 00:08:02.310 Awaish Kumar: try to spark jobs in EMR, right?
81 00:08:02.870 ⇒ 00:08:03.520 Al: Yeah.
82 00:08:04.480 ⇒ 00:08:09.420 Awaish Kumar: That’s exactly what I’m asking. You can write Spark jobs in Databricks as well?
83 00:08:10.130 ⇒ 00:08:19.899 Al: Yeah, so from my understanding, the question, how we connect the AMR to, like, the cluster itself from, in Databricks, right?
84 00:08:20.230 ⇒ 00:08:21.889 Awaish Kumar: No, that’s not a question, huh?
85 00:08:21.890 ⇒ 00:08:22.440 Al: Yeah.
86 00:08:23.710 ⇒ 00:08:24.200 Al: Nope.
87 00:08:24.200 ⇒ 00:08:27.190 Awaish Kumar: Databricks provides… Databricks is a platform.
88 00:08:27.350 ⇒ 00:08:28.080 Al: Correct.
89 00:08:28.080 ⇒ 00:08:36.009 Awaish Kumar: where… which you can run multiple… you can do multiple things, not just one, right? It can be used as a storage, then you can also run… use its own.
90 00:08:36.010 ⇒ 00:08:38.800 Al: Building dashboards, yes, I understand, yeah.
91 00:08:39.070 ⇒ 00:08:43.629 Awaish Kumar: You can even write, write, by Spark, India.
92 00:08:44.070 ⇒ 00:08:47.679 Al: Correct. Alright, and SQL, and Scala, yes, you’re right.
93 00:08:47.940 ⇒ 00:08:48.530 Awaish Kumar: Yeah.
94 00:08:49.280 ⇒ 00:08:52.070 Awaish Kumar: Okay. So, was there a need to use EMR?
95 00:08:53.470 ⇒ 00:09:11.809 Al: It’s… it’s… first of all, it’s a budget for the client, right? EMR can close it… we can close it down. Databricks itself, it’s expensive and costs more, like, if we can… especially if we have a huge data to process, right? So we choose EMR, like, for the workload, because of the cost. And Databricks, you know.
96 00:09:12.200 ⇒ 00:09:21.199 Al: We’re paying for the DPUs, on the top, on the EC2, basically, like, so we’re controlling the coach itself.
97 00:09:21.330 ⇒ 00:09:24.989 Al: And it’s, of course, the speed itself, like,
98 00:09:25.340 ⇒ 00:09:30.390 Al: So with AMR, you’re paying just for the EC2 incident directly.
99 00:09:30.810 ⇒ 00:09:34.449 Al: So, no DBU markup, no extra cost.
100 00:09:34.990 ⇒ 00:09:42.099 Al: So if we’re gonna do, like, if we’re going to do the heavy batches jobs, then, that’s run once or twice a day.
101 00:09:42.200 ⇒ 00:09:47.780 Al: like our large policy in Justing runs, it’s… it’s gonna be… it’s gonna be expensive, no?
102 00:09:48.200 ⇒ 00:09:57.400 Al: So, if we spin… we spin up the EMR cluster itself, playing with the configurations itself, and run the job, the AND, then after that, terminate it.
103 00:09:57.600 ⇒ 00:10:03.840 Al: There’s no idle cost. Basically, there is… that’s it, the cluster is dead. We kill the cluster, no cost.
104 00:10:04.010 ⇒ 00:10:06.290 Al: Correct me if I’m wrong.
105 00:10:06.290 ⇒ 00:10:10.999 Awaish Kumar: I’m just asking, because EMR, like, EMR is a…
106 00:10:11.620 ⇒ 00:10:20.620 Awaish Kumar: provides you a way to run your Spark jobs, and it’s a kind of a service provided by AWS. Similar service is provided by
107 00:10:21.280 ⇒ 00:10:25.920 Awaish Kumar: Databricks as well, where you can run Spark jobs.
108 00:10:26.250 ⇒ 00:10:29.350 Awaish Kumar: High volume on the high volume of data.
109 00:10:29.840 ⇒ 00:10:30.490 Awaish Kumar: Without…
110 00:10:30.490 ⇒ 00:10:31.089 Al: when… when…
111 00:10:31.090 ⇒ 00:10:32.230 Awaish Kumar: Come on, isn’t.
112 00:10:32.230 ⇒ 00:10:38.899 Al: Yeah, so when… as an example, when I started doing… when we started adjusting the data manually in… in,
113 00:10:39.230 ⇒ 00:10:40.599 Al: In Databricks.
114 00:10:40.970 ⇒ 00:10:49.319 Al: when the… we wasn’t less than 500 rows. I didn’t add any clusters to use, so it’s serverless, literally.
115 00:10:50.030 ⇒ 00:10:57.510 Al: And we was getting the data in, you know, I cleaned it sometimes in, in, what’s it called?
116 00:10:57.780 ⇒ 00:11:03.169 Al: In ServiceNow. And, now it’s different because we have a pipeline.
117 00:11:03.360 ⇒ 00:11:11.960 Al: And the data is way bigger, way, like, it’s huge. So that’s the main reason we’re using the EMR. It’s for the cost itself.
118 00:11:12.070 ⇒ 00:11:17.480 Al: I know it’s an AMR, and I know AMR very well, and
119 00:11:17.820 ⇒ 00:11:21.479 Al: And the main reason, like I mentioned, we… we cut the cost.
120 00:11:21.670 ⇒ 00:11:23.869 Al: You’re done with your job, you kill the cluster.
121 00:11:24.150 ⇒ 00:11:31.720 Al: And even, we’re using EMR for, if you called it, if you heard about it, it’s about Zeppelin. So, it’s all about the cost and the time.
122 00:11:32.060 ⇒ 00:11:35.800 Al: You don’t want the other developers, you know…
123 00:11:36.200 ⇒ 00:11:38.420 Al: I’m gonna make it brief, it’s…
124 00:11:39.040 ⇒ 00:11:41.159 Al: we’re saving a lot of money using EMR.
125 00:11:41.570 ⇒ 00:11:42.740 Al: That’s for sure.
126 00:11:44.590 ⇒ 00:11:49.720 Al: I understand your point, I understand your point very well.
127 00:11:50.040 ⇒ 00:11:52.669 Al: But, that’s my answer.
128 00:11:52.950 ⇒ 00:12:07.719 Awaish Kumar: Yeah, I understand what you are saying, you might have experience with AWS EMR, but there is a similar feature in Databricks where you can tune your clusters for Apache Spark jobs and do all of the things that you can do in AWS.
129 00:12:07.720 ⇒ 00:12:23.679 Al: It’s, it’s a pure, pure cost, savings, and, like, like, straightforward, like, it’s, it’s for jobs, MR, it’s cheaper, because we, we, we’re avoiding, basically, the data breach, DBU fees itself, like, that’s, that’s a huge…
130 00:12:24.050 ⇒ 00:12:26.020 Al: That’s from what I understand.
131 00:12:26.400 ⇒ 00:12:29.660 Awaish Kumar: Okay, so… Okay.
132 00:12:30.290 ⇒ 00:12:34.500 Awaish Kumar: My point is, like, sometimes, for the clients,
133 00:12:35.120 ⇒ 00:12:36.920 Awaish Kumar: the people I have worked with, like.
134 00:12:36.940 ⇒ 00:12:49.119 Awaish Kumar: they want to streamline their tech stack, right? If a feature is available in the tech stack we are already using, then going and using a third-party service.
135 00:12:49.120 ⇒ 00:12:58.349 Awaish Kumar: from another tech cloud provider, which is also expensive, like, AWS is not that cheap, right? If you say, okay, I’ve been using Apache Spark.
136 00:12:58.430 ⇒ 00:13:09.720 Awaish Kumar: self-hosted one version, or we had our in-house clusters for Apache Spark that help us reduce the cost.
137 00:13:09.980 ⇒ 00:13:15.400 Awaish Kumar: Or there was some, maybe, cheaper cloud providers. Like, I have used, like…
138 00:13:15.500 ⇒ 00:13:19.689 Awaish Kumar: Previously, in one of my jobs, I’ve been using Managed Postgres.
139 00:13:20.080 ⇒ 00:13:21.270 Awaish Kumar: from Hesnar.
140 00:13:21.710 ⇒ 00:13:24.889 Awaish Kumar: So… and although we were using GCP,
141 00:13:25.020 ⇒ 00:13:38.859 Awaish Kumar: And I’ve… I’ve been asked, like, the question, like, you… why you are use… going out of GCP when you can use, actually, Postgres… you can use CloudSQL when you have managed Postgres inside of GCP as well.
142 00:13:39.490 ⇒ 00:13:40.429 Awaish Kumar: So I guess…
143 00:13:40.430 ⇒ 00:13:41.730 Al: Manage what?
144 00:13:41.980 ⇒ 00:13:43.629 Awaish Kumar: Manage Postgres database.
145 00:13:43.630 ⇒ 00:13:45.829 Al: Oh, Bostcrest data, yeah, we have it too, yeah.
146 00:13:45.830 ⇒ 00:13:51.099 Awaish Kumar: Yeah, so… I made a choice to move… using… start using Hezner for that.
147 00:13:51.250 ⇒ 00:13:53.040 Awaish Kumar: I have a question in front of me.
148 00:13:53.520 ⇒ 00:14:07.410 Awaish Kumar: So if someone comes to me and says, while we have Cloud SQL service in GCP, that gives you the same features, why you are not using that instead of using HESDR, I can say there’s a significant difference in the cost.
149 00:14:07.720 ⇒ 00:14:19.310 Awaish Kumar: But when it comes to comparing AWS and the Databricks, the AWS is… I don’t think it’s that much cheaper compared to Databricks, right?
150 00:14:20.890 ⇒ 00:14:22.410 Al: I… okay.
151 00:14:23.190 ⇒ 00:14:29.390 Al: I got you, I got your point. We’re using BJAdmin, ProgressSQL,
152 00:14:29.850 ⇒ 00:14:39.229 Al: And keeping the tech stack simple, and consistent is better for maintenance, and, especially, onboarding, like, importing, and,
153 00:14:39.900 ⇒ 00:14:47.810 Al: I got you, but it’s the client, say, in the tech, right? Like, what they want, but when we have…
154 00:14:48.150 ⇒ 00:14:52.979 Al: Databricks. Like I mentioned before, we had PGAdmin.
155 00:14:53.080 ⇒ 00:15:01.369 Al: I was going to the snowmere data itself. I was checking the incidents, and it’s… it’s a disaster over there.
156 00:15:01.540 ⇒ 00:15:11.059 Al: And that’s why we decided, okay, you know what? We’re gonna choose Databricks for all the sources that we have, like Jira, Incidents, from the ServiceNow.
157 00:15:11.270 ⇒ 00:15:12.260 Al: So my question…
158 00:15:13.020 ⇒ 00:15:18.620 Awaish Kumar: Yeah. My question is, if you thought AWS EMR is a better choice for you.
159 00:15:18.760 ⇒ 00:15:23.379 Awaish Kumar: Then, why not started… why not we used, Redshift?
160 00:15:23.510 ⇒ 00:15:25.649 Awaish Kumar: Instead of, Databricks.
161 00:15:26.950 ⇒ 00:15:30.530 Al: Alright, so I’m gonna, first of all, start…
162 00:15:31.030 ⇒ 00:15:36.189 Al: It’s their budget, not mine. They choose what the work… what’s the best work for them.
163 00:15:36.530 ⇒ 00:15:46.390 Al: My job is to deliver within wherever stack they decided on, right? If WS is better for you, then why not using Redshift?
164 00:15:47.150 ⇒ 00:15:49.790 Al: I mean, yes, exactly, but…
165 00:15:50.350 ⇒ 00:15:58.879 Al: I didn’t design the pipeline from scratch. That’s… that’s what you have to understand. Like, it’s not… it wasn’t… I’m not the CEO who said, like, okay, we have to do this.
166 00:15:58.880 ⇒ 00:16:01.570 Awaish Kumar: I understand you properly.
167 00:16:03.000 ⇒ 00:16:09.720 Awaish Kumar: the… there are all the… some constraints from the client, from the stakeholder, I understand that.
168 00:16:11.030 ⇒ 00:16:30.449 Al: I came in, and it was already in the place. DataBricks with EMR were already the choosing stack when I joined. And the main reason… because you know how is it when, as an example, if you open a cluster and you kept it for the weekend? That’s… that’s… that’s…
169 00:16:30.720 ⇒ 00:16:31.630 Al: Nope.
170 00:16:32.130 ⇒ 00:16:46.430 Al: So when I joined Valley Program, my job was, you know, to work within that architecture, optimize it, deliver the results. If I were building it from the scratch.
171 00:16:47.090 ⇒ 00:16:49.909 Al: Most likely I’m… I’m gonna choose Redshift.
172 00:16:50.440 ⇒ 00:16:56.739 Al: Because we’re using the AMR, but you work with what the client and the team already have in place.
173 00:16:57.250 ⇒ 00:17:01.229 Al: This is the situation, this is what you got, this is what you’re gonna work on.
174 00:17:01.440 ⇒ 00:17:02.490 Al: I mean, I’m gonna put it this way.
175 00:17:02.490 ⇒ 00:17:10.100 Awaish Kumar: And we can talk about, maybe, do you have a… can you give me an example of a project where you actually worked end-to-end?
176 00:17:12.800 ⇒ 00:17:25.040 Al: Sure, so… it must… well, New York Life, I consider it end-to-end, but still, like I mentioned.
177 00:17:25.430 ⇒ 00:17:31.929 Al: But we can’t speak about Nielsen, like… Nielsen Media.
178 00:17:32.240 ⇒ 00:17:32.860 Awaish Kumar: Okay.
179 00:17:33.590 ⇒ 00:17:38.439 Al: So I worked in Nielsen Media. If you see my resume, I work with TCS.
180 00:17:38.440 ⇒ 00:17:41.940 Awaish Kumar: Yeah, I saw TCS, but…
181 00:17:42.150 ⇒ 00:17:45.489 Al: Yeah, yeah. Well, TCS, as you know, is an implementation company.
182 00:17:45.710 ⇒ 00:17:52.439 Al: If you want, I can… I lent most of my things from San Andre Bank as well.
183 00:17:52.440 ⇒ 00:17:56.589 Awaish Kumar: provided services as part of TCS to Nelson Media, right?
184 00:17:57.010 ⇒ 00:18:07.160 Al: Yes, correct. I worked with Nielsen Media for, for years. It’s, it’s, it’s, TSES, I don’t know about, I’m sure you… it’s one of the big companies.
185 00:18:07.320 ⇒ 00:18:19.699 Al: That’s, you know, it’s an implementation company, I don’t know. So we was, you know, over there, we was, like, processing terabytes of media, data.
186 00:18:20.020 ⇒ 00:18:28.779 Al: like, with different sources, different platforms. We… we was using, like, like, streaming, cable, broadcast.
187 00:18:29.420 ⇒ 00:18:38.260 Al: what else? Syndications. I start with, ingesting layer, raw data, like, from…
188 00:18:38.780 ⇒ 00:18:49.720 Al: the sources to the S3 bucket, the S3, in AWS from multiple sources systems, like, using both batch files, streaming through Kinesis, and,
189 00:18:49.720 ⇒ 00:18:51.870 Awaish Kumar: So, what tool is used for ingestion?
190 00:18:53.290 ⇒ 00:19:04.100 Al: Like I said, we’re using, stream code, Kinesis, and, it’s for the real time, and it’s, like, you know, in AWS.
191 00:19:04.630 ⇒ 00:19:07.030 Al: What else?
192 00:19:08.290 ⇒ 00:19:15.190 Al: we used, multiple… we was writing our code on, IntelliJ, we used GitLab.
193 00:19:15.540 ⇒ 00:19:30.880 Al: I used AWS Glue, for cataloging the raw data, and the EMR to run the Spark, jobs, like cleaning, duplicating, applying the business rules across the viewership.
194 00:19:31.160 ⇒ 00:19:37.309 Al: What else? We loaded the processing data into the Redshift, and
195 00:19:38.560 ⇒ 00:19:46.570 Al: I spent much of everything. Of course, we kept, we kept, enhancing the project by, by, by time.
196 00:19:46.830 ⇒ 00:19:55.560 Al: every, every, changes we was pushing it to the, the right branch. We used a schedule, like, airflow. We had…
197 00:19:56.630 ⇒ 00:19:59.470 Al: So many jobs, like, around the 24 hours.
198 00:19:59.900 ⇒ 00:20:10.599 Al: On Airflow, the scheduler. Redshift was the downstream for us, and made, to make it, like, to the tables for the downstream analytics team.
199 00:20:11.410 ⇒ 00:20:13.770 Al: That’s pretty much.
200 00:20:13.770 ⇒ 00:20:16.120 Awaish Kumar: So, what are you using inflow as in,
201 00:20:16.260 ⇒ 00:20:21.350 Awaish Kumar: Managed Airflow or self-hosted version, like the open source version of Airflow?
202 00:20:21.350 ⇒ 00:20:35.960 Al: We, we, we managed, we managed, like, the, the airflow, especially that’s, if, even if we have any failures, any, any problems, we was checking out the logs there.
203 00:20:36.080 ⇒ 00:20:37.190 Al: Sometimes.
204 00:20:37.190 ⇒ 00:20:42.979 Awaish Kumar: Did open… like, you were… Hosting it yourself, or you were using a managed service?
205 00:20:43.410 ⇒ 00:20:45.669 Al: We use managed, service,
206 00:20:46.280 ⇒ 00:20:51.570 Al: the managed service in AWS that MWWA, I believe, or…
207 00:20:51.810 ⇒ 00:21:01.660 Al: Yeah. So, it’s, it’s… listen, at the end of the day, it’s… it was, it was a tough project, especially that, that’s the main reason I left DCS, honestly.
208 00:21:02.140 ⇒ 00:21:07.900 Al: It’s, like I mentioned before, the MWA, NWS airflow.
209 00:21:08.380 ⇒ 00:21:10.619 Al: But over there, we had…
210 00:21:11.160 ⇒ 00:21:17.029 Al: around 100… I’m not sure how many, but it’s, like, 100 DACs, that running…
211 00:21:17.320 ⇒ 00:21:21.440 Al: Took us, like, for the broadcast, cable, syndication,
212 00:21:21.590 ⇒ 00:21:33.520 Al: minute by minute, quarter. So it was, it was, you know, we had, like, you know, you know, bigger traffic, like, for the client.
213 00:21:34.150 ⇒ 00:21:34.850 Awaish Kumar: Okay.
214 00:21:36.090 ⇒ 00:21:36.940 Al: But yeah.
215 00:21:37.550 ⇒ 00:21:41.900 Awaish Kumar: So, I’ll… How familiar are you with Airflow?
216 00:21:43.060 ⇒ 00:21:47.950 Al: I’m pretty familiar with it. I did support for some time in Nielsen.
217 00:21:49.360 ⇒ 00:21:50.610 Al: Especially, right?
218 00:21:52.300 ⇒ 00:21:57.469 Awaish Kumar: What do you think the… what are the different components of airflow architecture?
219 00:21:59.260 ⇒ 00:22:04.800 Al: So, therefore, like, the… if you’re asking about the production support, like, the CLA jobs, or…
220 00:22:05.160 ⇒ 00:22:07.400 Al: What the different, like, from the…
221 00:22:08.090 ⇒ 00:22:12.529 Awaish Kumar: I mean, there are components that make it, like, as a tool, like, there is a.
222 00:22:12.530 ⇒ 00:22:17.300 Al: Oh, you’re talking about the scheduling, the executor, and the workers, that’s what you’re asking for?
223 00:22:18.320 ⇒ 00:22:27.899 Al: Like, if, like, the scheduler, how we, how we do the schedule, and, you know, and how we continuously, like, monitoring the DAGs itself?
224 00:22:28.430 ⇒ 00:22:38.589 Al: Check the schedules, and of course, we have the executor, the execution deck, for each job. That’s how the task basically runs.
225 00:22:38.890 ⇒ 00:22:47.840 Al: We have, something called, workers, like, when… to execute the task itself.
226 00:22:48.250 ⇒ 00:22:55.740 Al: Of course, each DAC, it’s a… basically, it’s a Python file.
227 00:22:56.060 ⇒ 00:23:05.730 Al: That’s, you know, like, defining the task and their dependencies itself. We do some changes over there with…
228 00:23:05.840 ⇒ 00:23:19.150 Al: the cluster and everything, and, I, I said, sometimes… not… like, we was a big team, but I have an experience saying, the SLA, this, callback on critical DAX, so…
229 00:23:19.150 ⇒ 00:23:22.410 Awaish Kumar: the airflow, if I want to do…
230 00:23:23.430 ⇒ 00:23:29.980 Awaish Kumar: for a task, if I want to set a SLA that, if it doesn’t finish, You know…
231 00:23:30.330 ⇒ 00:23:33.000 Awaish Kumar: In 60 seconds, I wanted to fail.
232 00:23:33.420 ⇒ 00:23:36.610 Awaish Kumar: How would you… How would you do that in Airflow?
233 00:23:38.610 ⇒ 00:23:45.810 Al: So you’re talking about if it didn’t work for 60 minutes, 60 seconds, I’m sorry, 60 seconds?
234 00:23:46.190 ⇒ 00:23:49.780 Al: It killed the cluster, or it showed the failure, or how was it?
235 00:23:49.810 ⇒ 00:23:52.650 Awaish Kumar: Just fail the task, not kill the cluster.
236 00:23:52.950 ⇒ 00:23:54.960 Awaish Kumar: Just like that, a task.
237 00:23:55.190 ⇒ 00:23:55.860 Awaish Kumar: Okay.
238 00:23:55.860 ⇒ 00:23:56.500 Al: Nope.
239 00:23:56.500 ⇒ 00:23:58.510 Awaish Kumar: Part of a gas in airflow.
240 00:23:58.510 ⇒ 00:24:01.710 Al: Are we using… you’re talking about execution timer?
241 00:24:02.290 ⇒ 00:24:03.989 Al: That’s, as an example.
242 00:24:03.990 ⇒ 00:24:08.729 Awaish Kumar: Talking about the task itself, the execution time. Obviously, when it started.
243 00:24:09.200 ⇒ 00:24:13.830 Awaish Kumar: Executing, and if it does not finish in a… One minute.
244 00:24:14.110 ⇒ 00:24:16.190 Awaish Kumar: Then mark it as a fail.
245 00:24:16.800 ⇒ 00:24:18.790 Awaish Kumar: And send me the information in Slack.
246 00:24:18.790 ⇒ 00:24:26.359 Al: Well, SLA makes you say, like, on SLA, on the task itself. Like, if we… if we, like…
247 00:24:26.880 ⇒ 00:24:32.780 Al: If they only trigger the notification, it doesn’t actually kill the task, it just alerts you that, okay.
248 00:24:32.930 ⇒ 00:24:35.809 Al: It took longer than 60 seconds, and I…
249 00:24:36.230 ⇒ 00:24:40.119 Al: And, this is how we execute the timeout itself.
250 00:24:40.430 ⇒ 00:24:42.820 Al: So, as an example, if you want
251 00:24:42.980 ⇒ 00:24:49.890 Al: We want to actually fail, you have to change, like, you know, you have to do the execution timeout.
252 00:24:50.110 ⇒ 00:24:53.170 Al: Which is kill the job itself, and if there’s a cluster, they kill it.
253 00:24:53.170 ⇒ 00:24:55.100 Awaish Kumar: How do you set SLA, basically?
254 00:24:55.990 ⇒ 00:25:03.999 Al: The SLA… the SLA, in Python file itself, right? Like, if we… we…
255 00:25:04.120 ⇒ 00:25:15.530 Al: the Python operation, sorry, like, SLA equal time, time deltas, parquet second equal 60 seconds, 30 seconds, whatever you want.
256 00:25:15.790 ⇒ 00:25:18.979 Al: Okay. I say what time, you want, basically.
257 00:25:19.490 ⇒ 00:25:20.530 Al: Okay.
258 00:25:20.740 ⇒ 00:25:23.590 Al: But we don’t put it at 60 seconds, but yeah.
259 00:25:25.040 ⇒ 00:25:26.250 Awaish Kumar: Yeah, okay.
260 00:25:26.370 ⇒ 00:25:31.989 Awaish Kumar: And, how do you monitor data pipelines,
261 00:25:33.800 ⇒ 00:25:38.680 Awaish Kumar: I mean, in terms of data, Freshness and data quality.
262 00:25:39.050 ⇒ 00:25:41.450 Awaish Kumar: Not as a pipeline, right?
263 00:25:42.360 ⇒ 00:25:44.439 Awaish Kumar: But as a data quality.
264 00:25:44.440 ⇒ 00:25:50.250 Al: Are you talking about… are you talking about… we’re still about… talking about, Airflow itself, or, like, when you…
265 00:25:50.250 ⇒ 00:26:00.860 Awaish Kumar: And in general, I’m just talking about data pipelines, how do you ensure that data is fresh, and there are no data issues.
266 00:26:01.070 ⇒ 00:26:08.229 Al: We have multiple ways, but we was using, Snowflakes and dbt to check the, you know, the freshness of the data itself. DPT, like.
267 00:26:08.610 ⇒ 00:26:12.840 Al: I have to build the freshness objects, and
268 00:26:13.340 ⇒ 00:26:19.909 Al: I think that’s where we defined how old the source and the table can be. And the data itself…
269 00:26:20.060 ⇒ 00:26:28.629 Al: landed with that window, so dbt throw the warning error, or whatever. So, basically, the way you like it.
270 00:26:29.790 ⇒ 00:26:39.989 Awaish Kumar: That is not in DVD, I understand, but in your current project, how you are handling it, when you are using Python, PySpark, Databricks, EMR, in this
271 00:26:40.200 ⇒ 00:26:42.170 Awaish Kumar: Texting, how you are handling that?
272 00:26:43.520 ⇒ 00:26:53.400 Al: Okay, so, basically, when we have a failure in the pipeline itself, like, events.
273 00:26:53.520 ⇒ 00:26:59.250 Al: So, in New York Life, like, when we’re talking about Bali, right,
274 00:26:59.610 ⇒ 00:27:07.130 Al: we, we, we, in the Databricks itself, with the BiSpark, we added freshness, like, check directly.
275 00:27:07.410 ⇒ 00:27:10.899 Al: is different than Kinesis from the one that I worked with before.
276 00:27:11.250 ⇒ 00:27:23.810 Al: So we have it directly in the Python code, and before processing, I will check the max, what’s it called, the timestamp or max load, data of the incoming data. Like, it’s, it’s…
277 00:27:24.020 ⇒ 00:27:32.850 Al: that’s how we do it. If the data is so old, the job will fail, and, you know, Automatically.
278 00:27:33.010 ⇒ 00:27:38.290 Al: And, you know, trigger, like, an alert to let us know.
279 00:27:39.410 ⇒ 00:27:40.090 Awaish Kumar: Okay.
280 00:27:40.630 ⇒ 00:27:44.370 Al: That’s… that’s what we’re doing, in… in New York, life.
281 00:27:45.850 ⇒ 00:27:47.799 Awaish Kumar: Okay, yeah, thank you.
282 00:27:48.190 ⇒ 00:27:53.890 Awaish Kumar: I think we are just… 3 minutes left for this session, so…
283 00:27:53.890 ⇒ 00:27:57.430 Al: You asked me many questions, I have to ask a couple of questions, if you don’t mind.
284 00:27:57.430 ⇒ 00:27:59.750 Awaish Kumar: Yeah, I will leave this time for you to ask anyways.
285 00:28:00.540 ⇒ 00:28:07.069 Al: How is, like, the… so what’s the responsibility daily, day by day? Day by day.
286 00:28:07.070 ⇒ 00:28:10.560 Awaish Kumar: It’s like, we have, for each person.
287 00:28:10.840 ⇒ 00:28:17.290 Awaish Kumar: in… in Brainport, in Brain Forge. Every… every… Engineer or developer is…
288 00:28:17.430 ⇒ 00:28:20.540 Awaish Kumar: At least assigned to 2-3 clients.
289 00:28:21.600 ⇒ 00:28:25.240 Awaish Kumar: So your time is divided between 2 to 3 clients?
290 00:28:26.860 ⇒ 00:28:31.010 Awaish Kumar: Where the data engineering services are required.
291 00:28:32.570 ⇒ 00:28:34.000 Awaish Kumar: Based on that.
292 00:28:34.380 ⇒ 00:28:50.710 Awaish Kumar: We use Linear as our project management tool, where every… where you have all the tickets, right? So, we have a structure where, for every client, we have customer success owners that are responsible for managing the engagement with the client.
293 00:28:50.830 ⇒ 00:28:54.610 Awaish Kumar: And they bring the… The requirements for us.
294 00:28:54.900 ⇒ 00:28:59.709 Awaish Kumar: Right? If there is a… Ask from client to bring in
295 00:28:59.840 ⇒ 00:29:03.570 Awaish Kumar: Marketing data, and then model it, and create a dashboard for it.
296 00:29:03.680 ⇒ 00:29:06.380 Awaish Kumar: So this… this will come from CSO.
297 00:29:06.490 ⇒ 00:29:08.439 Awaish Kumar: Who is the engagement?
298 00:29:08.750 ⇒ 00:29:13.020 Awaish Kumar: who runs the engagement with the client? Then there is a team of engineers.
299 00:29:13.340 ⇒ 00:29:24.339 Awaish Kumar: And when there is… there is a work required for a data engineer, then maybe one of the data engineers will be assigned to that project, and… and based on that.
300 00:29:25.300 ⇒ 00:29:36.539 Awaish Kumar: project and the task, we are going to decide on client A how many hours of data engineering work is required, and based on that, you will be assigned to that client.
301 00:29:36.890 ⇒ 00:29:40.989 Awaish Kumar: And once you’re assigned, you have tickets in linear, you are going to…
302 00:29:41.190 ⇒ 00:29:51.760 Awaish Kumar: Yeah, basically manage authors and linear, give updates, work on that. Then we have Cursor that we use for our development. So we are really AI-heavy.
303 00:29:54.830 ⇒ 00:30:03.839 Awaish Kumar: So everybody here in Brainforge uses AI for our development. Even, non-technical stakeholders, like…
304 00:30:04.380 ⇒ 00:30:12.980 Awaish Kumar: sales team, marketing team, everybody has… is using AI to improve their workflows. And same goes for us. So we use AI,
305 00:30:13.230 ⇒ 00:30:17.419 Awaish Kumar: To speed up our delivery process.
306 00:30:17.420 ⇒ 00:30:20.870 Al: Sweet, we… Yeah,
307 00:30:20.870 ⇒ 00:30:21.960 Awaish Kumar: Then it comes…
308 00:30:22.790 ⇒ 00:30:29.959 Al: I just want to add something that, in New York Life, we’re using AI, but we have our own AI, it’s called MapXL.
309 00:30:30.880 ⇒ 00:30:37.179 Al: So, we have an AI team that built it. We use a different type of AIs, like Copilot.
310 00:30:38.300 ⇒ 00:30:39.699 Al: Of course, it’s, it’s busy.
311 00:30:39.910 ⇒ 00:30:42.300 Al: It’s only for, New York Life.
312 00:30:42.610 ⇒ 00:30:47.360 Al: So… Of course, Genie AI on Databricks.
313 00:30:47.480 ⇒ 00:30:56.769 Al: So, how… how big is the team? Because every engineer have 3 clients, you mentioned.
314 00:30:58.140 ⇒ 00:31:06.239 Awaish Kumar: Yeah, team is really, like, data engineers, we are… 3, 4, like…
315 00:31:07.980 ⇒ 00:31:12.620 Awaish Kumar: Five, six people in data team, and then we are kind of…
316 00:31:13.310 ⇒ 00:31:16.379 Awaish Kumar: 3 to 4 people in our AI team.
317 00:31:16.680 ⇒ 00:31:19.620 Awaish Kumar: Then we have some 2-3 people in our…
318 00:31:19.950 ⇒ 00:31:22.730 Awaish Kumar: As our… in our data analysts.
319 00:31:22.880 ⇒ 00:31:25.789 Awaish Kumar: And then we have a sales team, marketing team.
320 00:31:27.480 ⇒ 00:31:28.200 Al: Yeah.
321 00:31:28.200 ⇒ 00:31:35.779 Awaish Kumar: Yeah, then we have leadership team, yeah, leaders that basically… and yeah, customer success owners that run the engagement with the client.
322 00:31:36.830 ⇒ 00:31:38.500 Al: I got it. And,
323 00:31:39.630 ⇒ 00:31:47.479 Al: The tools, you guys are using, is it… depends on the client, is it… depends on the company itself?
324 00:31:47.700 ⇒ 00:31:52.090 Awaish Kumar: The tools we use are… depends…
325 00:31:52.440 ⇒ 00:31:54.179 Awaish Kumar: Obviously, it depends on the client.
326 00:31:54.320 ⇒ 00:31:55.290 Al: There’s the most part.
327 00:31:55.290 ⇒ 00:31:55.950 Awaish Kumar: Bing.
328 00:31:56.110 ⇒ 00:32:04.210 Awaish Kumar: But there’s a lot of… a lot of save from us. So, a client… when a client comes in, you do the discovery of…
329 00:32:04.330 ⇒ 00:32:06.699 Awaish Kumar: What the client’s pain points are.
330 00:32:07.090 ⇒ 00:32:11.389 Awaish Kumar: And based on that, We come up with our architecture.
331 00:32:11.540 ⇒ 00:32:18.979 Awaish Kumar: And then, if client already has some hard choices, like, okay, we… our organization is using
332 00:32:20.250 ⇒ 00:32:27.899 Awaish Kumar: AWS, and we are not able to use any other cloud providers, then we are forced to come… to suggest
333 00:32:28.100 ⇒ 00:32:47.650 Awaish Kumar: our, like, the services from AWS, but if that is not the case, then we come up, based on the use case, we come up with the tools that are best suited for that use case. It’s not also based on our preferences. It’s not like, okay, I’m good in Airflow, I’m going to suggest that. It’s not like that. It’s more like.
334 00:32:47.800 ⇒ 00:32:52.369 Awaish Kumar: Here’s the use case. What are the tools which are best suited for this use case?
335 00:32:52.530 ⇒ 00:33:03.480 Awaish Kumar: Right? That is our approach. But when the client comes and says, okay, these might be the good recommendations, but we have a hard requirement from our
336 00:33:05.050 ⇒ 00:33:13.160 Awaish Kumar: tech team that you can’t use GCP, or you can’t use any service from Azure, because we are on AWS.
337 00:33:13.280 ⇒ 00:33:16.379 Awaish Kumar: Then there’s a different, like, choices we have to make.
338 00:33:17.020 ⇒ 00:33:24.359 Al: So basically choosing the tool based on the use case itself to help the client, like IU, and
339 00:33:26.160 ⇒ 00:33:30.470 Al: So, I wish, did I pronounce your name right?
340 00:33:30.710 ⇒ 00:33:31.489 Awaish Kumar: Yes, yes.
341 00:33:31.490 ⇒ 00:33:32.840 Al: Okay, so…
342 00:33:33.380 ⇒ 00:33:41.440 Al: I never worked with multiple clients. I worked with different teams. I’m helping sometimes the business side, especially that building.
343 00:33:41.650 ⇒ 00:33:48.170 Al: the conference, dashboards and databases, and, in New York, New York Live.
344 00:33:51.110 ⇒ 00:33:55.029 Al: But, like, one client with many accounts.
345 00:33:55.400 ⇒ 00:33:59.299 Al: and different teams in Nielsen, it was… it was tough enough.
346 00:33:59.740 ⇒ 00:34:10.860 Al: But how is it in it? Like, let’s be honest, maybe I’m gonna get the job, maybe not, but I just wanna, you know, for future references.
347 00:34:10.860 ⇒ 00:34:12.470 Awaish Kumar: It’s a startup company.
348 00:34:12.670 ⇒ 00:34:26.359 Awaish Kumar: it’s a fast-paced company, and I also don’t want to, say things that are not real, right? I want to be real, so it is a success for both of us. Even if we hire you, we want you to success in that.
349 00:34:26.980 ⇒ 00:34:28.600 Awaish Kumar: role, and also…
350 00:34:29.219 ⇒ 00:34:38.450 Awaish Kumar: as a… you will be working in my team, so I want you to be successful, so that team will be successful. Otherwise… otherwise, we are not going to…
351 00:34:38.870 ⇒ 00:34:42.180 Awaish Kumar: Like, deliver what… what we’re supposed to deliver, right?
352 00:34:42.380 ⇒ 00:34:43.300 Al: Fair.
353 00:34:43.409 ⇒ 00:34:47.570 Awaish Kumar: So, the thing is, Obviously, it’s a fast-paced environment.
354 00:34:47.570 ⇒ 00:34:48.489 Al: Yes.
355 00:34:48.489 ⇒ 00:34:51.750 Awaish Kumar: Sometimes some weeks can be relaxed, but sometimes it can be…
356 00:34:51.900 ⇒ 00:35:03.189 Awaish Kumar: more work, and we have… we have, as I mentioned, two to three clients, so it doesn’t mean you’re… you are going to go very well beyond your 40 hours of work commitment.
357 00:35:03.760 ⇒ 00:35:04.320 Al: No.
358 00:35:04.320 ⇒ 00:35:05.760 Awaish Kumar: It’s not… not happening, right?
359 00:35:05.760 ⇒ 00:35:06.240 Al: that.
360 00:35:06.240 ⇒ 00:35:11.000 Awaish Kumar: You will work for 40 hours, but your 40 hours will be divided into multiple clients.
361 00:35:11.460 ⇒ 00:35:22.189 Al: Okay, I mean, even if sometimes, you know, let’s be realistic. Sometimes we have to stay more, sometimes we just finish early, depends on the task, depends on the story itself.
362 00:35:22.190 ⇒ 00:35:28.800 Awaish Kumar: But, like, what I’m trying to say is not a consistency, like, okay, consistently I say, okay, you have to work for 60 hours.
363 00:35:29.410 ⇒ 00:35:47.799 Awaish Kumar: That’s not the case. We are… like, everybody works here for 40 hours, but obviously, the case is, when you are… you have a relationship with client, and there is a buyer on Friday, you don’t want to leave them, like, as it is, right? Obviously, you want to work in two more hours to fix their issues.
364 00:35:47.940 ⇒ 00:35:53.430 Awaish Kumar: And, so that’s the… that happens everywhere, in every organization. Yes.
365 00:35:53.710 ⇒ 00:36:03.020 Al: It does. With Nielsen, we worked with different client, teams, sources, and sometimes we were staying late, especially if we have a failure on the…
366 00:36:03.220 ⇒ 00:36:11.109 Al: like, when I work with… for the Nielsen, like, media, they’ll work with, many account, teams, and,
367 00:36:11.330 ⇒ 00:36:16.299 Al: like, character and Comcast, and, well, I have to…
368 00:36:16.470 ⇒ 00:36:33.200 Al: you know, that’s experience itself, I have it, but I think it’s dissimilar with what you’re saying, and I have spoken to many clients, like, indie clients before, in Nielsen Media, because I don’t know if you’re familiar with Nielsen Media. Nielsen Media is a survey company that’s
369 00:36:33.430 ⇒ 00:36:35.720 Al: Dealing with many, many clients.
370 00:36:35.880 ⇒ 00:36:39.039 Al: But, yeah, so…
371 00:36:39.320 ⇒ 00:36:55.149 Al: from my side, I have this experience, I know the responsibilities, I know, like, the expectation, I know sometimes you have to put, like, an extra effort that’s, you know, for… like you mentioned, it’s a startup company, so it’s gonna…
372 00:36:55.310 ⇒ 00:37:06.580 Al: need to grow, and need to sometimes, you know, give a little bit more work, and I’m sure, like, you in this field enough to know how is it. And, for me, I…
373 00:37:06.830 ⇒ 00:37:09.320 Al: I have knowledge, but I’m still learning.
374 00:37:10.250 ⇒ 00:37:26.520 Al: We using AI, we looking up for the best solution sometimes to help the team, to help ourselves, and especially with AI now, saving a lot of time, saving us some challenges that we was dealing with before.
375 00:37:27.030 ⇒ 00:37:33.280 Al: I don’t… I… we passed the time. I’m really sorry about that. So…
376 00:37:33.280 ⇒ 00:37:35.160 Awaish Kumar: No worries, no worries, I am.
377 00:37:35.340 ⇒ 00:37:38.030 Awaish Kumar: I want you to be at this meeting really,
378 00:37:38.500 ⇒ 00:37:40.989 Awaish Kumar: We need… we understand each other, so that…
379 00:37:41.190 ⇒ 00:37:45.899 Al: Yeah, I don’t know if you have a hard stop or something, but.
380 00:37:45.900 ⇒ 00:37:51.580 Awaish Kumar: No, I want to make sure, because, like, this person is going to work directly with me, and I want to make sure
381 00:37:51.740 ⇒ 00:37:54.929 Awaish Kumar: That we are on the same page, right? If we want to… Yeah.
382 00:37:55.680 ⇒ 00:38:07.609 Al: Sometimes, if you’re gonna know me, sometimes I don’t know how to express myself the right way, but sometimes they have an idea, I have an opinion, sometimes I have to say it, I will say it.
383 00:38:08.180 ⇒ 00:38:13.279 Al: Maybe you’re not gonna like it, but maybe you’re gonna like it, I don’t know.
384 00:38:13.440 ⇒ 00:38:21.000 Al: But I’m trying my best to… to always to express my, my opinion. Maybe, we… I learn, you learn.
385 00:38:21.000 ⇒ 00:38:23.389 Awaish Kumar: It’s all about, like, the work, right?
386 00:38:23.390 ⇒ 00:38:25.090 Al: Yes, this is the closure.
387 00:38:26.690 ⇒ 00:38:35.840 Awaish Kumar: We support opinionated versions that basically give suggestions, talk about different ways of doing things.
388 00:38:36.240 ⇒ 00:38:42.809 Awaish Kumar: that’s okay, we can debate, we can talk about it, we can discuss, but what I want at the end is.
389 00:38:42.940 ⇒ 00:38:44.480 Awaish Kumar: Once we decided.
390 00:38:44.590 ⇒ 00:38:50.950 Awaish Kumar: Because of, for whatever reasons, even if you don’t like it, we have to then implement it as a team.
391 00:38:51.400 ⇒ 00:39:03.950 Al: Of course, and this is the culture I’m looking for. I don’t… you know, there’s a politics, but I don’t care about it. I’m just doing my work, I’m trying to do my best, I’m always trying to add the value to the team.
392 00:39:04.160 ⇒ 00:39:07.690 Al: My last question, I promise. What’s the next step?
393 00:39:07.990 ⇒ 00:39:09.810 Al: What’s, what’s, after this?
394 00:39:09.810 ⇒ 00:39:15.140 Awaish Kumar: Yeah, next steps are, like, after I submit my feedback, Kayla from our recruiters
395 00:39:15.280 ⇒ 00:39:23.629 Awaish Kumar: team, like, she’s going to come back with the next steps, and the next step will be meeting with one of my
396 00:39:24.050 ⇒ 00:39:24.870 Awaish Kumar: colleague.
397 00:39:25.100 ⇒ 00:39:28.889 Awaish Kumar: He’s also going to… it will also be a 30-minute session.
398 00:39:29.390 ⇒ 00:39:32.549 Awaish Kumar: Similar to this, asking about different things.
399 00:39:32.720 ⇒ 00:39:35.699 Awaish Kumar: Regarding data, pipelines and things like that.
400 00:39:35.920 ⇒ 00:39:38.969 Awaish Kumar: Enough that we will have a take-home assignment.
401 00:39:39.420 ⇒ 00:39:42.660 Awaish Kumar: And after that, there will be a panel interview discuss…
402 00:39:42.850 ⇒ 00:39:45.940 Awaish Kumar: Where we will be talking more about that.
403 00:39:46.840 ⇒ 00:39:54.650 Awaish Kumar: Task, and also, like, The technical questions, like, really deep diving into technicalities of the things.
404 00:39:54.650 ⇒ 00:39:59.580 Al: Is the take-home assignment taking, like, 3, 4, 5 hours, something like that, or no?
405 00:40:00.200 ⇒ 00:40:03.380 Awaish Kumar: Ideally, you should just take 1-2 hours, not more than that.
406 00:40:04.380 ⇒ 00:40:05.080 Al: Okay.
407 00:40:05.460 ⇒ 00:40:08.630 Awaish Kumar: But, yeah, yeah, okay, I’m gonna have…
408 00:40:09.190 ⇒ 00:40:14.030 Awaish Kumar: to be honest, in the age of AI, it doesn’t… Like…
409 00:40:14.030 ⇒ 00:40:14.790 Al: I won’t…
410 00:40:14.790 ⇒ 00:40:21.450 Awaish Kumar: 5-6 hours, right? Now everybody writes code as in using AI, so… must be okay.
411 00:40:21.450 ⇒ 00:40:24.329 Al: Yeah, I do, but I don’t trust it too much sometimes, you know?
412 00:40:24.590 ⇒ 00:40:26.310 Al: You know, to make some changes.
413 00:40:26.870 ⇒ 00:40:32.119 Awaish Kumar: Yeah, like, everybody, like, we are the reviewers, like, that’s… that’s why we have to be very…
414 00:40:32.430 ⇒ 00:40:38.659 Awaish Kumar: critical and be a strong reviewers. So we wanted to try to write the code, but we wanted to write it.
415 00:40:38.830 ⇒ 00:40:41.170 Awaish Kumar: Right, right, right?
416 00:40:41.650 ⇒ 00:40:47.629 Al: I mean, I like it, but I don’t like it, you know? I mean, I don’t know what’s gonna happen for the next years with AI, but yeah.
417 00:40:47.630 ⇒ 00:40:58.450 Awaish Kumar: Yeah, we have to write it using AI, but then we have to also review it properly so that it follows our guidelines and it implements what we are… and it is working as expected.
418 00:40:58.570 ⇒ 00:41:08.500 Awaish Kumar: basically, that’s what we want to… what to do with the AI. It’s not… we are not blind to just say AI to do anything and just push in production.
419 00:41:10.800 ⇒ 00:41:11.330 Al: No, okay.
420 00:41:11.330 ⇒ 00:41:16.230 Awaish Kumar: There are layers of… That you are the one, as a developer.
421 00:41:16.890 ⇒ 00:41:21.339 Awaish Kumar: as a junior developer, like, the code as an AI assistant.
422 00:41:21.340 ⇒ 00:41:21.920 Al: Man.
423 00:41:21.920 ⇒ 00:41:28.310 Awaish Kumar: some code, but now you, as a senior developer, it’s your job to look at that and verify it.
424 00:41:28.310 ⇒ 00:41:29.430 Al: abruptly.
425 00:41:29.740 ⇒ 00:41:30.680 Al: Yeah.
426 00:41:31.360 ⇒ 00:41:31.900 Al: And…
427 00:41:31.900 ⇒ 00:41:35.959 Awaish Kumar: That’s what we are going to talk about in the, maybe, panel interview.
428 00:41:36.120 ⇒ 00:41:43.249 Awaish Kumar: We are more going to talk about the technicalities of it, design choices. Why did you do that?
429 00:41:43.500 ⇒ 00:41:52.900 Awaish Kumar: talk about scenarios. So, which I didn’t have a chance to do that in this 30-minute session. If you reach at that point, we are going to discuss those scenarios.
430 00:41:53.300 ⇒ 00:41:53.850 Al: Brooke?
431 00:41:53.880 ⇒ 00:41:54.710 Awaish Kumar: Okay.
432 00:41:54.900 ⇒ 00:42:02.570 Al: Awish, I really appreciate your time. I have more questions, but I’m gonna save it for the next session if we have it.
433 00:42:02.740 ⇒ 00:42:03.160 Awaish Kumar: Okay.
434 00:42:03.320 ⇒ 00:42:10.949 Al: Yeah, so, yeah, I’m really excited about this opportunity, especially that it’s a startup company.
435 00:42:11.570 ⇒ 00:42:17.380 Al: I believe I’m still young, so I want to grow up with something, you know?
436 00:42:17.380 ⇒ 00:42:19.200 Awaish Kumar: Yeah, no worries.
437 00:42:19.510 ⇒ 00:42:21.750 Awaish Kumar: Thank you for your time, and yeah.
438 00:42:22.210 ⇒ 00:42:23.370 Awaish Kumar: Have a nice week.
439 00:42:23.580 ⇒ 00:42:24.910 Al: Alright, enjoy the weekend.