Meeting Title: Brainforge x Anish Gupta Interview Date: 2026-02-11 Meeting participants: Awaish Kumar, Anish Gupta
WEBVTT
1 00:00:15.820 ⇒ 00:00:16.970 Anish Gupta: Hey, can you hear me?
2 00:00:18.880 ⇒ 00:00:21.590 Awaish Kumar: Yes, I can. Hi.
3 00:00:21.870 ⇒ 00:00:22.210 Anish Gupta: Hi.
4 00:00:26.460 ⇒ 00:00:27.680 Awaish Kumar: How are you doing?
5 00:00:28.040 ⇒ 00:00:38.159 Anish Gupta: Doing good, how are you? Sorry, I was just struggling with, like, for some reason, like, my Zoom account wasn’t working, so I had to quickly make a new one. Wasn’t sure what was going on, but… sorry, sorry about being a little late.
6 00:00:38.670 ⇒ 00:00:41.360 Awaish Kumar: No worries. So, where are you located?
7 00:00:41.830 ⇒ 00:00:57.380 Anish Gupta: I’m in San Francisco, actually, so I don’t think I’m actually that far from you guys. I was talking, yeah, I mean, I’m San Francisco area. Right now I’m actually at home in Santa Clara, like, with family, but usually I’m in the city.
8 00:00:58.960 ⇒ 00:01:00.340 Awaish Kumar: Okay,
9 00:01:00.520 ⇒ 00:01:08.829 Awaish Kumar: So, yeah, I will, like, start with my introduction and the brain foods introduction, and then we are going to
10 00:01:08.930 ⇒ 00:01:15.070 Awaish Kumar: Mainly talk about your… Work and your, like,
11 00:01:15.350 ⇒ 00:01:21.220 Awaish Kumar: projects you have worked on and your, like, experiences. So, yeah, starting with
12 00:01:22.040 ⇒ 00:01:24.959 Awaish Kumar: Myself, my name is Abish Kumar, and I’m…
13 00:01:25.430 ⇒ 00:01:27.770 Awaish Kumar: I’ve been working as a data engineer
14 00:01:28.170 ⇒ 00:01:38.070 Awaish Kumar: For, like, the last 9 to 10 years, I’ve been working in growth stage companies and startups and enterprises, so…
15 00:01:38.400 ⇒ 00:01:43.460 Awaish Kumar: I’m leading the data engineering service and brain food.
16 00:01:44.200 ⇒ 00:01:54.779 Awaish Kumar: And what Brainforce does is we provide, data and AI consultancy services. It’s a completely remote company. We have people from all…
17 00:01:55.050 ⇒ 00:01:56.210 Awaish Kumar: working…
18 00:01:56.580 ⇒ 00:02:16.460 Awaish Kumar: For us from across the world, and, yeah, we enforced. Basically, also in terms of our clients, mostly based on… are in Europe, sorry, mostly based on the US, but, like, we… we basically target, on medium to large scale
19 00:02:16.590 ⇒ 00:02:18.040 Awaish Kumar: enterprises.
20 00:02:19.560 ⇒ 00:02:26.459 Awaish Kumar: yeah, that’s… that’s regarding Brainforge. So, starting with your introduction.
21 00:02:27.050 ⇒ 00:02:29.680 Awaish Kumar: If you can introduce yourself a little bit.
22 00:02:30.880 ⇒ 00:02:41.160 Awaish Kumar: Regarding, like, what you have, Worked on, and… And also, like, Any of your…
23 00:02:41.410 ⇒ 00:02:43.280 Awaish Kumar: Projects which you are proud of.
24 00:02:44.140 ⇒ 00:02:53.480 Anish Gupta: Sure, yeah, yeah. So, my name is Anish Gupta. I graduated from UC Berkeley in 2023 in Data Science and Chemical Engineering.
25 00:02:53.580 ⇒ 00:02:59.650 Anish Gupta: Started off my career more of the chemical engineering interest, actually, in my freshman year.
26 00:02:59.840 ⇒ 00:03:04.640 Anish Gupta: With some work I did at Moderna, so I was working on the drug product development team at first.
27 00:03:04.680 ⇒ 00:03:15.230 Anish Gupta: Which got me a lot of experience, kind of, with the customer side of things, but there I realized that data science and data engineering was much more of my interests, because
28 00:03:15.230 ⇒ 00:03:26.310 Anish Gupta: you know, I worked with some of the data teams and the stats teams there, so I was getting a lot more experience on the data science side, so I transitioned over to starting to pursue more of a data science focus, a data science
29 00:03:26.470 ⇒ 00:03:46.009 Anish Gupta: background, so I continued that work into my first major internship at West Monroe, which is a consulting company as well for tech consulting. So got a lot of sales engineering, data engineering, and, you know, working with a lot of customers, non-technical customers usually, on their data engineering
30 00:03:46.010 ⇒ 00:04:03.929 Anish Gupta: capabilities. This was mainly focused on data storage, helping them use services like S3, or helping them set up some data pipelines, like Airflow, or stuff like that. Things that tech leads aren’t usually as used to, but
31 00:04:04.010 ⇒ 00:04:12.350 Anish Gupta: wanting to give them the experience to be able to modernize their current data flow. Some of these companies are still using Excel for their work a lot, and storing things
32 00:04:12.410 ⇒ 00:04:29.039 Anish Gupta: In SharePoint, so, like, that’s not a very… obviously not a scalable approach at all. So, we were giving them… trying to reintroduce themselves into, like, a more modern data-focused environment, and that work kind of discussion continued to my most recent internship and then full-time position on Juniper Network, so…
33 00:04:29.040 ⇒ 00:04:38.979 Anish Gupta: There, Juniper Networks is a lot more focused on networks, right? We’re… it’s a networking company first and foremost, so a lot of routing, like, hardware work with, like, routers, and…
34 00:04:39.530 ⇒ 00:05:03.750 Anish Gupta: and modems, but my work is on the AIOps teams there, so a lot more software focused, a lot more data-focused. Again, since these routers have to be streaming so many packets of data, and it’s time series data, since it’s all real-time networking traffic and network tracking, got a lot of experience on working with time series data and large packets of data all at once using services like Kafka.
35 00:05:03.950 ⇒ 00:05:14.540 Anish Gupta: was still one of our main tools that we’re using. A bunch of other microservices, like Spark, and obviously Docker containers, so a lot of work there.
36 00:05:14.710 ⇒ 00:05:27.199 Anish Gupta: My most recent work has been actually working on creating an automated traffic prediction model. So, based off of, you know, tracking network… network…
37 00:05:27.620 ⇒ 00:05:46.920 Anish Gupta: sorry, tracking network data and predicting before we know these networks are gonna, or these devices are gonna fall down, or gonna fail, and whatever, for whatever reasoning, we can predict those errors happening, we can train models that are… that are using this, that are using past network data and past error data that we know of.
38 00:05:47.020 ⇒ 00:05:55.599 Anish Gupta: and start to predict these errors before they happen, and circumvent our services as needed. So a lot of real-time traffic data, a lot of real-time
39 00:05:55.770 ⇒ 00:06:00.510 Anish Gupta: data, data engineering, and that also is a customer-focused role, sorry.
40 00:06:01.000 ⇒ 00:06:06.220 Awaish Kumar: Yeah, so… If you can tell me about, like, like…
41 00:06:06.420 ⇒ 00:06:10.430 Awaish Kumar: One of the projects where you work end-to-end.
42 00:06:10.990 ⇒ 00:06:15.510 Awaish Kumar: Yeah, if you can share what tools and technologies were used.
43 00:06:15.830 ⇒ 00:06:19.430 Awaish Kumar: How, like, the project was planned, and…
44 00:06:19.850 ⇒ 00:06:22.169 Awaish Kumar: Yeah, what about the results?
45 00:06:22.880 ⇒ 00:06:42.849 Anish Gupta: Sure, yeah, yeah. So, one project, you know, which I work end-to-end, that I’m pretty… that I’m pretty proud of was my first project of internship at Juniper Networks. So there, I started off on our AIOps team, obviously, but I was working more on internal tools and internal, documentation tracking before we… I started to work with customers.
46 00:06:42.850 ⇒ 00:06:53.309 Anish Gupta: So, one issue that we had at the company was that a lot of our documentation, especially for our AI practices or our ML practices, were, stored not
47 00:06:53.310 ⇒ 00:07:10.750 Anish Gupta: we’re not, easily accessible and easily readable for employees across the… across our entire customer base, because, again, our customers are worldwide. So, I was working on creating a chatbot, which is going to be used for both internal customers, for internal use and customer use.
48 00:07:10.750 ⇒ 00:07:17.869 Anish Gupta: to be able to read our documentation and provide actual actionable insights on how we can solve a problem. So…
49 00:07:18.100 ⇒ 00:07:35.959 Anish Gupta: the first step for that, obviously, is we need to start gathering up our data. A lot of our data, a lot of our documentation was… was parsed using, using Langchain, so I parsed a lot of… parsed a lot of text-based data there, a lot of data cleaning to actually be able to,
50 00:07:36.250 ⇒ 00:07:38.420 Anish Gupta: Read that data in an efficient manner.
51 00:07:38.480 ⇒ 00:07:55.580 Anish Gupta: So that took a lot of, you know, not just understanding on my end, because I hadn’t worked with some of those services before, but, learning a lot more from people around me, people on my team, or people on other teams that have used a lot more of these AI or these LLM-based services. And then, from there.
52 00:07:55.580 ⇒ 00:08:00.699 Anish Gupta: developing this… developing this model and training it off of our text-based data, creating a RAG-based
53 00:08:00.730 ⇒ 00:08:07.950 Anish Gupta: Implementation was the next step, and then a lot of testing and documentation as well for that.
54 00:08:08.360 ⇒ 00:08:14.439 Anish Gupta: So then we ended up with a final product, which, was pretty efficient. It was able to help employees
55 00:08:14.550 ⇒ 00:08:27.599 Anish Gupta: get data about… I mean, for customer results, I was hearing great, great feedback on that, was speeding up our overall data service, or… sorry, our, information gathering and stuff like that pretty well, so yeah.
56 00:08:27.600 ⇒ 00:08:35.179 Awaish Kumar: So how, basically, You gave access to… To the external users.
57 00:08:35.409 ⇒ 00:08:38.170 Awaish Kumar: Like, positive, like… Was it a…
58 00:08:38.429 ⇒ 00:08:44.340 Awaish Kumar: Chatbox on the website, or how the… external sticking those music again.
59 00:08:44.760 ⇒ 00:09:00.459 Anish Gupta: Oh, yeah, yeah. So, it started off, again, internally, so it was… we have our normal employee page, our employee portal, for gathering data… our documentation, which is, I believe it’s in a… it’s, like, based off of SharePoint, but it’s not in SharePoint, it’s our…
60 00:09:00.460 ⇒ 00:09:05.040 Anish Gupta: own developed website that we have for employees to be able to track
61 00:09:05.340 ⇒ 00:09:23.760 Anish Gupta: everything from, like, from names of devices, or names of past projects, to your information that you might need for taking PTO. Like, all our documentation is stored on our… this, portal, which is… which stores all of our documentation. So this chatbot is in addition onto that current platform.
62 00:09:24.110 ⇒ 00:09:28.179 Anish Gupta: So that, was able to be used by users right away.
63 00:09:29.080 ⇒ 00:09:33.300 Awaish Kumar: Okay, so you mentioned using Langchen, but what was the process before that?
64 00:09:33.430 ⇒ 00:09:42.239 Awaish Kumar: So how that document moved from… That platform to… To some database, or where…
65 00:09:42.880 ⇒ 00:09:48.250 Awaish Kumar: Like, what was that pipeline look like?
66 00:09:48.790 ⇒ 00:09:51.959 Anish Gupta: For the, for the data itself, or sorry, for the…
67 00:09:51.960 ⇒ 00:09:52.390 Awaish Kumar: Right.
68 00:09:52.410 ⇒ 00:09:56.010 Anish Gupta: service. Nira was on some platform, right? Yeah, yeah, yeah.
69 00:09:56.350 ⇒ 00:10:02.799 Awaish Kumar: You… You must have wrote something, some script, some pipeline, some tools.
70 00:10:02.800 ⇒ 00:10:03.339 Anish Gupta: Yeah, yeah.
71 00:10:03.340 ⇒ 00:10:05.620 Awaish Kumar: To read that data, process it.
72 00:10:05.760 ⇒ 00:10:08.440 Awaish Kumar: So, what was that full pipeline?
73 00:10:09.050 ⇒ 00:10:25.209 Anish Gupta: Oh, oh yeah, for sure. So, it started off with just a local script for our small base… our small batch of data, our testing data, so I created a local Python script for that first, to, you know, test our initial models, but the majority of this data was stored in,
74 00:10:25.330 ⇒ 00:10:33.210 Anish Gupta: in S3. So we had our data stored for documentation, for device data, all these things were stored in S3.
75 00:10:33.290 ⇒ 00:10:49.509 Anish Gupta: we have a different server for our network data and a different server for our documentation. So there’s a lot of, reading data from S3, transforming that data in… into, Kafka, so we can actually package this data in a useful method.
76 00:10:49.700 ⇒ 00:10:52.669 Anish Gupta: Since it’s text-based data, we also need to parse through our…
77 00:10:52.670 ⇒ 00:10:56.760 Awaish Kumar: like… So you have all the documentation data on S3,
78 00:10:56.920 ⇒ 00:11:07.330 Awaish Kumar: You have Python, local Python script, Which reads from, S3, so why, then…
79 00:11:08.060 ⇒ 00:11:10.490 Awaish Kumar: Why didn’t it… the Kafka was involved.
80 00:11:10.950 ⇒ 00:11:24.030 Anish Gupta: Well, because we want to scale it up, and we want it to be able to gather new information, because this local script was useful for simple user testing, but as we are… as we wanted to increase the documents for
81 00:11:24.030 ⇒ 00:11:33.640 Anish Gupta: hundreds of thousands of documents and… and scaling it, and allowing customers to also be able to interact with it. We wanted to create an endpoint that wasn’t just based off of a local script.
82 00:11:33.690 ⇒ 00:11:38.949 Anish Gupta: So then we started to move into having this data automatically be stored and packaged by Kafka.
83 00:11:39.260 ⇒ 00:11:40.410 Anish Gupta: from S3.
84 00:11:42.170 ⇒ 00:11:45.459 Awaish Kumar: So from S3, it moves to Kafka Topics.
85 00:11:47.490 ⇒ 00:11:52.020 Awaish Kumar: And how does that… Movement was managed.
86 00:11:53.160 ⇒ 00:11:53.730 Awaish Kumar: promised.
87 00:11:55.090 ⇒ 00:12:07.730 Anish Gupta: this… that… that part, actually, I wasn’t working on. I was working more on the endpoint, but I believe, because I joined the team pretty recently, so I believe that was done through,
88 00:12:08.900 ⇒ 00:12:18.109 Anish Gupta: I actually don’t remember what that part was used… was transferred from, but I was reading the data from these Kafka topics, which was reading from S3. I wasn’t sure about that part of the pipeline, actually.
89 00:12:18.980 ⇒ 00:12:23.439 Awaish Kumar: Okay, so you were… If you had a Python script which was reading from Kafka.
90 00:12:24.820 ⇒ 00:12:25.460 Anish Gupta: Hmm.
91 00:12:26.170 ⇒ 00:12:27.040 Awaish Kumar: topics.
92 00:12:27.690 ⇒ 00:12:28.510 Awaish Kumar: Okay.
93 00:12:29.010 ⇒ 00:12:32.069 Awaish Kumar: But you could have also looked from real read from
94 00:12:33.360 ⇒ 00:12:40.970 Awaish Kumar: even if you are reading it from Kafka, you can obviously read it from S3,
95 00:12:43.270 ⇒ 00:12:48.249 Awaish Kumar: So I… okay, so I actually didn’t understand the use case for the Kafka here.
96 00:12:48.470 ⇒ 00:12:52.819 Awaish Kumar: Until you… unless you have something to add. So, yeah.
97 00:12:53.100 ⇒ 00:12:59.810 Awaish Kumar: How… can you please… You can continue, like, how that looked like after the Kafka.
98 00:13:00.910 ⇒ 00:13:13.959 Anish Gupta: Oh, sure, yeah. So, after we loaded our data from… after I loaded the data from Kafka topics, because… since we… we need the different topics, because we have different documentation stores, like, we have…
99 00:13:14.160 ⇒ 00:13:30.850 Anish Gupta: things such as employee information, or employee… not information, but important information for employees, like PTO timings, or scheduling, things like that, like, employee-focused topics, and then we have more customer-focused topics, things such as our different device types that we have, our different
100 00:13:30.940 ⇒ 00:13:37.470 Anish Gupta: products that we’re storing. So this is all being able to be updated and… Oh, sorry.
101 00:13:37.470 ⇒ 00:13:46.640 Awaish Kumar: Yeah, what I’m saying is, okay, I understand the data came from S3, it got landed in… Kafka topics.
102 00:13:47.280 ⇒ 00:13:48.069 Awaish Kumar: That’s okay.
103 00:13:48.780 ⇒ 00:13:52.509 Awaish Kumar: So how, like, what was happening after that?
104 00:13:53.920 ⇒ 00:14:12.650 Anish Gupta: So then after that, I needed to create an actual portal that users could start querying from… for this data, because obviously users can’t just read it all manually. So then that’s when I created a more front-end-based approach, and for that, I used Node.js to create,
105 00:14:12.650 ⇒ 00:14:13.260 Awaish Kumar: Oh.
106 00:14:14.140 ⇒ 00:14:18.290 Awaish Kumar: even before that, like, the theoretiz in Kafka, you…
107 00:14:18.890 ⇒ 00:14:21.450 Awaish Kumar: that you, you mentioned you have Python script.
108 00:14:22.110 ⇒ 00:14:24.689 Awaish Kumar: So, what that Python script was doing.
109 00:14:25.120 ⇒ 00:14:35.000 Anish Gupta: Oh, oh, sorry, I thought I’d mention it earlier, but we parsed our data with… we parsed our data into JSON files, to be able to be transferred into a…
110 00:14:35.210 ⇒ 00:14:45.430 Anish Gupta: to be transferred into Langchain, right? Because Langchain needs to be able to read this documentation and create and to train this LLM, right? So we used Langchain for that.
111 00:14:45.430 ⇒ 00:14:47.590 Awaish Kumar: Your JSON files were stored.
112 00:14:48.730 ⇒ 00:14:52.400 Anish Gupta: Those JSON files were stored,
113 00:14:52.600 ⇒ 00:15:01.699 Anish Gupta: in S3 as well. So we had, like, a transferring system, but it was a different server. So those are stored on our virtual servers that we have.
114 00:15:02.080 ⇒ 00:15:05.550 Anish Gupta: for documentation, so we have these, like, VMs for…
115 00:15:05.870 ⇒ 00:15:19.349 Anish Gupta: our product, and we used these VMs, or when I was doing this testing, we had used these VMs. I think this implementation got changed because I had switched teams over, so this implementation was changed later, but these were stored on VMs at.
116 00:15:19.350 ⇒ 00:15:20.509 Awaish Kumar: Honey, I don’t listen.
117 00:15:21.420 ⇒ 00:15:28.060 Awaish Kumar: So, you were storing in S3, so, like, what was… what was the purpose… purpose of… Having VMs?
118 00:15:29.160 ⇒ 00:15:33.309 Anish Gupta: the VMs were just for development’s sake, because our actual…
119 00:15:33.480 ⇒ 00:15:48.239 Anish Gupta: Our actual, product was… our actual services were running on these virtual machines that users can access, that customers are going to be using our product on. Our, our main network trap… network tracking product.
120 00:15:48.420 ⇒ 00:15:54.180 Anish Gupta: So to be able to access documentation for a said product, the information needs to be able to be accessed
121 00:15:54.300 ⇒ 00:16:03.480 Anish Gupta: On this… on the VMs, which are gonna be… so the product is called EOP, or it’s called Epic On-Prem, and so this service.
122 00:16:03.480 ⇒ 00:16:09.450 Awaish Kumar: It’s like, let’s stay at that pipeline. So, you are reading data from
123 00:16:10.480 ⇒ 00:16:16.970 Awaish Kumar: Kafka, then you process it and convert it into some JSON files.
124 00:16:17.670 ⇒ 00:16:21.139 Awaish Kumar: those JSLs files you mentioned that were stored in S3.
125 00:16:21.510 ⇒ 00:16:22.180 Anish Gupta: Hmm.
126 00:16:24.020 ⇒ 00:16:25.679 Awaish Kumar: Yeah, not den.
127 00:16:25.800 ⇒ 00:16:30.829 Awaish Kumar: after those GSO files are missed S3, Then what happens?
128 00:16:33.090 ⇒ 00:16:48.720 Anish Gupta: Then we need to be able to take that data out, and in timeframes, because the user’s gonna be querying about specific data, specific questions, so we have it stored based off of information like.
129 00:16:48.920 ⇒ 00:17:03.230 Anish Gupta: if this document is more based on devices, so we have different folders in our S3, and then from there, our model’s gonna train, based off of the query that the user is using, it’ll access these folders in different
130 00:17:03.790 ⇒ 00:17:08.069 Anish Gupta: Different… it’ll access the folders based off of the topic, so…
131 00:17:08.079 ⇒ 00:17:08.659 Awaish Kumar: Okay.
132 00:17:08.660 ⇒ 00:17:10.589 Anish Gupta: So that… It’s gonna be like, yeah.
133 00:17:11.190 ⇒ 00:17:20.569 Awaish Kumar: Okay, so that was more like, this project more like in… kind of… were you using LLMs, or what were you using?
134 00:17:21.119 ⇒ 00:17:26.629 Anish Gupta: The LLM is gonna be used more on the… was used on the user side, so, like, when the user asked a question.
135 00:17:26.630 ⇒ 00:17:26.989 Awaish Kumar: We are.
136 00:17:27.079 ⇒ 00:17:35.029 Anish Gupta: queries our system. This LLM is used to transfer that query into accessing said data.
137 00:17:35.150 ⇒ 00:17:54.220 Anish Gupta: Right? Because we need to know what file paths to access what data we’re going to be… we’re gonna be using, sorry. And then from there, we’re gonna return this data, parse the data as needed, and then deliver an end result and an answer to our user, which is where our LM is being used there.
138 00:17:55.150 ⇒ 00:18:07.029 Awaish Kumar: Yeah, I understand that flow, like, this is… what I’m trying to say is this is more like an AI projects flow, where you normally take data, do cleanups, store it in somewhere where
139 00:18:07.270 ⇒ 00:18:11.379 Awaish Kumar: you can train your LLMs on, or you can provide as a…
140 00:18:11.490 ⇒ 00:18:19.810 Awaish Kumar: That as a context, and then you ask the question, you have the context, and based on that, you get some results.
141 00:18:20.380 ⇒ 00:18:28.689 Awaish Kumar: So… If you can give an example of a data project, basically, a data…
142 00:18:29.750 ⇒ 00:18:33.140 Awaish Kumar: Then, like, we can maybe talk more about that.
143 00:18:33.660 ⇒ 00:18:44.940 Anish Gupta: Okay, sure. So, I think the project that I was offering at first was kind of, like, my first experience in data engineering, so I was hoping that I can kind of deliver that, but…
144 00:18:45.010 ⇒ 00:18:59.300 Anish Gupta: We can focus on a project that I more recently did, so that could probably help more. It’s a lot more data engineering focused. So this project is, again, it’s at Juniper, so it’s the most recent project I was just working on.
145 00:18:59.590 ⇒ 00:19:06.170 Anish Gupta: And it involves the automatic… part of the automatic network traffic prediction that I was talking about.
146 00:19:06.440 ⇒ 00:19:10.349 Anish Gupta: As my last… my most recent work that I’ve been working on, so yeah.
147 00:19:11.720 ⇒ 00:19:18.769 Awaish Kumar: Yeah, let me just ask a question here. Like, when you were talking about predictions, were you using any machine learning?
148 00:19:19.540 ⇒ 00:19:37.929 Anish Gupta: Yes, yes, so that is still… that part with the prediction is still a work in progress, so I was gonna kind of walk through what we started doing and what the goal is for later, but the project that I worked on was a portion of that initial process. This, like, this is not a… like, this network prediction is not fully implemented.
149 00:19:37.930 ⇒ 00:19:41.070 Anish Gupta: Because it’s the main crux of our project, but…
150 00:19:41.070 ⇒ 00:19:45.860 Anish Gupta: There’s steps that we’ve been working towards that have, you know, been developed and are…
151 00:19:45.860 ⇒ 00:19:56.919 Anish Gupta: the key components that we’ll need later. So, yeah. So, because since these networks can be going down for so many reasons, we need to develop different… different use cases and different…
152 00:19:57.070 ⇒ 00:20:00.240 Anish Gupta: different ways to track said use cases. Like, we have…
153 00:20:00.760 ⇒ 00:20:07.430 Anish Gupta: implementations for reasons such as CPU failures, such as hardware failures and things like that.
154 00:20:07.430 ⇒ 00:20:11.019 Awaish Kumar: We’re gonna talk about the data engineering part of it.
155 00:20:11.540 ⇒ 00:20:13.719 Anish Gupta: Yeah, okay, sounds good. So…
156 00:20:13.840 ⇒ 00:20:28.980 Anish Gupta: We have to… obviously, because these devices have so many potential things that can go wrong, we need to simulate and test multiple services that can run use cases of what would this situation look like if these devices did go down.
157 00:20:28.980 ⇒ 00:20:40.610 Anish Gupta: So one popular example that we’ve been using recently that I’ve been working on is this thing we call black hole detection, but it’s basically when a device is noticed… is marked to be down.
158 00:20:40.670 ⇒ 00:20:57.019 Anish Gupta: the network, the traffic data is going to be streaming into this device, right, on the input, but since the device itself is down, there’s no output that’s able to be streamed out. So it’s the term that we call a black hole. So we want to be able to develop our system, our product, to be able to detect when there is a black hole.
159 00:20:57.260 ⇒ 00:21:06.669 Anish Gupta: So, that is an alert that we call, like, just a black hole that pops up, and to simulate that, we have these… we have DAGs running in Airflow.
160 00:21:06.970 ⇒ 00:21:13.470 Anish Gupta: To… that are running as a service, and this simulates what would happen if we have
161 00:21:13.690 ⇒ 00:21:16.259 Anish Gupta: Input coming into a device, which we have…
162 00:21:16.490 ⇒ 00:21:24.820 Anish Gupta: Streaming things such as… such as, network protocols, like, time series information.
163 00:21:25.030 ⇒ 00:21:37.359 Anish Gupta: like, all this different data, these packets of data that are stored in… that are transferred into as JSON files, typically, or that are transferred… sorry, that are transferred out as JSON files, so we can read it in different services.
164 00:21:37.600 ⇒ 00:21:49.890 Anish Gupta: And the input is streamed from Kafka into these services. So that, that process is simulated for each different service, for each different issue that we’re simulating.
165 00:21:50.590 ⇒ 00:21:55.320 Awaish Kumar: Okay, so, so… Did you have a device?
166 00:21:55.640 ⇒ 00:21:59.660 Awaish Kumar: Yes. You are simulating the data, right, using Dexter.
167 00:22:00.020 ⇒ 00:22:05.839 Anish Gupta: Yeah, yeah, so we’re simulating the data using, just text, or pushing with, yeah, using… using text.
168 00:22:06.000 ⇒ 00:22:08.709 Anish Gupta: That we are running,
169 00:22:08.860 ⇒ 00:22:19.749 Anish Gupta: Using text that we can push a command to our DAG, which is running in Airflow. So we can say, like, okay, start this error that we have, and then we can
170 00:22:19.840 ⇒ 00:22:31.330 Anish Gupta: Start this simulated error, and then once that simulated error is picked up by our device, then our product can say, okay, there’s an error there, and then what do we do as a result of that error being detected?
171 00:22:32.700 ⇒ 00:22:34.410 Awaish Kumar: Okay,
172 00:22:34.890 ⇒ 00:22:42.379 Awaish Kumar: Okay, and yeah, I have asked a few questions on, like, what are you looking for in your tax role?
173 00:22:44.210 ⇒ 00:22:52.270 Anish Gupta: Yeah, so the main thing that we’re looking for, apart from the error message, which is a pretty generic error message, because.
174 00:22:53.340 ⇒ 00:22:57.299 Awaish Kumar: Not about this project. My question is, what are you looking for in your next role?
175 00:22:57.300 ⇒ 00:23:10.840 Anish Gupta: Oh, oh, oh, so sorry. Oh, I didn’t hear what you… I didn’t hear… I thought you were… I thought you were saying what you were looking for in the project. I think what I’m looking for is a chance to get to work on not just different types of products or different types of services.
176 00:23:10.840 ⇒ 00:23:17.880 Anish Gupta: but just got… just to learn as much as I can, and I feel like the consulting type of positions where I can…
177 00:23:18.040 ⇒ 00:23:30.749 Anish Gupta: work on a bunch of different projects are perfect for that. You know, like, building up not just my experience, but helping out clients on developing things that they can learn about in their future, and use in their different services as well.
178 00:23:31.140 ⇒ 00:23:35.879 Awaish Kumar: Like, I’m, like, where do you see yourself in the next 5 years?
179 00:23:36.280 ⇒ 00:23:42.050 Anish Gupta: I see myself… I see, sorry. I see myself working in, I think, a field where
180 00:23:42.190 ⇒ 00:23:54.560 Anish Gupta: I’m not just, you know, typing away at a computer, developing a project, but I’m actually helping deliver a project that I worked on to a customer, and helping them
181 00:23:54.560 ⇒ 00:24:04.169 Anish Gupta: You know, like, whether it’s a client, or whether it’s a customer on, like, a freelance project, whatever it is, just helping this customer develop their business and their ideas using
182 00:24:04.300 ⇒ 00:24:11.980 Anish Gupta: Actionable data, and using data engineering to be able to drive their business ideas forward.
183 00:24:13.460 ⇒ 00:24:17.790 Awaish Kumar: Okay, so you mentioned that you have experience working in consultancy.
184 00:24:17.970 ⇒ 00:24:21.740 Anish Gupta: Can you give an example where you had to.
185 00:24:21.740 ⇒ 00:24:22.590 Awaish Kumar: Push back.
186 00:24:22.810 ⇒ 00:24:23.839 Awaish Kumar: To your client.
187 00:24:24.790 ⇒ 00:24:40.960 Anish Gupta: Yeah, yeah. So, like I said in my experience, it was typically working with clients that weren’t as technically focused. So, I think there’s always that slight question of, you know, if you’re trying to add an implementation or add a feature,
188 00:24:41.360 ⇒ 00:24:55.659 Anish Gupta: there’s always a question on the client’s end of the timeline, like, there’s always… they expect that timeline… that these services can be done in, like, a day or two, or, like, a week max. But then, as a developer, as, like, on the, consulting side.
189 00:24:55.900 ⇒ 00:25:11.989 Anish Gupta: the times you should push back is when you have to explain that these timelines don’t align with what they believe. So, one particular example was, I was working with a company that was actually an insurance company, so they’re working on a dental insurance company, so they are dealing with a lot of
190 00:25:12.160 ⇒ 00:25:16.909 Anish Gupta: individual customer data, where, you know, there’s different use… different…
191 00:25:17.350 ⇒ 00:25:27.339 Anish Gupta: Different types of customers, and we try to predict which particular purchasing plans they can use for their customers to be able to, you know, like, help their customers out the most.
192 00:25:27.440 ⇒ 00:25:37.419 Anish Gupta: And provide the best service. So, we had developed a model which was training on this past customer data to be able to predict, based off of
193 00:25:37.870 ⇒ 00:25:51.569 Anish Gupta: different metrics, like their age, their current income, their past history, what specific plans we can offer them. So on the client side, clients were not being able to provide us with enough of our training data, and not
194 00:25:51.700 ⇒ 00:26:03.520 Anish Gupta: being able to provide us with enough training data, with enough time, to be able to develop that model. So we had to push back on our… on the consulting side, on our consultants. We had to start saying, like, you know, we… we need…
195 00:26:03.520 ⇒ 00:26:12.039 Anish Gupta: more data to be able to actually help you guys out. So, we have to kind of walk through our current process and our thought process and break everything down individually.
196 00:26:12.200 ⇒ 00:26:20.220 Anish Gupta: And I think once you help explain the information to someone who isn’t as technically inclined, it really does help.
197 00:26:20.450 ⇒ 00:26:25.839 Anish Gupta: you know, like, deliver the process. Deliver the…
198 00:26:26.160 ⇒ 00:26:31.429 Anish Gupta: Get what you need at the end, like, you know, like, get your end result, get your goal that you’re looking for.
199 00:26:33.910 ⇒ 00:26:40.789 Awaish Kumar: Okay, I think that’s it from my side. If you have any questions, Yeah, you may ask.
200 00:26:42.760 ⇒ 00:26:48.399 Anish Gupta: Yeah, no, I mean, I think just, like, what is… what is your favorite part about Brainforge and, like, working at the company?
201 00:26:50.330 ⇒ 00:26:54.190 Awaish Kumar: Yeah, well, like… It’s, startup.
202 00:26:55.330 ⇒ 00:26:59.309 Awaish Kumar: And a consistency services company, so that
203 00:26:59.920 ⇒ 00:27:04.379 Awaish Kumar: I, in my whole career, I mostly worked at startups.
204 00:27:04.520 ⇒ 00:27:08.309 Awaish Kumar: and growth stage companies, so I… I like
205 00:27:08.680 ⇒ 00:27:11.440 Awaish Kumar: To work in a fast-paced environment.
206 00:27:11.680 ⇒ 00:27:18.980 Awaish Kumar: Overall, I… Like, learn and grow, at the same speed as the company.
207 00:27:19.560 ⇒ 00:27:27.029 Awaish Kumar: So, yeah, that’s the… Like, the… and with… with a lot of,
208 00:27:28.070 ⇒ 00:27:32.190 Awaish Kumar: ownership and accountability. So, basically, you…
209 00:27:32.400 ⇒ 00:27:37.910 Awaish Kumar: The com… the startups at the… at the… of this stage, and the… Wow.
210 00:27:38.580 ⇒ 00:27:42.129 Awaish Kumar: Like, they’re pretty flexible, like, in terms of…
211 00:27:42.270 ⇒ 00:27:48.330 Awaish Kumar: What do you want to use, what tools and technologies you are… got to use.
212 00:27:48.920 ⇒ 00:27:54.200 Awaish Kumar: Setting up, like, helping set up the processes, define,
213 00:27:54.610 ⇒ 00:28:00.520 Awaish Kumar: policies and all of these things. Basically, you can leave a mark, while…
214 00:28:00.690 ⇒ 00:28:08.969 Awaish Kumar: being at its growth stage or startup companies. So that’s what I… I like being at Brainforge, also, like, we…
215 00:28:09.130 ⇒ 00:28:16.490 Awaish Kumar: It’s pretty fast-paced, we leverage AI for our development, not just engineers, but…
216 00:28:16.640 ⇒ 00:28:23.439 Awaish Kumar: anyone, everyone in the company, like, from sales, marketing, everybody uses AI.
217 00:28:23.960 ⇒ 00:28:29.260 Awaish Kumar: Everybody ever started using IDs.
218 00:28:29.260 ⇒ 00:28:31.360 Anish Gupta: Basically, speed up their…
219 00:28:31.370 ⇒ 00:28:35.510 Awaish Kumar: the workflows, and,
220 00:28:36.650 ⇒ 00:28:41.630 Awaish Kumar: We are pretty flexible in terms of what tools you want to use, how you want to use it.
221 00:28:42.480 ⇒ 00:28:48.630 Awaish Kumar: Yeah, so that’s… that’s basically… gives you a lot of… Flexibility.
222 00:28:49.250 ⇒ 00:28:52.389 Awaish Kumar: And ownership of the tools and systems
223 00:28:53.200 ⇒ 00:29:01.580 Awaish Kumar: But also, it comes with a lot of… responsibility and accountability that So you have to, basically.
224 00:29:02.660 ⇒ 00:29:09.989 Awaish Kumar: Be accountable for… For what you are doing, what is the… Outcome of your work, right?
225 00:29:09.990 ⇒ 00:29:12.200 Anish Gupta: So, you can do…
226 00:29:12.410 ⇒ 00:29:16.179 Awaish Kumar: You can ask for any tools you want, but you have to, like.
227 00:29:16.950 ⇒ 00:29:24.890 Awaish Kumar: be account… be accountable for the outcome of the tool, like, if that’s not great, but if there are better options.
228 00:29:25.060 ⇒ 00:29:32.590 Awaish Kumar: That’s… that’s… That gives you, like, Privilege, and also responsibility.
229 00:29:33.350 ⇒ 00:29:42.970 Anish Gupta: Okay, okay, so you got, like, a lot of autonomy as an individual developer, but also just, like, as a team, as a company as a whole, which is… that sounds… that’s pretty cool, that’s pretty interesting. That’s nice.
230 00:29:43.590 ⇒ 00:29:44.370 Awaish Kumar: Yeah.
231 00:29:44.520 ⇒ 00:29:45.950 Awaish Kumar: Any other questions?
232 00:29:47.130 ⇒ 00:29:48.850 Anish Gupta: No, I think that should be it for me.
233 00:29:49.960 ⇒ 00:29:56.650 Awaish Kumar: Okay, so I think, we are pretty… we are pretty much at the end of our interview.
234 00:29:58.010 ⇒ 00:30:00.649 Awaish Kumar: I’m going to submit my feedback with the team.
235 00:30:01.100 ⇒ 00:30:07.239 Awaish Kumar: And operations team is going to reach out to you, for the next steps.
236 00:30:07.410 ⇒ 00:30:08.740 Anish Gupta: Okay. Right.
237 00:30:08.960 ⇒ 00:30:10.560 Anish Gupta: Alright, thank you so much. Thanks for taking the time.