Meeting Title: Brainforge Interview w- Pranav Date: 2026-05-11 Meeting participants: Farhaj Shahid, Fireflies.ai Notetaker Farhaj Shahid, Pranav Narahari
WEBVTT
1 00:01:28.210 ⇒ 00:01:29.350 Pranav Narahari: Hey, Farage!
2 00:01:30.040 ⇒ 00:01:32.210 Farhaj Shahid: Hi, Pranav, how’s it going?
3 00:01:32.210 ⇒ 00:01:33.660 Pranav Narahari: Good, good, how are you doing?
4 00:01:33.850 ⇒ 00:01:34.830 Farhaj Shahid: I’m doing well.
5 00:01:35.010 ⇒ 00:01:35.730 Farhaj Shahid: Nice.
6 00:01:35.730 ⇒ 00:01:36.300 Pranav Narahari: size.
7 00:01:36.440 ⇒ 00:01:39.940 Pranav Narahari: Pretty good, pretty good. Give me just one second to set up here.
8 00:01:40.140 ⇒ 00:01:41.320 Farhaj Shahid: Of course, take your time.
9 00:01:41.580 ⇒ 00:01:42.350 Pranav Narahari: Thank you.
10 00:02:08.740 ⇒ 00:02:12.480 Pranav Narahari: How did your, first-round interview go? Did you… who did you speak with?
11 00:02:13.240 ⇒ 00:02:17.810 Farhaj Shahid: I spoke with Sam, yeah, it was… it was a good conversation, yeah.
12 00:02:18.930 ⇒ 00:02:23.760 Farhaj Shahid: really chilled and, for the other interviews I’ve done. I really enjoyed that.
13 00:02:24.200 ⇒ 00:02:31.319 Pranav Narahari: Nice, yeah, yeah. Yeah, so Sam kind of… he, he probably already went into this, but he’s more of, like, a…
14 00:02:31.870 ⇒ 00:02:42.699 Pranav Narahari: a individual contributor on the AI service side. So what he helps with is, on the AI service side, kind of being a little bit more of a, like, a senior role.
15 00:02:42.850 ⇒ 00:02:57.390 Pranav Narahari: What I do is… I am more of, like, a technical architect, as well as a client relationship owner. So I’ll be kind of having most of the direct communication with the client.
16 00:02:57.630 ⇒ 00:03:08.800 Pranav Narahari: And then I’ll come to the team, kind of defining a product, also coming up with, like, what the technical implementation will look like. But, you know, it’s a small team.
17 00:03:09.050 ⇒ 00:03:16.980 Pranav Narahari: it’s very collaborative, so Sam and I are… we work pretty close with one another to figure out what a solution looks like, so…
18 00:03:16.980 ⇒ 00:03:17.520 Farhaj Shahid: Nice.
19 00:03:18.180 ⇒ 00:03:33.719 Pranav Narahari: Yeah, so that’s, like, a little bit about, like, what I do at, Brainforge. I’m curious, kind of, like, in just, like, maybe a quick 60 seconds, before we get into the interview, just tell me a little bit about, you know, your background, and then, like, what brings you to, Brainforge.
20 00:03:34.570 ⇒ 00:03:58.450 Farhaj Shahid: Okay, perfect, yeah. So, hi, I’m Farhaj, I’m a current senior at Bucknell. I’ve been studying computer science, and I’ve been also pursuing a visual design minor here. I’m someone that’s both left-brained and right-brained. I love to think critically and analytically about systems, but I love to build technology that is functional, but also looks and feels amazing, so the visual aspect is also something that I take a lot of pride in.
21 00:03:58.690 ⇒ 00:04:11.330 Farhaj Shahid: I interned for Prometric last summer as a full-stack engineering intern, and I continued my senior year as an engineer, working on, like, client-side problems and doing some production configes.
22 00:04:11.440 ⇒ 00:04:18.240 Farhaj Shahid: Moreover, I’ve just been building… I’m a builder apart. I’ve been shipping inside and outside the classroom these past 4 years.
23 00:04:18.380 ⇒ 00:04:22.549 Farhaj Shahid: I founded my own, development agency, and have
24 00:04:22.700 ⇒ 00:04:30.959 Farhaj Shahid: products for actual clients from 0 to 1, one of them being Waitlist, which is a digital UK health, firm.
25 00:04:31.300 ⇒ 00:04:33.150 Farhaj Shahid: And then I also built…
26 00:04:33.150 ⇒ 00:04:57.280 Farhaj Shahid: a full web app for my hometown’s, co-working space, just so I could get free office space to work for the summer and pursue my projects. So yeah, really into, like, the AI ecosystem as well, and chronically online and Twitter, following, like, the Frontier Labs and researchers, what they’re working on, because super cool stuff happening at the moment. And that’s what led me to Brainforge, because you guys are doing some cool
27 00:04:57.330 ⇒ 00:05:00.059 Farhaj Shahid: Who, projects and clients as well.
28 00:05:00.580 ⇒ 00:05:17.830 Pranav Narahari: Yeah, no, that sounds awesome. Thanks for that overview. How I kind of like to run through these interviews is, first, we kind of go through a project that you’re super, passionate about, that’s kind of similar to the type of projects that we work at here, which are…
29 00:05:18.020 ⇒ 00:05:20.670 Pranav Narahari: mostly, like, RAG-based, applications.
30 00:05:20.670 ⇒ 00:05:21.470 Farhaj Shahid: Sure, sure.
31 00:05:21.470 ⇒ 00:05:37.580 Pranav Narahari: Anything that involves creating embeddings, setting up some type of embedding pipeline, or anything involving, like, MCP servers, building out a chat space with the integrations to other SaaS tools or other just, softwares in general.
32 00:05:38.100 ⇒ 00:05:38.730 Farhaj Shahid: Nice.
33 00:05:39.050 ⇒ 00:05:47.719 Pranav Narahari: So, what comes to mind when I bring that up? Is there a certain project that you felt like you had a lot of ownership of that you’d like to talk about right now?
34 00:05:48.270 ⇒ 00:05:58.340 Farhaj Shahid: 100%. So, when you mentioned RAG pipelines, I actually developed a RAG pipeline for one of my projects, Fazel, which was an ed tech platform geared towards high school students.
35 00:05:58.470 ⇒ 00:06:20.159 Farhaj Shahid: So the first stage was, just giving a very nice, like, web platform so students can browse the resources and, like, exams they wanted to practice in. So I… so all these… the resources were behind really terrible websites and UI, and you took, like, multiple ads and, like, different clicks to get through to the paper you wanted before being served an ad in your face.
36 00:06:20.160 ⇒ 00:06:24.510 Farhaj Shahid: And that was really distracting to me as a student, and so, like, I,
37 00:06:24.510 ⇒ 00:06:36.719 Farhaj Shahid: the opportunity to, A, first scrape the entire dataset, which had more than, like, 50,000 plus, like, papers, and, and then set up a database for them, and so you could get your,
38 00:06:36.720 ⇒ 00:06:49.510 Farhaj Shahid: a resource really quickly. Second, I’ve developed a rack pipeline, so students could have… could upload the papers that they had solved, either, written or electronically, and then get it graded through,
39 00:06:49.510 ⇒ 00:07:06.859 Farhaj Shahid: error pipeline, basically, because sometimes the bottleneck is students tend to overestimate the quality of their answers, and so I just wanted to make a cool tool to help with that. So I was in charge of the entire thing. I also did some marketing for it on social media.
40 00:07:06.950 ⇒ 00:07:20.899 Farhaj Shahid: it got, like, 7 million plus views on, like, some of the reviews that we did, and we had, like, users from more than, like, 12 countries at the time. So that was, like, a cool project that I did. Got, like.
41 00:07:20.900 ⇒ 00:07:22.760 Pranav Narahari: just on how to.
42 00:07:22.760 ⇒ 00:07:23.260 Farhaj Shahid: Of course.
43 00:07:23.260 ⇒ 00:07:31.950 Pranav Narahari: how the product works, because, yeah, I’d love to kind of just talk about, okay, what was the product meant to do, and then we kind of talk about the technical implementation.
44 00:07:31.950 ⇒ 00:07:33.460 Farhaj Shahid: So… so…
45 00:07:33.520 ⇒ 00:07:35.349 Pranav Narahari: Yeah, no, go ahead.
46 00:07:35.350 ⇒ 00:07:43.310 Farhaj Shahid: Yeah, first and foremost, it was supposed to be a really, easy-to-use and fast resource, So…
47 00:07:43.430 ⇒ 00:08:01.699 Farhaj Shahid: platform, so you have all these past papers that are… you can just practice, view them, download them, and then you could practice and take a quiz online. That was a second component. I needed some API credits to do that, and so the… I applied for a grant here at our School of Entrepreneurship Center, got about 1,000 credits to do that.
48 00:08:01.700 ⇒ 00:08:06.790 Farhaj Shahid: And for the time it was live, we got a bunch of users doing that as well.
49 00:08:06.790 ⇒ 00:08:15.599 Farhaj Shahid: So it was a lot of data scraping, and then also, RAD pipe automatic grading. So those are the two main components, I would say.
50 00:08:16.610 ⇒ 00:08:25.399 Pranav Narahari: Okay, so when you say papers, these aren’t, like, essays, right? They sound like… or are they, like, kind of similar to essays?
51 00:08:25.730 ⇒ 00:08:42.540 Farhaj Shahid: So, most of them, for the first trial set, I wanted to be a bit more deterministic, so I first took a lot of multiple-choice questions, so you had, a lot of answers right then and there. And then the second I started with was short essay questions, not really gigantic, like, English language ones, but…
52 00:08:42.730 ⇒ 00:08:48.620 Farhaj Shahid: Like, similar to, like, a biology, or… related subjects.
53 00:08:48.820 ⇒ 00:08:52.150 Farhaj Shahid: So, yeah, those were the two first exam types.
54 00:08:53.300 ⇒ 00:08:59.730 Pranav Narahari: Okay, and did you focus on any specific, courses, or any specific, majors?
55 00:09:00.260 ⇒ 00:09:12.719 Farhaj Shahid: Yep, so a lot of the STEM, all the STEM, subjects had an MCQ component associated with them, as long as some short answer question ones, so I focused on, like, the big five.
56 00:09:12.840 ⇒ 00:09:16.270 Farhaj Shahid: For them, chemistry, biology, physics.
57 00:09:16.490 ⇒ 00:09:18.910 Farhaj Shahid: Because those could be mapped really easily.
58 00:09:19.510 ⇒ 00:09:22.389 Pranav Narahari: And so, who would be using these? Like, who would.
59 00:09:22.390 ⇒ 00:09:22.810 Farhaj Shahid: So…
60 00:09:22.980 ⇒ 00:09:27.049 Pranav Narahari: The quizzes? Who would be, like, using this interface?
61 00:09:27.450 ⇒ 00:09:35.630 Farhaj Shahid: Yes, so, these quizzes, were pre-generated by, like, an exam board, so I had, like.
62 00:09:36.210 ⇒ 00:09:55.699 Farhaj Shahid: historic database of basically all the questions, for example, since 2006 to 2026, right? For instance, so the, like, and so students could choose if they wanted a random paper, or if they wanted to practice a specific past paper. And because this was made for the British system, so not…
63 00:09:55.750 ⇒ 00:10:06.480 Farhaj Shahid: the GCSE, GCE, or A-level system, which is pretty common in Europe and some schools in the US as well.
64 00:10:07.330 ⇒ 00:10:15.569 Pranav Narahari: Okay, interesting, yeah. I think I’m not super familiar with that system, but that sounds cool. I think, what I’m more interested in for this interview is…
65 00:10:15.570 ⇒ 00:10:29.979 Pranav Narahari: the technical implementation. So, talk a little bit to, you know, this web scraping component, how you manage to create all of these different… or pull in papers, from all these different subjects, and…
66 00:10:29.980 ⇒ 00:10:38.910 Pranav Narahari: Yeah. How that ended up, because it seems like the main purpose of this was to generate new quizzes for, users, right, for their studying purposes?
67 00:10:39.500 ⇒ 00:11:04.429 Farhaj Shahid: It was… no, so they… students… that was… that was in the roadmaps for the next iteration, to be able to generate new courses on the fly. The first set was to just give students an exam simulation, basically, online, like, if they were doing that exam in person at the MCQ paper component, and so it would quickly help them grade it and see where they were
68 00:11:04.430 ⇒ 00:11:07.940 Farhaj Shahid: went wrong, and give some, you know, suggestions on where they could do better, so…
69 00:11:08.290 ⇒ 00:11:18.740 Farhaj Shahid: So that was the main goal. Some of the technical implementations was, like, used React on the front end, I used API service on the back end, I used…
70 00:11:18.900 ⇒ 00:11:21.570 Farhaj Shahid: I’m ChromaDB as my vector database.
71 00:11:22.420 ⇒ 00:11:25.529 Farhaj Shahid: And, like, the first…
72 00:11:25.530 ⇒ 00:11:48.959 Farhaj Shahid: stage was the hardest, I think, like, getting all the data and getting it cleaned to first even get it through the embedding stage. So I used… first, I tried a bunch of different things. I used a Selenium and a Playwrights paper to get it and automate it using those really bad websites that I’d mentioned beforehand, right? But that was taking too much time. So I sort of reverse-engineered their API and found it. It did not have,
73 00:11:48.960 ⇒ 00:12:07.510 Farhaj Shahid: like, for education, so I was just able to extract a really clean folder of the data that I needed of exam PDFs, basically. And the second stage was messy was converting all of those messy PDFs into structured chunks, so to speak, so…
74 00:12:07.550 ⇒ 00:12:23.680 Farhaj Shahid: I opted for a slightly longer chunking strategy, just so that it could get enough context of the entire question, and I made sure it had, like, overlapping, an overlapping window, so any information in, like, the corners doesn’t get lost on the edges.
75 00:12:24.680 ⇒ 00:12:44.730 Farhaj Shahid: Then I used the available open source OpenAI embedding model to embed it and store those chunks in the vector database, and then I just used, like, the top 5, like, most frequent chunks to retrieve the outputs, and I used, like, Gemini to be, like, the main LLM to determine some of the outputs.
76 00:12:44.730 ⇒ 00:13:00.219 Farhaj Shahid: because I had really good multimodal capabilities, so if, like, some of the, like, extraction, wasn’t, like, optimal, because it had still access to the root PDF, it could, like, that if it wanted to.
77 00:13:00.310 ⇒ 00:13:24.550 Farhaj Shahid: And, sometimes, and then towards the end, I was testing, like, maybe using, like, a stronger model to be, like, sort of the judge, right, of the MLM or, like, RAG infrastructure. So I experimented with Claude, like, Opus at the time, which, like, really burned through a lot of the tokens. But it did give a lot of good critiques to some of the nuances that maybe, like, the cheaper Gemini model
78 00:13:24.550 ⇒ 00:13:25.330 Farhaj Shahid: missed.
79 00:13:25.330 ⇒ 00:13:27.800 Farhaj Shahid: But Gemini had a logger, yeah, I can’t…
80 00:13:27.800 ⇒ 00:13:40.670 Farhaj Shahid: Yeah, Gemini had, like, a really longer context window, so that’s why I really preferred it, because you want, you want, like, the, sort of, like, the exam memory to stay un… like, it, like, not, I guess, auto-compact, or, like…
81 00:13:41.240 ⇒ 00:13:42.849 Pranav Narahari: Okay, I have a question on…
82 00:13:42.850 ⇒ 00:13:43.310 Farhaj Shahid: Yeah.
83 00:13:43.310 ⇒ 00:13:46.489 Pranav Narahari: kind of, why you chose the RAG approach here.
84 00:13:47.080 ⇒ 00:13:54.410 Pranav Narahari: And so, it sounds like, yeah, so you’re… it was in the roadmap in the future, to be able to generate new exams.
85 00:13:54.410 ⇒ 00:13:55.320 Farhaj Shahid: Yes.
86 00:13:55.320 ⇒ 00:13:58.259 Pranav Narahari: On the exams that you scraped online.
87 00:13:58.390 ⇒ 00:13:59.600 Pranav Narahari: So…
88 00:13:59.750 ⇒ 00:14:11.730 Pranav Narahari: It seems like, with the exams that you scraped online, did… there’s… is there, like, a source of truth in terms of each of the… for the answers of each question? Like, are you scraping the answers as well?
89 00:14:12.010 ⇒ 00:14:15.129 Farhaj Shahid: Yes, so you have the question paper, yes.
90 00:14:15.130 ⇒ 00:14:29.999 Pranav Narahari: would be that for when you’re… you’re probably just gonna ask word for word, like, what that question… the question to a user of your application. Right? And so then, where was the need for embeddings in,
91 00:14:30.520 ⇒ 00:14:35.380 Pranav Narahari: in this… in this interface. Like, why didn’t you just have, you know, the user wants…
92 00:14:35.570 ⇒ 00:14:37.479 Pranav Narahari: This type of exam.
93 00:14:37.680 ⇒ 00:14:46.739 Pranav Narahari: So then you just show them that exam with the answers redacted, and then you just use an LLM as a judge for their response versus the source of truth response.
94 00:14:48.130 ⇒ 00:14:50.669 Farhaj Shahid: Okay, cause, okay, so…
95 00:14:51.320 ⇒ 00:14:57.369 Farhaj Shahid: To make sure I understand that correctly, so you’re saying, why is there a need to use a non-deterministic approach in the first place?
96 00:14:59.920 ⇒ 00:15:03.399 Farhaj Shahid: based on the MCQ’s, I guess.
97 00:15:03.400 ⇒ 00:15:07.580 Pranav Narahari: So even LLM as a judge would be a non-deterministic approach, right?
98 00:15:07.580 ⇒ 00:15:08.150 Farhaj Shahid: Yes.
99 00:15:08.150 ⇒ 00:15:16.420 Pranav Narahari: So, my question is, I’m not seeing the benefit of using, like, embeddings in this approach.
100 00:15:17.280 ⇒ 00:15:31.280 Pranav Narahari: Okay. Can you kind of tell me a little bit about, like, why you chose that, that approach versus just, using LLM as a judge on a per response? Like, just, just comparing each response with the source of truth response?
101 00:15:32.410 ⇒ 00:15:42.160 Farhaj Shahid: Okay, the purpose for that was purely for experimentation purposes, like, you know, and just seeing, whether or not it would perform better than just having a big alert.
102 00:15:42.160 ⇒ 00:15:53.000 Farhaj Shahid: output. And then the second use case of why I even use RAG in the first place is because, like, you can have a deterministic, like, I guess, MCQ question just mapped
103 00:15:53.000 ⇒ 00:15:57.260 Farhaj Shahid: The answer. But, parsing that, takes a lot of time.
104 00:15:58.350 ⇒ 00:16:17.259 Farhaj Shahid: And a lot of manpower that I didn’t have at the time. The second use case was it completely that fails, if there’s even somewhat of a semi-short answer question, response that you want to type out, or even solve, right? And because we wanted this solution to sort of be sort of all-encompassing.
105 00:16:17.460 ⇒ 00:16:23.470 Farhaj Shahid: Like, I chose that approach because, you would get way better answers and feedbacks based on
106 00:16:23.470 ⇒ 00:16:48.080 Farhaj Shahid: on shorter answer-based questions. And then we also had question papers and also the answer keys, and we also had, for example, some key artifacts that they made, like, for example, like an examiner… examiner critique of the overall, like, years, right? So they have sample examples, they’re graded answers, what failed, what didn’t in that answer, so that’s sort of what I used as well.
107 00:16:48.080 ⇒ 00:16:51.250 Farhaj Shahid: And some of the system prompts to train it.
108 00:16:51.250 ⇒ 00:17:08.290 Farhaj Shahid: The other part is I didn’t think fine-tuning would be the answer here as well, because you have new exams coming in every, like, couple or so months and years, right? So you need to keep just, like, updating the database, and fine-tuning would have been more expensive there at the time.
109 00:17:08.290 ⇒ 00:17:13.969 Farhaj Shahid: So, like, did that answer some of your questions? If not, feel free to, like, ask me again, like, I would.
110 00:17:13.970 ⇒ 00:17:14.599 Pranav Narahari: Yeah, so…
111 00:17:14.609 ⇒ 00:17:15.639 Farhaj Shahid: Left hand to get all the.
112 00:17:15.640 ⇒ 00:17:31.130 Pranav Narahari: Yeah, your first kind of, response is just, like, it’s for experimentation. Totally understand that. You know, building these applications are fun, you know, like, you’re trying to get your feet wet into, all these different tools and abilities that, like.
113 00:17:31.240 ⇒ 00:17:46.559 Pranav Narahari: are out there, and what is being used for, like, some of these production-grade applications as well. Embeddings are, super useful. I think what we’re noticing nowadays, though, is that the context windows of these LLMs are already pretty high, so then even if you’re…
114 00:17:46.620 ⇒ 00:17:57.560 Pranav Narahari: submitting a whole essay for a response, right? A 2-3 page essay. You can just use LLM as a judge to compare, okay, what is a source of truth essay? The…
115 00:17:57.980 ⇒ 00:18:04.539 Pranav Narahari: they respond, responded by the user, and then comparing the two to make sure that they hit on key points. Yep.
116 00:18:04.870 ⇒ 00:18:11.609 Pranav Narahari: That would probably be the best approach here, is from the… from what I understand about this project.
117 00:18:11.880 ⇒ 00:18:17.519 Pranav Narahari: I wonder… I’m just wondering, like, do you agree with that? Do you see actually, like, a…
118 00:18:17.600 ⇒ 00:18:22.289 Pranav Narahari: a benefit to this RAG pipeline that I’m not seeing right now.
119 00:18:22.350 ⇒ 00:18:39.619 Pranav Narahari: Yeah, I’m just trying to get your understanding, because, like, a really important thing at this position, right, you know, you’re not going to be building things just for experimentation. Yes. You’re going to be building things because, you’re going to be trying to hit a timeline, you’re going to be trying to not overcomplicate things, so it’s really.
120 00:18:39.620 ⇒ 00:18:39.940 Farhaj Shahid: Yes.
121 00:18:40.190 ⇒ 00:18:53.739 Pranav Narahari: to, not just make a solution because you know it’s gonna work, it’s also making the optimal solution given all the parameters that the client gives you. So, yeah, that’s just kind of where, like, my question’s coming from. So, yeah, I’m just wondering…
122 00:18:53.740 ⇒ 00:18:54.180 Farhaj Shahid: Hmm.
123 00:18:54.180 ⇒ 00:18:55.879 Pranav Narahari: Kind of going back, like.
124 00:18:56.460 ⇒ 00:19:01.530 Pranav Narahari: Do you kind of see what I’m saying in terms of LLM as a judge versus your… the RAG pipeline?
125 00:19:02.870 ⇒ 00:19:19.939 Farhaj Shahid: I do see where you’re coming from, and definitely if I were to rebuild the project again now, I’d have to take a bunch of different approaches. And for me personally, like, I think there’s been a lot of improvements in drag and, like, the way we retrieve, I think, in the past year.
126 00:19:20.140 ⇒ 00:19:43.190 Farhaj Shahid: based on its own, like, there’s some methods now that train on, like, I guess the semantics or the hierarchy of the document, so that could be even easier for me, because if you know if a person needs to practice paper from 2017 for physics, maybe, you can direct your agent to do a much more refined research, and that would even be, like, better for the context printer that it has.
127 00:19:44.200 ⇒ 00:19:45.350 Pranav Narahari: Interesting, yeah.
128 00:19:45.490 ⇒ 00:19:46.600 Pranav Narahari: Okay, cool.
129 00:19:48.380 ⇒ 00:19:59.670 Pranav Narahari: Yeah, so then, usually after, like, we kind of talk about a, a project, like, that you just kind of described, I kind of go into specific scenarios that we’ve had here at Brainforge.
130 00:19:59.930 ⇒ 00:20:06.440 Pranav Narahari: So, here’s one scenario, I’m just curious to see, like, how you think about… think through these problems.
131 00:20:06.730 ⇒ 00:20:11.760 Pranav Narahari: So, and also, I’m gonna just turn off my video real quick. I think my Wi-Fi is a little bit laggy.
132 00:20:11.870 ⇒ 00:20:12.490 Pranav Narahari: Okay.
133 00:20:12.490 ⇒ 00:20:13.200 Farhaj Shahid: Go for it.
134 00:20:13.200 ⇒ 00:20:13.800 Pranav Narahari: Yeah.
135 00:20:17.080 ⇒ 00:20:30.010 Pranav Narahari: But, yeah, so one of our clients in the past was, like, an e-commerce platform, or an e-commerce company, and what their business was is managing a bunch of different other, e-commerce brands.
136 00:20:30.670 ⇒ 00:20:44.030 Pranav Narahari: And so, what we built for them is, like, a standalone application that connects to various different softwares. Shopify, Meta Ads, Google Ads, to name a few.
137 00:20:44.160 ⇒ 00:20:58.210 Pranav Narahari: One problem that the client brought up with us after we had already pushed to production was that when they would ask questions, to the chat interface and had the Shopify MCP enabled, they would get back
138 00:20:58.440 ⇒ 00:21:00.110 Pranav Narahari: data that…
139 00:21:00.420 ⇒ 00:21:11.529 Pranav Narahari: did not match what they would see in the Shopify platform. So, like, for example, they would ask, what are my last 7 days of revenue? And then they would get back a response that, as if
140 00:21:11.690 ⇒ 00:21:24.870 Pranav Narahari: you know, the AI knew exactly what the last 7 days of revenue was, and it would give back numbers, but then if you were to fact-check that against what’s actually in the Shopify dashboard, you would see that they… the numbers don’t match at all.
141 00:21:25.780 ⇒ 00:21:26.530 Farhaj Shahid: Okay.
142 00:21:26.690 ⇒ 00:21:36.609 Pranav Narahari: So, just given that information right there, what would be your next steps to kind of further, like, just diagnose this issue, and then we can kind of go into potential solutions?
143 00:21:37.180 ⇒ 00:21:42.420 Farhaj Shahid: Okay, I would, start by, like, first, reproduce the issue,
144 00:21:43.000 ⇒ 00:21:56.460 Farhaj Shahid: Then maybe some different iterations where it needs to reference a knowledge base of truth that I can double-check it against. It continues, I guess, hallucinating or producing entirely fake data.
145 00:21:56.880 ⇒ 00:22:05.660 Farhaj Shahid: then, I would, I guess, further investigate this issue, to see where is it going wrong, and I guess, depending on
146 00:22:05.970 ⇒ 00:22:21.489 Farhaj Shahid: like, the technical details of how this entire system was, like, unconfigured, would, like, foresee the tool calling, like, to see, like, if it’s an agent that is supposed to be answering these questions and making those decisions by itself.
147 00:22:21.550 ⇒ 00:22:28.519 Farhaj Shahid: What type of tool calling it’s using, what’s the reasoning behind that, to see if I can trace an error there.
148 00:22:28.680 ⇒ 00:22:31.040 Farhaj Shahid: If it’s more for, like, a fixed…
149 00:22:31.560 ⇒ 00:22:42.719 Farhaj Shahid: Orchestration, see then, like, if there, like, the specific node that is responsible for retrieving, is that failing anywhere?
150 00:22:42.810 ⇒ 00:23:02.089 Farhaj Shahid: And I guess, depending on technical implementation, there’s, like, some, like, there are ways you could, like, even go even deeper within the stack and see where the thing, fails. Maybe it just doesn’t have access, like, maybe I could recheck that if there’s anything else.
151 00:23:02.220 ⇒ 00:23:10.110 Farhaj Shahid: And, like, I guess the microservices, maybe, that connect these different services? Maybe there’s something that’s going wrong in the pipeline.
152 00:23:10.290 ⇒ 00:23:13.600 Farhaj Shahid: That’s Guess that?
153 00:23:13.600 ⇒ 00:23:14.000 Pranav Narahari: No, it doesn’.
154 00:23:14.000 ⇒ 00:23:15.559 Farhaj Shahid: That’s how it would start.
155 00:23:15.910 ⇒ 00:23:20.159 Pranav Narahari: Okay, yeah, that’s, like, a good first, like.
156 00:23:20.330 ⇒ 00:23:24.990 Pranav Narahari: Good start. Let me give you kind of a little bit more information, too.
157 00:23:26.120 ⇒ 00:23:29.290 Pranav Narahari: So, like, if you were to look into the…
158 00:23:30.760 ⇒ 00:23:33.469 Pranav Narahari: The thinking generation logs of this.
159 00:23:33.920 ⇒ 00:23:36.960 Pranav Narahari: Of the LLM request.
160 00:23:36.960 ⇒ 00:23:37.930 Farhaj Shahid: Right? Yep.
161 00:23:38.300 ⇒ 00:23:43.790 Pranav Narahari: We noticed that, the AI was actually generating data.
162 00:23:44.230 ⇒ 00:23:44.710 Farhaj Shahid: Mmm.
163 00:23:44.710 ⇒ 00:23:51.180 Pranav Narahari: after the MCP tool didn’t, didn’t…
164 00:23:52.030 ⇒ 00:23:55.519 Pranav Narahari: bring back the Shopify data. Basically, it failed.
165 00:23:55.520 ⇒ 00:23:57.570 Farhaj Shahid: Okay, it failed, okay, so the connection…
166 00:23:57.570 ⇒ 00:24:03.310 Pranav Narahari: And then the LLM saw that it failed and said, okay, let me just generate data, and then output that.
167 00:24:04.650 ⇒ 00:24:11.110 Pranav Narahari: So as humans, you know, me and you know that this is not the right… this is not the right solution to the problem.
168 00:24:11.110 ⇒ 00:24:11.640 Farhaj Shahid: you have…
169 00:24:11.640 ⇒ 00:24:16.680 Pranav Narahari: So, given that information.
170 00:24:16.680 ⇒ 00:24:18.080 Farhaj Shahid: What are your next steps?
171 00:24:18.080 ⇒ 00:24:20.529 Pranav Narahari: You would put in place to, like, fix this.
172 00:24:20.850 ⇒ 00:24:37.130 Farhaj Shahid: Okay, perfect. I think so, one of the things that I would immediately put in is, like, like I said, like, checking, I guess, those tool calls, the NCP server referring to failure, and in that case, the LLM should, like, handle this very gracefully. It should have, in its system.
173 00:24:37.290 ⇒ 00:24:45.820 Farhaj Shahid: immediate instruction. If a… if a knowledge-based retrieval action or tool called failed, it has to say, I don’t know, I don’t know the answer.
174 00:24:45.920 ⇒ 00:24:52.870 Farhaj Shahid: Or even try to, to retry that pool call to see maybe if I could get,
175 00:24:54.170 ⇒ 00:25:11.950 Farhaj Shahid: And then, like, even before, I guess, my next thing that I would have, potentially, like, I was thinking about, like, before I heard this more information, is maybe, like, the MCP did a lot of, bloat into the context of, I guess, the LLM tool call, depending on if…
176 00:25:11.950 ⇒ 00:25:21.999 Farhaj Shahid: It’s configured properly, or if it’s, like, optimized or not, because that can also, if it’s using up most of the context window, some of the answers that we might get also might be a bit dumber on the dumber side.
177 00:25:22.390 ⇒ 00:25:27.179 Farhaj Shahid: So those would be some of the two immediate things, after hearing this.
178 00:25:29.070 ⇒ 00:25:34.689 Pranav Narahari: Yeah, I think that’s, those are all… That’s really good insights.
179 00:25:35.050 ⇒ 00:25:39.020 Pranav Narahari: Another type of… like, let me just keep on giving you more information just to see how.
180 00:25:39.020 ⇒ 00:25:40.709 Farhaj Shahid: Go for it.
181 00:25:40.710 ⇒ 00:25:55.999 Pranav Narahari: So, the LLM that we were using here to actually output a response to the user, and then, you know, full orchestration of, you know, taking in the MCP tool calls, etc, or taking in the MCP output, I should say.
182 00:25:56.360 ⇒ 00:26:03.220 Pranav Narahari: was… I think we were using, like, Sonnet, and we had thinking enabled for this
183 00:26:03.550 ⇒ 00:26:05.410 Pranav Narahari: For this use case.
184 00:26:05.520 ⇒ 00:26:17.139 Pranav Narahari: What do you think about having thinking enabled here? Do you think that’s a pro? Do you think that’s a con? What are, like, the benefits? What are the cons of having thinking being enabled for an LLM?
185 00:26:17.930 ⇒ 00:26:25.390 Farhaj Shahid: Okay, I think, it’s… I think depending on the task, either enhance it, not to degrade the answer.
186 00:26:25.740 ⇒ 00:26:30.370 Farhaj Shahid: Thinking about retrieving a specific… I apologize.
187 00:26:30.520 ⇒ 00:26:39.509 Farhaj Shahid: I think the thinking component would be before the total call, even, where it decides, okay, the user had this query to generate
188 00:26:39.650 ⇒ 00:26:51.039 Farhaj Shahid: the past 3 weeks, for example, maybe it then gets a specific time frame from the user, and then it thinks to itself, okay, now I need to initiate the tool call.
189 00:26:51.300 ⇒ 00:27:02.739 Farhaj Shahid: from within the database, and these are the day parameters. The next stage, I guess, where the next stage should not, I think, handle this on it, it should be handled by Haiku, or even, like, a lower
190 00:27:02.970 ⇒ 00:27:08.839 Farhaj Shahid: like a faster LLM, because it should just only help execute.
191 00:27:08.950 ⇒ 00:27:13.810 Farhaj Shahid: the next stage in the workflow, that has already been, I guess, predefined.
192 00:27:14.690 ⇒ 00:27:15.070 Pranav Narahari: Gotcha.
193 00:27:15.800 ⇒ 00:27:20.880 Pranav Narahari: What makes you say, like, it should be faster? Like, why does speed matter here?
194 00:27:21.770 ⇒ 00:27:29.300 Farhaj Shahid: Because you could have a faster model here with the temperature to a lower quantity for a more deterministic
195 00:27:29.960 ⇒ 00:27:37.060 Farhaj Shahid: input, because… Sometimes, if you have thinking enabled, then maybe with the perimeters of text.
196 00:27:37.160 ⇒ 00:27:37.860 Farhaj Shahid: That’s…
197 00:27:37.990 ⇒ 00:27:52.629 Farhaj Shahid: degree, it could make up a more creative response, or hallucinate some data, I guess. Yeah. Because… yeah, so maybe, I guess, putting up some more guardrails around this, around this orchestration layer might help get more around it.
198 00:27:53.500 ⇒ 00:28:03.399 Pranav Narahari: Yeah, that makes sense. I think, you know, speed in and of itself is not the issue, but it sounds like, you know, that temperature, and having thinking disabled.
199 00:28:03.760 ⇒ 00:28:20.280 Pranav Narahari: Those are just going to lead to faster responses, because you’re going to have less, less tokens being utilized for reasoning, right? So, yeah, I totally agree. I think, that was one flaw in the system that we had, which was by having
200 00:28:20.730 ⇒ 00:28:23.560 Pranav Narahari: A more free-thinking capability.
201 00:28:23.930 ⇒ 00:28:24.370 Farhaj Shahid: Weird.
202 00:28:24.370 ⇒ 00:28:32.629 Pranav Narahari: called LLM. It had the ability to just produce these very non-deterministic outputs, like, you know.
203 00:28:32.900 ⇒ 00:28:44.200 Pranav Narahari: creating data. And I actually, I think it’s a really good insight as well to have maybe thinking enabled for routing different MCP tool calls. However, for the synthesis of the output.
204 00:28:44.460 ⇒ 00:28:51.830 Pranav Narahari: A more deterministic, configured LLM is probably the way to go.
205 00:28:52.140 ⇒ 00:29:00.829 Pranav Narahari: I just want to run through one more example before we run into questions. Do you have a hard stop at one… at a 30-minute mark, or…
206 00:29:00.980 ⇒ 00:29:03.429 Farhaj Shahid: No, feel free to go over.
207 00:29:03.430 ⇒ 00:29:04.810 Pranav Narahari: Okay, cool. Yeah.
208 00:29:04.810 ⇒ 00:29:09.389 Farhaj Shahid: Yeah, it’s been a cool… yeah, it’s been a cool conversation, like, I’m loving these brain teasers.
209 00:29:09.550 ⇒ 00:29:17.630 Pranav Narahari: Okay, cool, cool. Yeah, this is exactly the type of problems you’re gonna have to be facing if you… if you join Brainforge, so… Yeah, one other…
210 00:29:18.680 ⇒ 00:29:21.780 Pranav Narahari: One other design decision that I’m…
211 00:29:22.200 ⇒ 00:29:26.899 Pranav Narahari: Curious to see how you think through. Is, for another client,
212 00:29:27.470 ⇒ 00:29:32.449 Pranav Narahari: to not go into super detail, because it’s not super relevant, essentially what we did was we…
213 00:29:33.000 ⇒ 00:29:37.580 Pranav Narahari: Brought in a bunch of different documentation from across the organization.
214 00:29:37.790 ⇒ 00:29:42.900 Pranav Narahari: The class organization. And, we put it into this thing called a central dock.
215 00:29:43.010 ⇒ 00:29:48.689 Pranav Narahari: So this central dock… For our purposes, let’s just talk about one of them.
216 00:29:49.590 ⇒ 00:29:52.840 Pranav Narahari: It, you know, it’s just basically a…
217 00:29:53.010 ⇒ 00:29:55.480 Pranav Narahari: we just connected all of their PDFs.
218 00:29:56.070 ⇒ 00:30:09.440 Pranav Narahari: together into just one document, which we then embedded. So then, to a chat interface, they could ask any question about that department’s logic, and they should be outputted an answer.
219 00:30:09.980 ⇒ 00:30:13.710 Pranav Narahari: Now, the issue that we saw was that the…
220 00:30:14.830 ⇒ 00:30:28.319 Pranav Narahari: the type of content that was in each of these documents differed in structure. It wasn’t all just text. There was images, there was tables, there was diagrams…
221 00:30:28.440 ⇒ 00:30:33.389 Pranav Narahari: Which our embedding pipeline wasn’t taking…
222 00:30:33.540 ⇒ 00:30:38.359 Pranav Narahari: wasn’t embedding properly, you know, I had no way of embedding, actually.
223 00:30:38.790 ⇒ 00:30:44.889 Pranav Narahari: Now, most of the information is text, right? So…
224 00:30:45.180 ⇒ 00:30:49.000 Pranav Narahari: For the… the current solution is, we just kind of…
225 00:30:49.520 ⇒ 00:30:56.419 Pranav Narahari: took out all the images, so then those are just not being used as context. Not the best solution, but it’s the fastest solution.
226 00:30:56.420 ⇒ 00:30:57.050 Farhaj Shahid: Fair enough.
227 00:30:57.310 ⇒ 00:31:05.970 Pranav Narahari: Definitely a, A shortcut that we took, given the timeline, and, you know, we… we…
228 00:31:06.900 ⇒ 00:31:12.470 Pranav Narahari: But one thing going forward is we want to be able to support these type of media as well.
229 00:31:12.580 ⇒ 00:31:15.750 Pranav Narahari: How would you think about supporting these types of media?
230 00:31:18.870 ⇒ 00:31:24.990 Farhaj Shahid: Interesting, because, yeah, definitely when working with PDFs, it does get messy.
231 00:31:25.170 ⇒ 00:31:29.259 Farhaj Shahid: Especially when it’s filled with rich content, like tables and charts.
232 00:31:30.850 ⇒ 00:31:37.640 Farhaj Shahid: So, I think, we could either,
233 00:31:38.690 ⇒ 00:31:42.000 Farhaj Shahid: We have these certain type of media, we could use a multi…
234 00:31:42.120 ⇒ 00:31:48.129 Farhaj Shahid: To analyze all of them, and maybe input a description of what… of what…
235 00:31:49.410 ⇒ 00:31:53.049 Farhaj Shahid: In lieu of the image or the diagram.
236 00:31:54.510 ⇒ 00:32:00.640 Farhaj Shahid: Love them, because, like you said, we would like to have the capability to Arrogance.
237 00:32:01.160 ⇒ 00:32:06.760 Farhaj Shahid: these as provided knowledge base, right? So that’s one thing that comes to mind.
238 00:32:07.830 ⇒ 00:32:19.920 Farhaj Shahid: But at the same time, we would have to, like, I guess, really test and carefully do this to make sure that, A, the model, A, doesn’t hallucinate when they’re seeing these images, or inputting false information.
239 00:32:21.250 ⇒ 00:32:24.160 Farhaj Shahid: Maybe we could use, like, some…
240 00:32:24.870 ⇒ 00:32:36.899 Farhaj Shahid: If it’s, like, for example, legal or medical work, like, we could use some pre-trained multimodal models for that specific domain. Maybe that could yield better results.
241 00:32:39.670 ⇒ 00:32:44.239 Farhaj Shahid: Yeah, so… that’s… I’m thinking about it at the moment.
242 00:32:44.740 ⇒ 00:32:58.300 Farhaj Shahid: I also really wonder, like, was putting the entire knowledge base into one document, did that actually yield tangible results for you guys, or did that add, I guess, more complexities when dealing with some of the nuance?
243 00:33:00.770 ⇒ 00:33:02.130 Pranav Narahari: Yeah,
244 00:33:02.470 ⇒ 00:33:13.729 Pranav Narahari: That’s a good question as well. I think this document had, multiple uses. It wasn’t, as it was just initially scoped out,
245 00:33:14.000 ⇒ 00:33:29.679 Pranav Narahari: I kind of joined this project after it was already kind of baked in, you know, there’s already this central document. It was already part of the design, so I kind of just, like, continued with it. But the idea behind it was that this central doc
246 00:33:29.680 ⇒ 00:33:34.350 Pranav Narahari: We’ll also be used by the internal company to find all the information.
247 00:33:34.350 ⇒ 00:33:42.219 Pranav Narahari: An issue with having documentation all over the company is you don’t know what is the latest information, what should be actually referenced.
248 00:33:42.220 ⇒ 00:33:42.930 Farhaj Shahid: That’s true.
249 00:33:42.930 ⇒ 00:33:50.540 Pranav Narahari: You know, the… the actual policies versus what is outdated. So it also had that…
250 00:33:51.300 ⇒ 00:33:53.170 Pranav Narahari: Capability as well, so that was kind of.
251 00:33:53.170 ⇒ 00:33:53.740 Farhaj Shahid: Good.
252 00:33:53.740 ⇒ 00:34:13.129 Pranav Narahari: kept it in one doc. But yeah, I’m actually, now that you bring that up, I’m wondering, do you… what… what are the pros and cons about having, because what we, you know, what we could do instead of just having one file, is we could just have one folder that holds all the separate documents, right?
253 00:34:13.630 ⇒ 00:34:16.260 Pranav Narahari: What would you say is the pros and cons to either approach?
254 00:34:17.320 ⇒ 00:34:21.709 Farhaj Shahid: Okay, so… So, for one of the pros.
255 00:34:21.940 ⇒ 00:34:41.760 Farhaj Shahid: segregated documents is, for example, if you have different departments within the company that, I guess, operate differently, have different, like, layouts or systems in place already, having separate queries or tool calls for them over their knowledge base could yield better results for the workers individually might be better for their productivity.
256 00:34:42.179 ⇒ 00:34:49.480 Farhaj Shahid: Some… but another thing is, like, another con, like, if the single document is really big enough.
257 00:34:49.920 ⇒ 00:34:57.020 Farhaj Shahid: As if I’m assuming it amasses information from all… the company. It might, like.
258 00:34:57.560 ⇒ 00:35:06.210 Farhaj Shahid: drain a lot of the context for any LLM call, and it has to go through the entire master document, and it could generate
259 00:35:08.140 ⇒ 00:35:09.840 Farhaj Shahid: No. No.
260 00:35:10.440 ⇒ 00:35:22.209 Farhaj Shahid: and non-optable answers, because the sweet spot is using, like, after, like, I guess the 50% or, like, 40% mark of context usage, the answers degrade a lot in quality.
261 00:35:22.480 ⇒ 00:35:31.790 Farhaj Shahid: The LLM tends to forget, like, information in the middle of its context window, then compared to, like, the beginning or the end at the same time.
262 00:35:34.170 ⇒ 00:35:38.399 Farhaj Shahid: Some pros, though, of having a single source of truth,
263 00:35:38.530 ⇒ 00:35:45.210 Farhaj Shahid: document that the LLM can, reference, if it’s, in a manageable size, is,
264 00:35:45.290 ⇒ 00:35:58.849 Farhaj Shahid: like, I think good responses, the… because before every question, that document is injected, it has… the LLM can really reference the knowledge within that, and really help the user at the same time.
265 00:35:58.880 ⇒ 00:36:11.750 Farhaj Shahid: And if the document is structured well as well, like, the prompts or the tool calls are defined really well, sometimes it could yield for a… for…
266 00:36:11.830 ⇒ 00:36:16.400 Farhaj Shahid: A nice retrieval, and, a lot…
267 00:36:16.850 ⇒ 00:36:19.639 Farhaj Shahid: Very similar answers, like, can maybe help
268 00:36:20.130 ⇒ 00:36:23.229 Farhaj Shahid: It improves the quality of the overall system.
269 00:36:23.420 ⇒ 00:36:27.480 Farhaj Shahid: the best I could have, like, pros and cons that I’ve… I don’t think of.
270 00:36:28.300 ⇒ 00:36:35.170 Pranav Narahari: Yeah, that makes sense. Cool. I think, if you have any questions, I’m happy to answer them now before we wrap up.
271 00:36:36.870 ⇒ 00:36:52.249 Farhaj Shahid: Of course, so, like, I had a couple questions regarding, for example, the nature of the work that you were doing. Like, you mentioned you were doing a lot of, like, forward-deployed work. So, when you’re handling, I guess, most of these new clients or new projects.
272 00:36:52.380 ⇒ 00:36:59.190 Farhaj Shahid: What do you do, I guess, in the first couple of days, or, like, the first week or so, to really understand, I guess, the problem?
273 00:37:00.590 ⇒ 00:37:03.359 Pranav Narahari: Yeah, so, I think there’s…
274 00:37:03.600 ⇒ 00:37:16.320 Pranav Narahari: A couple things that I have gotten better at, to be completely, candid with you, which I think is super important in, yeah, how you accurately described as, like, being forward deployed.
275 00:37:17.700 ⇒ 00:37:32.060 Pranav Narahari: It’s, really try to understand the organization, really try to understand their incentives, trying to understand their business model, and most importantly, just understand their problems that you’re building the solution for.
276 00:37:32.560 ⇒ 00:37:48.320 Pranav Narahari: you’ll interact with all different types of clients. You’ll interact with clients that, you know, have no clue what AI can do, which can be easier in some regard, because what they’ll do is just come to you with their problems, and they fully trust you to be the…
277 00:37:49.250 ⇒ 00:37:52.649 Pranav Narahari: the… the solution architect. Yep.
278 00:37:53.110 ⇒ 00:37:59.819 Pranav Narahari: And so, that’s great. You’ll also interact with some clients that, you know, in today’s world, like.
279 00:38:00.240 ⇒ 00:38:06.499 Pranav Narahari: people are very excited about AI, so they’re reading up a lot about it. You’re also gonna be working a lot with,
280 00:38:06.740 ⇒ 00:38:19.869 Pranav Narahari: you know, leadership at these clients, so they’re going to be the ones that are most interested in seeing how AI can help their company. Maybe they’ve already run through a few different AI projects internally,
281 00:38:20.250 ⇒ 00:38:23.749 Pranav Narahari: However, it’s really important for you to be the…
282 00:38:24.240 ⇒ 00:38:27.780 Pranav Narahari: you should understand the full capabilities of AI, and you should be.
283 00:38:27.780 ⇒ 00:38:28.290 Farhaj Shahid: gift.
284 00:38:28.290 ⇒ 00:38:31.040 Pranav Narahari: Very open about pushing back on
285 00:38:31.210 ⇒ 00:38:38.580 Pranav Narahari: the solutions that are architected by the client. Because at the end of the day, you’re going to be the one building it, and so…
286 00:38:39.340 ⇒ 00:38:45.749 Pranav Narahari: You should be the one actually fully co-signing on whatever approach you’re taking.
287 00:38:46.260 ⇒ 00:38:47.300 Farhaj Shahid: 100%.
288 00:38:47.510 ⇒ 00:38:48.150 Pranav Narahari: Yeah.
289 00:38:49.090 ⇒ 00:38:58.759 Farhaj Shahid: And that sort of leads into my next question, so suppose, like, how do you, I guess, decide whether or not, like, a solution, really does require, like, leveraging LLMs or…
290 00:38:59.290 ⇒ 00:39:01.970 Farhaj Shahid: Versus more deterministic approaches.
291 00:39:03.010 ⇒ 00:39:09.659 Pranav Narahari: Yeah, I mean, I think where LLMs fit in is that they make the complexity of systems much…
292 00:39:11.160 ⇒ 00:39:16.569 Pranav Narahari: Much smaller, or they make systems that were before impossible, now possible.
293 00:39:16.860 ⇒ 00:39:17.480 Farhaj Shahid: Hmm.
294 00:39:17.480 ⇒ 00:39:30.419 Pranav Narahari: You know, non-determinism is… it has its place, where it could be a feature, but in a lot of cases, you want a very deterministic output, or at least 95% deterministic in terms of, like.
295 00:39:30.780 ⇒ 00:39:36.940 Pranav Narahari: and everything, and then maybe the wording, you want a little bit of variation. So…
296 00:39:38.110 ⇒ 00:39:50.819 Pranav Narahari: it’s really important to… to understand that, you know, it’s just, like, it’s not as simple as, just throwing a problem to ChatGPT and then just giving out the response.
297 00:39:52.400 ⇒ 00:39:55.350 Pranav Narahari: I think, yeah, so… LLM…
298 00:39:55.680 ⇒ 00:40:11.360 Pranav Narahari: understanding the application of that is super important, and then just in general, like, I think we touched upon it a little bit in the beginning of the interview, where, the design itself is very important. Nowadays, implementation has actually…
299 00:40:11.590 ⇒ 00:40:19.110 Pranav Narahari: not been as big of an issue. People can build out projects in a matter of days now, and, like, used to take months.
300 00:40:19.230 ⇒ 00:40:24.039 Pranav Narahari: However, the real bottleneck, and where the real,
301 00:40:24.630 ⇒ 00:40:28.499 Pranav Narahari: I would say critical thinking happens now, is…
302 00:40:29.270 ⇒ 00:40:34.439 Pranav Narahari: what is the thing you are trying to build? What is the exact architecture?
303 00:40:34.440 ⇒ 00:40:37.169 Farhaj Shahid: In the system design component.
304 00:40:37.170 ⇒ 00:40:39.710 Pranav Narahari: Yeah, exactly.
305 00:40:39.710 ⇒ 00:40:40.109 Farhaj Shahid: Excuse me.
306 00:40:40.110 ⇒ 00:40:55.370 Pranav Narahari: There’s a bunch of different decisions that you’re making here, which is like, okay, should we build an embedding pipeline? Should we have a data warehouse? What are the pros and cons to those approaches? And then also, what I’ve found to be really useful nowadays is,
307 00:40:56.010 ⇒ 00:41:00.129 Pranav Narahari: After you build out this technical approach for a product.
308 00:41:01.340 ⇒ 00:41:18.319 Pranav Narahari: what are the key aspects of the project that are going to make or break the product itself? Because I think, for the most part, you’ll see a lot of similarities with, the things that you’re building from a technical perspective, what you’re trying to build, you know, building, like, a dashboard, building, like, a…
309 00:41:18.500 ⇒ 00:41:25.860 Pranav Narahari: MCP integration, authentication, those type of things are more or less just, like, plug and play from
310 00:41:26.190 ⇒ 00:41:34.360 Pranav Narahari: But what is the thing that’s really critical in terms of giving the client the output that they want, and then.
311 00:41:34.360 ⇒ 00:41:35.280 Farhaj Shahid: Interesting.
312 00:41:35.280 ⇒ 00:41:42.439 Pranav Narahari: Knowing for sure that the approach that you are presenting is the best way to implement that,
313 00:41:43.370 ⇒ 00:42:02.310 Pranav Narahari: is the best way to implement that. What I’ve even stressed, too, for, like, projects that I’ve worked on is, okay, let’s… let’s build a POC of just that most critical technical part, just so we can be sure that when we build everything around it, that this core part isn’t going to be the bottleneck.
314 00:42:03.150 ⇒ 00:42:07.279 Pranav Narahari: Yeah, so this is how I kind of think about it in those first couple days.
315 00:42:07.630 ⇒ 00:42:30.519 Farhaj Shahid: Yeah, no, that’s super interesting, and thank you for giving a very thorough answer. Like, there were some really, like, interesting nuggets of knowledge in there that I’ll take with me. And so I guess my last question would be, like, as, I’m gonna be graduating soon, and so as someone who’s interested in working more with AI, and, like, more technically with AI, and looking into breaking into this space.
316 00:42:30.520 ⇒ 00:42:33.850 Farhaj Shahid: What’s one piece of advice you would give to a binding engineer?
317 00:42:34.900 ⇒ 00:42:45.209 Pranav Narahari: Yeah, that’s a great question. You know, I wasn’t there too long ago. I think you’re in a really good position in a lot of ways.
318 00:42:45.380 ⇒ 00:42:48.040 Pranav Narahari: Because this whole industry is changing, you know?
319 00:42:48.040 ⇒ 00:42:48.490 Farhaj Shahid: That’s weird.
320 00:42:49.010 ⇒ 00:43:01.579 Pranav Narahari: industry in general is changing a lot. There is more of a need for what people are calling AI engineers, forward-deployed engineers, which basically means how
321 00:43:02.100 ⇒ 00:43:07.320 Pranav Narahari: How do you think about problems, and how can you then build solutions quick?
322 00:43:07.520 ⇒ 00:43:11.709 Pranav Narahari: So really, it’s much more problem-solving.
323 00:43:12.890 ⇒ 00:43:14.130 Pranav Narahari: So…
324 00:43:14.540 ⇒ 00:43:26.739 Pranav Narahari: There’s… that shift is super important. Understanding that syntax, writing code in and of itself is not as much of a leg up anymore.
325 00:43:27.250 ⇒ 00:43:27.850 Farhaj Shahid: Right.
326 00:43:27.850 ⇒ 00:43:40.300 Pranav Narahari: is… I think that’s a… that’s a big shift that software engineers should have, in my… in my opinion. Another shift is, really staying up to date with all of the latest,
327 00:43:40.690 ⇒ 00:43:42.069 Pranav Narahari: Tools that are out there.
328 00:43:42.360 ⇒ 00:43:43.190 Pranav Narahari: a common…
329 00:43:43.190 ⇒ 00:43:43.550 Farhaj Shahid: centr.
330 00:43:43.550 ⇒ 00:43:50.990 Pranav Narahari: that you’ll also notice, like, if you’re working at, like, an agency like Brainforge, is that all your clients are going to know
331 00:43:51.110 ⇒ 00:43:54.789 Pranav Narahari: About different products that are out there, and you need to be able to speak to them.
332 00:43:55.950 ⇒ 00:44:00.659 Pranav Narahari: So, something common that just came out in the last month or so is Claude Co-work.
333 00:44:01.460 ⇒ 00:44:02.090 Farhaj Shahid: Yep.
334 00:44:02.270 ⇒ 00:44:05.979 Pranav Narahari: So, a common question that I get asked is, like, why can’t I just do this with clogged cowork?
335 00:44:06.800 ⇒ 00:44:07.300 Farhaj Shahid: Period.
336 00:44:07.300 ⇒ 00:44:09.030 Pranav Narahari: Really important for me to…
337 00:44:09.780 ⇒ 00:44:15.900 Pranav Narahari: first of all, be honest if it can be done in Cloud Co-work, because a lot of things can be done in Cloud Cowork, but then…
338 00:44:16.070 ⇒ 00:44:33.800 Pranav Narahari: being able to accurately describe what cannot be done there, and why what we’re building is still relevant to them. So, basically, yeah, staying up to date on the latest tools, I think, is gonna give you a huge leg up, and then, yeah, just having that, that shift in what it means to be a…
339 00:44:33.990 ⇒ 00:44:36.730 Pranav Narahari: An engineer going forward is another big thing.
340 00:44:37.990 ⇒ 00:44:48.399 Farhaj Shahid: Thank you so much, Pranav. Those were, like, some really amazing answers, and I’m gonna be thinking more about these after our call, but it was a pleasure talking to you. Thank you for taking out the time.
341 00:44:49.450 ⇒ 00:44:55.460 Pranav Narahari: Yeah, it was great talking to you as well. Thanks for the questions, and yeah, Kayla, I think, who you’ve already been talking to, should,
342 00:44:55.810 ⇒ 00:44:56.540 Pranav Narahari: Check out soon.
343 00:44:57.150 ⇒ 00:45:00.060 Farhaj Shahid: Perfect. Thank you so much. Have a great rest of your day. Bye-bye.
344 00:45:00.410 ⇒ 00:45:01.080 Pranav Narahari: Bye.