Meeting Title: Brainforge x Vishnu Interview Date: 2026-03-20 Meeting participants: Vishnu Kakaraparthi, Pranav Narahari
WEBVTT
1 00:05:34.920 ⇒ 00:05:36.130 Pranav Narahari: Hey, Vishnu!
2 00:05:36.130 ⇒ 00:05:37.860 Vishnu Kakaraparthi: Hey, Pranav, how are you doing?
3 00:05:38.400 ⇒ 00:05:39.989 Pranav Narahari: I’m good, I’m good, how are you?
4 00:05:39.990 ⇒ 00:05:42.769 Vishnu Kakaraparthi: I’m good. Thanks for being flexible.
5 00:05:43.220 ⇒ 00:05:48.150 Pranav Narahari: Of course, of course, yeah. Sorry, my schedule’s a little bit packed today, but I’m glad we were able to fit this in.
6 00:05:48.390 ⇒ 00:05:48.930 Vishnu Kakaraparthi: Perfect.
7 00:05:48.930 ⇒ 00:05:50.509 Pranav Narahari: Yeah, how’s your day, man?
8 00:05:50.950 ⇒ 00:05:56.140 Vishnu Kakaraparthi: It’s good, yeah, I mean, I had a few meetings here and there, but then, yeah, it’s,
9 00:05:56.310 ⇒ 00:06:03.909 Vishnu Kakaraparthi: I would say Fridays are more mellow for me than all other days. Mondays are the most hectic in general, like, all that, yeah.
10 00:06:05.100 ⇒ 00:06:06.869 Pranav Narahari: Gotcha, yeah, that makes sense.
11 00:06:07.020 ⇒ 00:06:17.639 Pranav Narahari: Yeah, I guess, yeah, I was just looking, briefly at, like, your LinkedIn. I saw that, so you’re in your PhD, or you’ve completed your PhD?
12 00:06:17.840 ⇒ 00:06:22.700 Vishnu Kakaraparthi: I’m completing my PhD, yeah, that’s… Close to getting done.
13 00:06:23.000 ⇒ 00:06:24.880 Pranav Narahari: Gotcha, gotcha. Tell me a little bit about that.
14 00:06:25.180 ⇒ 00:06:44.189 Vishnu Kakaraparthi: So, yeah, so my research is mostly in vision-based models, especially video-based vision models. So we are developing a wearable, it’s like a camera on your smartwatch. I am very interested in the ventral side of the view. We have patents where it says you can have the camera on any side, but then, yeah.
15 00:06:44.190 ⇒ 00:06:59.240 Vishnu Kakaraparthi: what I am looking at is what people do with their hands and fingers, especially, and how they interact with objects in real world. Within that, I’m looking at activities of daily living for older adults and people with dementia. Even within that subset, I’m looking at medication adherence.
16 00:06:59.560 ⇒ 00:07:12.169 Vishnu Kakaraparthi: I want to make sure that, anyone who’s using this device, like, or, like, for medication adherence specifically, is, able… we are able to track the medication going into their mouth.
17 00:07:12.300 ⇒ 00:07:28.789 Vishnu Kakaraparthi: So the ingestion problem is what is the biggest factor we are trying to solve, because there are smart pillboxes, there are reminders, things like those, but they don’t look at how, they’re… are they putting it in their mouth, did they drop it in between, and things like those, so…
18 00:07:28.970 ⇒ 00:07:31.999 Vishnu Kakaraparthi: Gotcha. Yeah, so from the start of the…
19 00:07:32.000 ⇒ 00:07:32.500 Pranav Narahari: Sounds great, yeah.
20 00:07:32.500 ⇒ 00:07:47.510 Vishnu Kakaraparthi: development and a lot of focus groups, interviews, and things like those, then I’m… I currently developed vision models and stuff. We collected data with, like, 50 participants right now for, like.
21 00:07:47.510 ⇒ 00:07:48.050 Pranav Narahari: Joe.
22 00:07:48.270 ⇒ 00:07:53.740 Vishnu Kakaraparthi: 20 minutes each, so about 1,000 minutes of video data that’s being technically added.
23 00:07:53.740 ⇒ 00:08:00.029 Pranav Narahari: Actually, also, just kind of curious, like, that’s a very unique background, what, makes you interested with Brainforge?
24 00:08:00.750 ⇒ 00:08:14.420 Vishnu Kakaraparthi: So I like working on multiple projects and, different use cases and things like those. I’m very into startup environments and things like those, so that’s what attracted Brainforge to me.
25 00:08:14.730 ⇒ 00:08:23.860 Vishnu Kakaraparthi: And yeah, with LLM stuff, I have done internships with, BrainChip and, Boomi. Boomi was the biggest LLM
26 00:08:24.100 ⇒ 00:08:33.619 Vishnu Kakaraparthi: based research project that has been production-ready and shipped already. Like, it’s probably been shipped, like, a year ago, so it’s been…
27 00:08:33.760 ⇒ 00:08:43.679 Vishnu Kakaraparthi: working for a year or more. Obviously, people probably are still working and improving upon it, so I worked on, like, a documentation generation model with…
28 00:08:43.850 ⇒ 00:08:48.339 Vishnu Kakaraparthi: a lot of LLM routing, semantic understanding, and, like, prompt,
29 00:08:48.340 ⇒ 00:08:49.380 Pranav Narahari: Very interesting, yeah.
30 00:08:49.640 ⇒ 00:08:50.410 Vishnu Kakaraparthi: Yeah, proper.
31 00:08:50.410 ⇒ 00:08:59.459 Pranav Narahari: So, with your current, like, like, setup with, with your PhD, does that allow you to have, like, a full-time role, or are you looking more for a part-time position?
32 00:08:59.700 ⇒ 00:09:07.759 Vishnu Kakaraparthi: I am looking for a full-time role, I am wrapping up my PhD program, so, yeah, yeah, I… yeah, I just have to…
33 00:09:08.290 ⇒ 00:09:18.160 Vishnu Kakaraparthi: do my final reserves checks and things like those, and defend, so I’m also kind of, like, delaying it based on all the job prospects and everything.
34 00:09:18.160 ⇒ 00:09:19.710 Pranav Narahari: Gotcha. That makes sense.
35 00:09:19.710 ⇒ 00:09:20.720 Vishnu Kakaraparthi: So, yeah.
36 00:09:20.720 ⇒ 00:09:29.590 Pranav Narahari: So yeah, what I want to hop into next is just, probably just, like, talking about, like, an LLM or RAG system that you’ve shipped before,
37 00:09:29.770 ⇒ 00:09:38.859 Pranav Narahari: And based on that, I just kind of will do, like, some deep dives onto certain things, and then after that, we’ll just go through, like, a few scenarios to just wrap up the interview.
38 00:09:39.000 ⇒ 00:09:51.140 Pranav Narahari: But yeah, like, when I say a RAG LLM system that you’ve built before, if you were to, like, explain it, like, in 60 seconds or less, like, what would be the system that comes to mind?
39 00:09:51.730 ⇒ 00:10:09.019 Vishnu Kakaraparthi: So, the system that we developed at Boomi, it’s the documentation generation model, and we, had to take in, like, company… so, Boomi generally, especially makes, pipelines, software pipelines, SaaS-based, connections and connectors and stuff, and…
40 00:10:09.020 ⇒ 00:10:27.069 Vishnu Kakaraparthi: when these pipelines are generated, they want documentation for these pipelines, so they are company-specific, they have keywords and terms and things like those, so they upload these documents, which are kind of very industry-specific and things like those. So that’s where I use RAGs before.
41 00:10:27.510 ⇒ 00:10:28.320 Vishnu Kakaraparthi: And…
42 00:10:28.740 ⇒ 00:10:46.089 Vishnu Kakaraparthi: Yeah, so basically, it generates a documentation summary based on the pipeline that they created, and the documentations they, uploaded, and there are 3 levels of summaries. There is a very short summary, there is a long description, and there is a full-page report, kind of.
43 00:10:46.130 ⇒ 00:11:00.959 Vishnu Kakaraparthi: a system, so based on their queries and things like those, there’s an LLM routing, which we both… I mean, we developed an LLM router, we also developed an NLP-based, just Word, based,
44 00:11:01.710 ⇒ 00:11:11.499 Vishnu Kakaraparthi: Classification model, using decision trees and things like those, which does the prompt, and then it goes to the specific prompt based on whatever they want.
45 00:11:12.420 ⇒ 00:11:17.690 Pranav Narahari: Gotcha. And so, can you describe a little bit what your, responsibility was on this project?
46 00:11:17.950 ⇒ 00:11:32.829 Vishnu Kakaraparthi: I would say most of it is my responsibility, but at the end of my work, I had to hand over, and we were doing a lot of, other experimentations, too, with crew and all that stuff we wanted to do, like, because…
47 00:11:32.830 ⇒ 00:11:39.009 Vishnu Kakaraparthi: Right now, even their deployment is called… is, like, they call it an AI studio, and people can do…
48 00:11:39.880 ⇒ 00:11:49.720 Vishnu Kakaraparthi: come up with different, prompts and stuff. They can change things based on whatever they want, like, they can do their own kind of prompts and things like those, so…
49 00:11:49.720 ⇒ 00:11:50.230 Pranav Narahari: Gotcha.
50 00:11:50.230 ⇒ 00:11:55.889 Vishnu Kakaraparthi: Yeah, so that’s where the handoff happened, and that’s kind of where I.
51 00:11:55.890 ⇒ 00:11:56.220 Pranav Narahari: Well, I guess.
52 00:11:56.220 ⇒ 00:11:57.250 Vishnu Kakaraparthi: project, but the.
53 00:11:57.250 ⇒ 00:11:57.640 Pranav Narahari: Yeah.
54 00:11:57.640 ⇒ 00:12:01.610 Vishnu Kakaraparthi: Yeah, the project started with, understanding the pipeline, which
55 00:12:01.930 ⇒ 00:12:07.399 Vishnu Kakaraparthi: which was easy. The pipeline technically exports as a JSON file, so.
56 00:12:07.400 ⇒ 00:12:19.199 Pranav Narahari: Okay, yeah, actually, that’s kind of a new question, is, if you were to explain, like, the architecture of what you hand… before you handed it off, you know, after you handed it off, it may have changed a little bit, but from how you remember the system.
57 00:12:20.170 ⇒ 00:12:21.110 Pranav Narahari: architecture.
58 00:12:21.470 ⇒ 00:12:38.849 Vishnu Kakaraparthi: So, the architecture was all put on AWS, so they have Lambda functions, which is like a call to the… like, they have the pipeline ID, the pipeline ID literally gives… the Lambda function literally returns the JSON of the pipeline.
59 00:12:38.890 ⇒ 00:12:46.239 Vishnu Kakaraparthi: and how it’s built and things like those. You take that JSON, that JSON is used like,
60 00:12:46.750 ⇒ 00:12:48.849 Vishnu Kakaraparthi: The first stage of that
61 00:12:49.190 ⇒ 00:13:02.870 Vishnu Kakaraparthi: that’s, like, the prompt, one of the sections of the prompt, and you have the user prompt, which automatically gets classified into… I wouldn’t say automatically, but that’s a lambda call queue, but then, it gets…
62 00:13:02.980 ⇒ 00:13:14.369 Vishnu Kakaraparthi: classified into, like, short summary, long description, versus the full report and things like those. And based on that, it goes to, like, this is all, on…
63 00:13:14.430 ⇒ 00:13:23.279 Vishnu Kakaraparthi: like, the LLM side, or the routing side is on SageMaker, but then there is the bedrock calls that happen.
64 00:13:23.430 ⇒ 00:13:24.850 Vishnu Kakaraparthi: Which…
65 00:13:25.310 ⇒ 00:13:42.699 Vishnu Kakaraparthi: technically takes the response from the LLM routing, or the NLP-based routing, and then the JSON and the prompt based on what we selected… we decided the prompt classification is.
66 00:13:42.860 ⇒ 00:13:45.129 Vishnu Kakaraparthi: And then it returns the…
67 00:13:45.850 ⇒ 00:13:52.830 Vishnu Kakaraparthi: output in the format, and we push it back as a lambda function, so it technically is an API call.
68 00:13:53.850 ⇒ 00:13:55.060 Pranav Narahari: Okay, gotcha.
69 00:13:55.110 ⇒ 00:13:55.999 Vishnu Kakaraparthi: That’s more of that.
70 00:13:56.650 ⇒ 00:13:57.200 Vishnu Kakaraparthi: rupture.
71 00:13:57.200 ⇒ 00:14:11.430 Pranav Narahari: one thing that I’m interested in is, like, with pushing this project into production, what are some failures or patches that you needed to make after… after the fact? And what was your process? And first,
72 00:14:11.530 ⇒ 00:14:16.520 Pranav Narahari: Receiving or diagnosing the issue, and then finally patching the issue.
73 00:14:17.620 ⇒ 00:14:20.850 Vishnu Kakaraparthi: I…
74 00:14:23.770 ⇒ 00:14:30.629 Vishnu Kakaraparthi: I’m not 100% sure if we had big issues in general. There were more, yeah, I mean, we were.
75 00:14:30.630 ⇒ 00:14:35.249 Pranav Narahari: I guess, hypothetically then, if you didn’t… if no issue’s coming to mind, what would be your process?
76 00:14:37.400 ⇒ 00:14:54.399 Vishnu Kakaraparthi: So, we had some basic software engineering issues, things like those. We had logs and everything, we were looking at logs, why the calls didn’t happen, and things like those. That’s mostly to the extent that we worked with, like, we had issues with, the major
77 00:14:54.580 ⇒ 00:15:03.369 Vishnu Kakaraparthi: hurdles, if I have to say, were probably the prompt sizes were too big when they started doing this project.
78 00:15:03.500 ⇒ 00:15:13.540 Vishnu Kakaraparthi: And they had, like, big, huge examples of, pipelines, and they have, like, obviously different terms and terminologies and things like those. So, I…
79 00:15:13.860 ⇒ 00:15:17.180 Pranav Narahari: And what would you say is, like, the downside of having those large prompts?
80 00:15:17.420 ⇒ 00:15:19.690 Vishnu Kakaraparthi: It… it…
81 00:15:20.420 ⇒ 00:15:37.559 Vishnu Kakaraparthi: In simple words, it’s more like hallucination. It generally tries to put in words, just because it sees those in context, and it doesn’t need to in technical terms, so we broke them down. The whole LLM routing stuff was developed
82 00:15:37.620 ⇒ 00:15:52.160 Vishnu Kakaraparthi: to simplify this problem. So, at the start, they had only one call, they had one prompt that was trying to do all the summarization, all the long description, and everything. So, the introduction of this either
83 00:15:52.170 ⇒ 00:16:02.079 Vishnu Kakaraparthi: call it LLM router, or just classic NLP router. I don’t know what they’re using right now, but we have both systems in place, which, whatever they want to use,
84 00:16:02.450 ⇒ 00:16:17.820 Vishnu Kakaraparthi: the idea was to simplify the prompt, simplify, any hallucination, and things like those, because they didn’t want to retrain any of these models for their data set and things like those, so we didn’t want to… Gotcha.
85 00:16:18.110 ⇒ 00:16:26.960 Vishnu Kakaraparthi: we want to kind of, like, obviously, it’s not a guardrail, but I would probably call it grounding. I would say it’s more grounded to the context.
86 00:16:27.330 ⇒ 00:16:33.870 Vishnu Kakaraparthi: After breaking them into, like, three subproblems than thinking about it as one single problem, yeah.
87 00:16:34.050 ⇒ 00:16:34.760 Pranav Narahari: Okay.
88 00:16:34.900 ⇒ 00:16:54.329 Pranav Narahari: I know you mentioned how, like, you had some logs to diagnose issues. What other types of monitoring did you have in place in case, you know, an API request failed, or just any other type of, monitoring, alerts, or just overall metrics that you had set in place?
89 00:16:54.350 ⇒ 00:16:59.470 Pranav Narahari: what did you do for this AI system, or… you can also pick up… pick any other system that you built.
90 00:17:00.320 ⇒ 00:17:01.600 Vishnu Kakaraparthi: I’m…
91 00:17:03.210 ⇒ 00:17:12.419 Vishnu Kakaraparthi: I think we had CICD pipelines, we did get emails if something broke, but since most of it was under, like, development.
92 00:17:12.680 ⇒ 00:17:23.930 Vishnu Kakaraparthi: framework, we didn’t get a lot of those, because we were actively working on it, and we were testing it actively. But yeah, I have developed a lot of CICD pipelines for
93 00:17:24.060 ⇒ 00:17:26.749 Vishnu Kakaraparthi: Be it cron jobs, be it,
94 00:17:27.160 ⇒ 00:17:41.719 Vishnu Kakaraparthi: like, checks on GitHub for any updates, or when push requests happen, and things like those. The biggest other thing that I worked was for, like, it’s a hackathon, it’s called Mayhem. It’s a fuzzy…
95 00:17:41.790 ⇒ 00:17:58.559 Vishnu Kakaraparthi: a cybersecurity tool, so I integrated their tool with 48, repositories for, security testing of… it’s a fuzzy security testing toolkit, so that’s all CICD stuff. I was…
96 00:17:58.980 ⇒ 00:18:06.689 Vishnu Kakaraparthi: top second contributor for their projects, in the… in whole of United States. Yeah, I think…
97 00:18:06.690 ⇒ 00:18:08.309 Pranav Narahari: And you said this was, like, a hackathon?
98 00:18:08.310 ⇒ 00:18:27.360 Vishnu Kakaraparthi: It… it was… it started as a hackathon, but it’s, like, a 10… like, it’s like… it ran for, like, a one-year whole program where you can keep contributing to it, and yeah, in a one-month-ish span, I think I… I think it was about one to two month span, I did about 48.
99 00:18:27.540 ⇒ 00:18:30.900 Vishnu Kakaraparthi: repositories with, yeah, you can see that on my GitHub.
100 00:18:31.270 ⇒ 00:18:32.899 Pranav Narahari: Oh, interesting, yeah, I’ll take a look at that.
101 00:18:33.330 ⇒ 00:18:48.670 Pranav Narahari: I kind of want to hop into some of the scenarios now as well, and I like doing this during interviews because it’s, real-world things that happen here at Brainforge. And so, with, one of our clients, an e-commerce company.
102 00:18:48.930 ⇒ 00:19:01.659 Pranav Narahari: I just say that to kind of give you a little bit of, like, context, is, we were working with different MCP servers in a chat interface. One of those MCP servers was, was a Shopify MCP server.
103 00:19:01.660 ⇒ 00:19:10.780 Pranav Narahari: And after we pushed to production, we were noticing and getting some feedback from the client saying that the MCP server was generating
104 00:19:10.970 ⇒ 00:19:13.080 Pranav Narahari: demo data.
105 00:19:13.210 ⇒ 00:19:21.090 Pranav Narahari: So it wasn’t extract… it wasn’t pulling the demo data from somewhere else, it wasn’t getting it from Shopify, it was generating it itself.
106 00:19:22.620 ⇒ 00:19:30.100 Pranav Narahari: At that point, once you get that information, what are some other things that you would look into, and then how would you think about patching that issue?
107 00:19:38.620 ⇒ 00:19:42.079 Pranav Narahari: You can think, like, super quick fixes, and then you can think of, like, more…
108 00:19:42.080 ⇒ 00:19:49.760 Vishnu Kakaraparthi: sure if it is using any RAG kind of stuff, like, but if it is doing that, I would probably go and look into, like, re-ranking those
109 00:19:49.960 ⇒ 00:20:07.349 Vishnu Kakaraparthi: rag-based systems and things like those, to see if maybe some of the documents have demo data in those, and it’s probably pulling that stuff from somewhere else, from its, like, context or things like those. Maybe that’s one thing I would probably do.
110 00:20:07.400 ⇒ 00:20:12.140 Vishnu Kakaraparthi: Within the same framework, maybe, like, different chunking, or, like, different…
111 00:20:12.850 ⇒ 00:20:18.509 Vishnu Kakaraparthi: different re-ranking algorithms… I mean, the first one I said was re-ranking and things like those.
112 00:20:19.160 ⇒ 00:20:19.840 Pranav Narahari: Yeah.
113 00:20:20.060 ⇒ 00:20:21.160 Vishnu Kakaraparthi: Yeah, so…
114 00:20:21.160 ⇒ 00:20:24.149 Pranav Narahari: I guess, are there any other… so that’s one hypothesis.
115 00:20:24.150 ⇒ 00:20:24.700 Vishnu Kakaraparthi: Yeah.
116 00:20:24.700 ⇒ 00:20:29.899 Pranav Narahari: Are there any other hypotheses that you have? And for each one of these, like, how would you…
117 00:20:30.010 ⇒ 00:20:31.470 Pranav Narahari: Prove or disprove them.
118 00:20:38.520 ⇒ 00:20:43.539 Vishnu Kakaraparthi: I don’t know, I mean…
119 00:20:43.940 ⇒ 00:20:51.389 Vishnu Kakaraparthi: If we could go tool by tool, we can probably, like, even now, I think the recent cloud release was
120 00:20:51.530 ⇒ 00:20:57.070 Vishnu Kakaraparthi: Where it can, based on all the skills and the tools, it’s gonna…
121 00:20:57.380 ⇒ 00:21:09.239 Vishnu Kakaraparthi: based on the prompts and stuff, it’s gonna re-rank them and re-index the tools and see which tool is most useful. In the same way, probably, I would go and see which tool is generating these
122 00:21:09.370 ⇒ 00:21:16.190 Vishnu Kakaraparthi: this kind of… behavior, is what I would… Probably try doing first.
123 00:21:16.530 ⇒ 00:21:25.080 Vishnu Kakaraparthi: Within… yeah, within the… all the MCP tools, I guess, from that side.
124 00:21:25.540 ⇒ 00:21:26.290 Vishnu Kakaraparthi: Damn.
125 00:21:26.560 ⇒ 00:21:29.000 Vishnu Kakaraparthi: Hmm, what else could create?
126 00:21:31.190 ⇒ 00:21:37.159 Vishnu Kakaraparthi: data. I’m just, you know, should be one of… the tools, I guess, are… Yes, it’s.
127 00:21:37.160 ⇒ 00:21:39.350 Pranav Narahari: Give you a little bit more direction, maybe.
128 00:21:39.350 ⇒ 00:21:39.880 Vishnu Kakaraparthi: Huh.
129 00:21:39.880 ⇒ 00:21:41.140 Pranav Narahari: It’s so…
130 00:21:41.420 ⇒ 00:21:59.619 Pranav Narahari: you can kind of think of this in a vacuum. It’s really just one MCP server getting information from Shopify. And then on top of that, there’s a… it’s a chat interface, so there’s an LLM, that’s also helping you… you make the tool calls to the MCP server.
131 00:22:00.200 ⇒ 00:22:14.269 Pranav Narahari: So there’s no… there’s no knowledge base that is used, like, as, like, a… where it’s, like, doing a similarity search against embeddings, so it’s not pulling the demo data from anywhere, it’s just straight up generating it.
132 00:22:16.000 ⇒ 00:22:17.140 Vishnu Kakaraparthi: I mean, hmm.
133 00:22:17.360 ⇒ 00:22:20.720 Vishnu Kakaraparthi: Is this, like, user data that… or, like…
134 00:22:20.840 ⇒ 00:22:24.519 Vishnu Kakaraparthi: Shopping data, or things like those.
135 00:22:24.520 ⇒ 00:22:33.050 Pranav Narahari: Yeah, so maybe a little bit more context, too, is, what’s happening here is you can assume that the user is asking, what is… what is yesterday’s sales?
136 00:22:33.600 ⇒ 00:22:37.500 Pranav Narahari: And provide it into a report in XYZ format.
137 00:22:38.090 ⇒ 00:22:43.370 Vishnu Kakaraparthi: I would probably think it’s more hallucination there, in that case, where it’s like.
138 00:22:43.700 ⇒ 00:22:49.269 Vishnu Kakaraparthi: Let’s say, for an example, it would say, like, oh, yeah, most of, like, if you just ask…
139 00:22:49.620 ⇒ 00:22:51.269 Vishnu Kakaraparthi: Like, okay, what’s,
140 00:22:52.510 ⇒ 00:22:58.879 Vishnu Kakaraparthi: What discounts, do the store run, or something like that, then it would probably,
141 00:22:59.220 ⇒ 00:23:16.590 Vishnu Kakaraparthi: like, a general LLM would probably say, yeah, most of these stores run 50% discount on a Black Friday, or something like that, because that’s where it’s probably seeing most of the data, so we should probably think about, like, looking into more grounding towards the date
142 00:23:16.700 ⇒ 00:23:20.720 Vishnu Kakaraparthi: data side, I would probably think about it.
143 00:23:21.760 ⇒ 00:23:24.229 Vishnu Kakaraparthi: From that scenario, where it’s not.
144 00:23:24.230 ⇒ 00:23:24.660 Pranav Narahari: Gotcha.
145 00:23:24.690 ⇒ 00:23:27.519 Vishnu Kakaraparthi: Yeah, it’s probably not… it’s just…
146 00:23:27.710 ⇒ 00:23:30.030 Vishnu Kakaraparthi: the… I mean, it’s probably more…
147 00:23:30.030 ⇒ 00:23:31.739 Pranav Narahari: Two follow-up questions.
148 00:23:31.740 ⇒ 00:23:32.070 Vishnu Kakaraparthi: Yeah.
149 00:23:32.070 ⇒ 00:23:34.570 Pranav Narahari: I think that’s great. So…
150 00:23:35.420 ⇒ 00:23:43.739 Pranav Narahari: In terms of parameters that you can use when you talk about hallucination, what is some parameters that you can modulate to,
151 00:23:44.150 ⇒ 00:23:50.819 Pranav Narahari: affect hallucination. If you’re noticing high hallucination, how can you decrease hallucination by changing a parameter?
152 00:23:51.610 ⇒ 00:24:06.780 Vishnu Kakaraparthi: Obviously, you can play around with temperature, then I would also play just around with the prompt itself, like, that’s probably the easiest way to go, especially if the prompts are not huge, and if you’re…
153 00:24:06.890 ⇒ 00:24:17.589 Vishnu Kakaraparthi: If it’s just simple, okay, what are my sales, specifically with the order number or order, like, shop ID or something like that, I’m guessing that would… that’s how it would work.
154 00:24:17.920 ⇒ 00:24:18.250 Pranav Narahari: Yeah.
155 00:24:18.250 ⇒ 00:24:18.820 Vishnu Kakaraparthi: And.
156 00:24:18.820 ⇒ 00:24:24.059 Pranav Narahari: To kind of, like, describe what, like, temperature is, how would you best explain that to somebody?
157 00:24:24.450 ⇒ 00:24:25.840 Vishnu Kakaraparthi: It’s more…
158 00:24:26.120 ⇒ 00:24:40.600 Vishnu Kakaraparthi: it’s more like randomness, is what I would say, like, do you want it to freely think, or do you want it to be restrictive between some context, right? That’s how I would generally explain temperature as… Yeah.
159 00:24:40.600 ⇒ 00:24:41.370 Pranav Narahari: I think that’s great.
160 00:24:41.880 ⇒ 00:24:44.059 Vishnu Kakaraparthi: But someone who doesn’t understand them.
161 00:24:45.100 ⇒ 00:24:48.830 Pranav Narahari: Yeah, I think that’s… I think that’s a great answer.
162 00:24:48.950 ⇒ 00:24:53.069 Pranav Narahari: Yeah, I also wanted to kind of…
163 00:24:53.650 ⇒ 00:25:01.840 Pranav Narahari: ask you about what you would do in terms of grounding as well. Is it a similar answer? I just kind of wanted you to explain that a little bit further.
164 00:25:03.240 ⇒ 00:25:08.490 Vishnu Kakaraparthi: You could ground it based on examples first, like, in the prompt, that would
165 00:25:09.000 ⇒ 00:25:16.430 Vishnu Kakaraparthi: That’s probably the first starting point, and if it’s still doing that, then…
166 00:25:17.190 ⇒ 00:25:27.570 Vishnu Kakaraparthi: then it’s… it’s gonna be interesting, I guess. I would say, just with examples and… Yeah.
167 00:25:28.130 ⇒ 00:25:31.879 Vishnu Kakaraparthi: I would start with examples, probably, and yeah, this is what…
168 00:25:31.880 ⇒ 00:25:32.650 Pranav Narahari: And…
169 00:25:32.650 ⇒ 00:25:35.890 Vishnu Kakaraparthi: It should do when this is how it is, yeah.
170 00:25:36.120 ⇒ 00:25:43.819 Pranav Narahari: Yeah, so I guess, we’ve talked about a few different things, right? We talked about system prompts, we talked about temperature,
171 00:25:44.750 ⇒ 00:25:57.910 Pranav Narahari: and I’ve kind of painted this situation where… let’s say, for example, again, it’s just the user asking for revenue data from the last week or for the last day, and they want it in XYZ format.
172 00:25:58.830 ⇒ 00:26:01.610 Pranav Narahari: What would you say is the right type of…
173 00:26:03.340 ⇒ 00:26:16.529 Pranav Narahari: parameter… you mentioned temperature as being one, and then also system prompt, like, what are… how could you use both of those parameters to reduce hallucinations?
174 00:26:20.170 ⇒ 00:26:27.569 Vishnu Kakaraparthi: If that’s the general prompts… Yeah. Yeah, if that’s the general prompts, then it would be, like,
175 00:26:27.930 ⇒ 00:26:29.769 Vishnu Kakaraparthi: You would… you would give them…
176 00:26:30.360 ⇒ 00:26:33.780 Vishnu Kakaraparthi: I don’t know how Shopify does the…
177 00:26:34.610 ⇒ 00:26:39.289 Vishnu Kakaraparthi: outputs from the MCP server, but let’s assume it gives you, like, a JSON with.
178 00:26:39.290 ⇒ 00:26:40.100 Pranav Narahari: Yeah. Excellent.
179 00:26:40.100 ⇒ 00:26:44.809 Vishnu Kakaraparthi: XYZ, already in it, or… Yeah.
180 00:26:44.840 ⇒ 00:26:51.639 Vishnu Kakaraparthi: a bunch of X1Y1s, which combine to XYZ, or something like that, then you… you would
181 00:26:51.640 ⇒ 00:27:09.650 Vishnu Kakaraparthi: probably grab that as an example from day A or B, something like that, and then you say, okay, here, that would probably be part of the system prompt, here is your example. Like, there are other frameworks, too, like CoStar and things like those, where you talk about each of these,
182 00:27:09.650 ⇒ 00:27:15.960 Vishnu Kakaraparthi: like, put them in an XML format and things like those for it to understand how
183 00:27:16.470 ⇒ 00:27:35.659 Vishnu Kakaraparthi: the response should be based on these criteria and things like those. So I would design that as part of the system prompt, and probably… I would keep temperature to 1, so that I don’t want it to harness in it, but maybe 0.9 based on .995, something above 0.9 is…
184 00:27:35.770 ⇒ 00:27:37.820 Vishnu Kakaraparthi: Make sure it’s the…
185 00:27:39.330 ⇒ 00:27:47.069 Pranav Narahari: Yeah, gotcha. Let’s talk about another… scenario here at Brainforge.
186 00:27:47.370 ⇒ 00:28:01.220 Pranav Narahari: So, we have this for another client, a document that serves as their source of truth. However, it’s a document that is continuously getting changes based on different…
187 00:28:01.380 ⇒ 00:28:08.810 Pranav Narahari: Let’s say, as their business evolves, this document also evolves.
188 00:28:09.150 ⇒ 00:28:24.410 Pranav Narahari: And think of this in its most… we can just make this as simple as, as we can. It’s really just a chatbot that has a RAG system, onto this, document that’s continuously evolving. This document also is being
189 00:28:24.470 ⇒ 00:28:34.210 Pranav Narahari: updated by the client, who is non-technical, and they would prefer to just be making the changes directly into Google Docs. So…
190 00:28:34.910 ⇒ 00:28:36.880 Pranav Narahari: Understanding, kind of.
191 00:28:37.040 ⇒ 00:28:48.410 Pranav Narahari: this… this RAG system that I am… that I just presented to you, how would you… if, like, just give me a quick, like, architecture design for how we would make this chatbot work.
192 00:28:52.970 ⇒ 00:29:07.030 Vishnu Kakaraparthi: In general, you will have something that is ingesting, so you have to do the pull, or however you’re trying to do it, or if the user wants to upload it on their own after it, it’s… that’s the ingest method stuff.
193 00:29:07.080 ⇒ 00:29:21.050 Vishnu Kakaraparthi: then we probably need some kind of an embedding or, like, a vector store, so I would convert that into a vector store. I don’t know how big these documents are, but you can think about different chunking techniques and things like those.
194 00:29:21.150 ⇒ 00:29:32.190 Vishnu Kakaraparthi: And… Yeah, that would probably feed into your, yeah, I mean, you could… if there are…
195 00:29:33.810 ⇒ 00:29:35.809 Vishnu Kakaraparthi: If that’s the only document, then yeah.
196 00:29:36.040 ⇒ 00:29:37.730 Vishnu Kakaraparthi: That would probably.
197 00:29:37.730 ⇒ 00:29:38.320 Pranav Narahari: Gotcha.
198 00:29:38.320 ⇒ 00:29:50.289 Vishnu Kakaraparthi: be it, and that would probably feed into the LLM, or you have, like, K-Search, or you can also have other re-ranking methods based on, like, top K chunks.
199 00:29:50.610 ⇒ 00:29:58.769 Vishnu Kakaraparthi: Then within that, you can also do, like, maximal relevance or things like those, and that goes into the context based on the user query.
200 00:29:59.140 ⇒ 00:30:00.900 Vishnu Kakaraparthi: Then you can use that.
201 00:30:01.360 ⇒ 00:30:02.120 Vishnu Kakaraparthi: Awesome.
202 00:30:02.120 ⇒ 00:30:08.540 Pranav Narahari: With a continuously updated document, what is something that maybe worries you about the system, and something that you…
203 00:30:08.760 ⇒ 00:30:17.780 Pranav Narahari: That you would implement in the initial infrastructure of the project to prevent it from losing accuracy over time?
204 00:30:23.550 ⇒ 00:30:31.789 Vishnu Kakaraparthi: So, you’re talking about keeping it updated, or just making sure that it it knows right now.
205 00:30:31.790 ⇒ 00:30:33.710 Pranav Narahari: Making sure the chat pop is still…
206 00:30:33.990 ⇒ 00:30:43.469 Pranav Narahari: just still, like, accurately giving, the right answers, and, like, in the future doesn’t start drifting in terms of… or I guess a better word is regressing.
207 00:30:47.100 ⇒ 00:30:54.189 Pranav Narahari: I guess you can think about what are some things that could cause regression with a dynamically updated, document.
208 00:30:58.590 ⇒ 00:31:08.209 Vishnu Kakaraparthi: I mean, if you’re building everything, if you’re rebuilding all the embeddings from… the… new document again, I don’t…
209 00:31:08.350 ⇒ 00:31:12.220 Vishnu Kakaraparthi: I’m not 100% sure if it… It would
210 00:31:12.630 ⇒ 00:31:16.629 Vishnu Kakaraparthi: Deviate much, but then you can always have,
211 00:31:16.870 ⇒ 00:31:22.529 Vishnu Kakaraparthi: Checks based on how my old system is working versus the new system is working, like…
212 00:31:22.690 ⇒ 00:31:30.569 Vishnu Kakaraparthi: have some… you can even do human in loop, or another LLM as a judge, or something like that, to see how much it is drifting, and…
213 00:31:30.790 ⇒ 00:31:36.279 Vishnu Kakaraparthi: But then… we’ll probably come to why, I guess?
214 00:31:36.650 ⇒ 00:31:39.159 Vishnu Kakaraparthi: The why is the bigger question, where…
215 00:31:40.450 ⇒ 00:31:48.220 Pranav Narahari: Yeah, I guess one… one concern could be that people are adding duplicate information or conflicting information, which could then…
216 00:31:48.360 ⇒ 00:31:54.060 Pranav Narahari: you know, cause for regression in the output.
217 00:31:54.710 ⇒ 00:31:55.430 Vishnu Kakaraparthi: Yeah.
218 00:31:55.430 ⇒ 00:32:13.349 Pranav Narahari: what is a system that you can put in place that prevents… and we can make it… we can change it, I guess, from allowing the client to automatically up the document, or update the document, to then changing the system so that we don’t see these regressions due to, you know, client behavior of adding…
219 00:32:13.620 ⇒ 00:32:20.199 Pranav Narahari: certain information. Like, what’s the system that you would put into place to, like, prevent that type of regression and accuracy?
220 00:32:26.600 ⇒ 00:32:28.070 Vishnu Kakaraparthi: Question.
221 00:32:46.410 ⇒ 00:32:53.820 Vishnu Kakaraparthi: 100% sure here, but then, yeah, I was thinking just comparing it with the previous system, and then coming up with…
222 00:32:53.980 ⇒ 00:33:00.180 Vishnu Kakaraparthi: Why, but then you can also look into what parameters have changed, but this is probably…
223 00:33:00.810 ⇒ 00:33:03.949 Vishnu Kakaraparthi: I don’t know if you’ll have another agent that is looking
224 00:33:04.060 ⇒ 00:33:17.800 Vishnu Kakaraparthi: At all the previous documents versus the new documents, and seeing, or, like, answering why this anomaly is happening, or, like, is this document way too far from the previous documents, or…
225 00:33:18.740 ⇒ 00:33:19.490 Pranav Narahari: I guess…
226 00:33:19.490 ⇒ 00:33:21.229 Vishnu Kakaraparthi: Just a reference check.
227 00:33:22.380 ⇒ 00:33:26.879 Pranav Narahari: how I would, to give, like, a little bit more guidance here is,
228 00:33:27.550 ⇒ 00:33:36.100 Pranav Narahari: Say that a change that would normally just be added to the document without any… checks.
229 00:33:36.270 ⇒ 00:33:42.429 Pranav Narahari: Along the way, that’s something that could cause the accuracy to regress.
230 00:33:42.510 ⇒ 00:33:56.850 Pranav Narahari: What is the system that you would put in place so that maybe the client doesn’t automatically update the document, but it goes through some process that ensures that whatever information is added to the document is only going to increase accuracy, and is not going to cause regression?
231 00:34:03.690 ⇒ 00:34:05.160 Vishnu Kakaraparthi: I mean, low.
232 00:34:06.330 ⇒ 00:34:13.679 Vishnu Kakaraparthi: That would probably be, like, an LLM as a judge, I guess. Like, you’re using the same LLM, probably, and deciding if…
233 00:34:14.469 ⇒ 00:34:18.699 Vishnu Kakaraparthi: It’s more or less like A-B testing two different
234 00:34:19.320 ⇒ 00:34:25.879 Vishnu Kakaraparthi: Input stuff, and if… if it is regressing way too much, then that’s probably the…
235 00:34:26.790 ⇒ 00:34:39.819 Vishnu Kakaraparthi: document, then you probably raise issues with the document, things like those. If it is improving your… whatever key parameters are for judging accuracy, then you should probably be good.
236 00:34:39.960 ⇒ 00:34:43.679 Vishnu Kakaraparthi: That’s what I would… That would probably be the easiest.
237 00:34:43.949 ⇒ 00:34:45.230 Vishnu Kakaraparthi: way, I would say.
238 00:34:45.230 ⇒ 00:34:48.150 Pranav Narahari: Yeah, so, I know we’re wrapping…
239 00:34:48.400 ⇒ 00:34:54.830 Pranav Narahari: I know we’re wrapping up here, and, like, I want to give you some questions, but, yeah, I’m happy to stay on for a couple more minutes if, if you’d like.
240 00:34:54.830 ⇒ 00:34:56.859 Vishnu Kakaraparthi: We do. Yeah, I’m fine, yes.
241 00:34:56.860 ⇒ 00:35:04.780 Pranav Narahari: Perfect, yeah, I mean, I just kind of wanted to wrap up, like, that scenario. So one way you could go about doing it is by…
242 00:35:06.920 ⇒ 00:35:22.790 Pranav Narahari: taking, like, let’s say… I mentioned how it’s a very large document, right? So, having some type of structured document is what’s going to be a great value add here, because then you can assess where would this information be located within that document.
243 00:35:22.790 ⇒ 00:35:31.570 Pranav Narahari: And then assess… and then compare the information that’s currently there with the information that you’re trying to add, and then, like you said, like, we can do some type of…
244 00:35:31.570 ⇒ 00:35:36.640 Vishnu Kakaraparthi: Let’s do embeddings and see if the embeddings are far too different, or something like that.
245 00:35:37.730 ⇒ 00:35:39.330 Pranav Narahari: Yeah, you can do something like that as well.
246 00:35:39.330 ⇒ 00:35:43.079 Vishnu Kakaraparthi: You can do a cosine, or something like that.
247 00:35:43.080 ⇒ 00:35:43.890 Pranav Narahari: Yeah.
248 00:35:44.010 ⇒ 00:35:55.490 Pranav Narahari: And so, yeah, duplication’s one thing that you want to check for, because you don’t want to make this document excessively large for no reason, because with a chatbot, you don’t want your execution time as well to be…
249 00:35:55.980 ⇒ 00:35:57.140 Vishnu Kakaraparthi: Yeah.
250 00:35:57.560 ⇒ 00:35:58.640 Vishnu Kakaraparthi: tokens, and…
251 00:35:59.590 ⇒ 00:36:01.319 Pranav Narahari: Yup. So…
252 00:36:01.460 ⇒ 00:36:10.229 Pranav Narahari: Yeah, I know we’re a little bit over time, let me know if you have any questions, though, I’m happy to stay on for a couple more minutes, or
253 00:36:10.370 ⇒ 00:36:11.090 Pranav Narahari: Yeah.
254 00:36:11.550 ⇒ 00:36:21.929 Vishnu Kakaraparthi: I don’t think I have any, I think I had a good talk with Sam, too, so yeah, he mostly covered all my questions before, so I’m good.
255 00:36:22.250 ⇒ 00:36:32.540 Vishnu Kakaraparthi: That’s awesome. If you need anything, let me know. I can send you my GitHub link if you don’t have access to it and have all the CI-CD contributions that I did, but it wasn’t complicated.
256 00:36:32.540 ⇒ 00:36:35.479 Pranav Narahari: Sure, yeah, if you want to send that to me in an email, that would be great.
257 00:36:36.630 ⇒ 00:36:37.350 Vishnu Kakaraparthi: Sure.
258 00:36:37.870 ⇒ 00:36:38.270 Pranav Narahari: Cool.
259 00:36:38.270 ⇒ 00:36:38.700 Vishnu Kakaraparthi: I’ll do that.
260 00:36:38.700 ⇒ 00:36:40.099 Pranav Narahari: Alright, this is great.
261 00:36:40.540 ⇒ 00:36:41.150 Vishnu Kakaraparthi: Thank you.
262 00:36:41.480 ⇒ 00:36:43.220 Vishnu Kakaraparthi: Have a great weekend. Bye!
263 00:36:43.590 ⇒ 00:36:44.789 Pranav Narahari: You too, have a good one.
264 00:36:45.000 ⇒ 00:36:45.560 Vishnu Kakaraparthi: Okay.