Meeting Title: Brainforge Interview Follow-Up Date: 2026-04-08 Meeting participants: Vimarsh Patel, Pranav
WEBVTT
1 00:06:11.520 ⇒ 00:06:12.220 Vimarsh Patel: Hello.
2 00:06:13.230 ⇒ 00:06:14.380 Pranav: Hey, how’s it going?
3 00:06:15.210 ⇒ 00:06:17.079 Vimarsh Patel: Hey, hi Pranav, I’m good, how are you?
4 00:06:17.080 ⇒ 00:06:20.609 Pranav: I’m good, I’m good. Sorry, I’m hopping straight from another meeting.
5 00:06:20.610 ⇒ 00:06:21.570 Vimarsh Patel: No worries at all.
6 00:06:22.020 ⇒ 00:06:22.810 Vimarsh Patel: Eastern.
7 00:06:23.050 ⇒ 00:06:24.969 Pranav: 2 minutes, if you don’t mind, I’m just gonna…
8 00:06:24.970 ⇒ 00:06:26.770 Vimarsh Patel: Yeah, please, no, no, please take your time.
9 00:06:27.280 ⇒ 00:06:28.370 Pranav: Perfect, thank you.
10 00:07:06.230 ⇒ 00:07:11.400 Pranav: Okay, cool. Yeah, thank you so much for being flexible. How’s your day been so far?
11 00:07:12.000 ⇒ 00:07:13.889 Vimarsh Patel: Day’s been good so far, yeah.
12 00:07:14.290 ⇒ 00:07:14.700 Pranav: Cool.
13 00:07:14.700 ⇒ 00:07:18.300 Vimarsh Patel: That’s hectic. How about you? I’m assuming it’s pretty hectic for you.
14 00:07:18.300 ⇒ 00:07:23.919 Pranav: It’s been pretty busy today, yeah, yeah. But, you know, that can be good sometimes, too.
15 00:07:24.540 ⇒ 00:07:29.970 Pranav: So, yeah, how I like to usually just, like, structure interviews at this stage is,
16 00:07:30.190 ⇒ 00:07:37.329 Pranav: I like to get to know, you know, a little bit about your background, just kind of briefly. I know you probably already had a conversation with Sam, right?
17 00:07:37.330 ⇒ 00:07:38.950 Vimarsh Patel: Yes, yes, I did, yeah.
18 00:07:38.950 ⇒ 00:07:48.080 Pranav: So you had a conversation with Sam, he kind of, wrote a lot of notes that, helped me out as well, but I might go over some things as well, just to kind of hear it.
19 00:07:48.080 ⇒ 00:07:48.580 Vimarsh Patel: Yeah, true.
20 00:07:50.390 ⇒ 00:08:02.020 Pranav: But then also, after that, I kind of get into just, one or two of, like, specific projects that you worked on, then we kind of go into detail about that. And then at the end…
21 00:08:02.120 ⇒ 00:08:09.790 Pranav: going into some specific cases that I ran into at Brainforge, just to get a sense of how you think through problems.
22 00:08:10.610 ⇒ 00:08:11.230 Vimarsh Patel: Beautiful.
23 00:08:11.350 ⇒ 00:08:23.629 Pranav: Yeah, let’s, if that sounds good, let’s just, start. I would like to get, you know, maybe just, like, a 60-second overview of just, like, how your career has, like, progressed up until this point.
24 00:08:23.630 ⇒ 00:08:24.500 Vimarsh Patel: sharpened.
25 00:08:24.500 ⇒ 00:08:25.510 Pranav: Yeah, that’d be great.
26 00:08:25.930 ⇒ 00:08:40.390 Vimarsh Patel: Sure, yeah, perfect. So, like, I did my undergrad from back in Chennai, India. I moved here for my master’s back in 2022. Between that, I had… I was working with one of my professors. She started her own company, so…
27 00:08:40.390 ⇒ 00:08:47.599 Vimarsh Patel: I was helping her out initially. Even while I was studying, I was, like, we were building a SaaS tool for a school back in India.
28 00:08:47.600 ⇒ 00:09:03.190 Vimarsh Patel: Then, after that, I have been working on the side while I was studying as well, just to gain experience, not for… I was not earning anything out of it, just working, building SaaS tools for small, medium enterprises, like a driving school, or maybe, like, a restaurant, something like that.
29 00:09:03.310 ⇒ 00:09:14.540 Vimarsh Patel: Then I was, once I was about to end my master’s, I was, working with an Italian food distributor out here in Jersey. I was helping them build out their website, and…
30 00:09:14.580 ⇒ 00:09:33.980 Vimarsh Patel: like an e-commerce for front-end, for ordering and something like that. After that, I worked with Dow Jones for almost a year. It was a contract role. And currently, I’m working at White Collar Technologies as an AI engineer. I am helping them, automate a couple of workflows in their internal process, mainly for HR purposes.
31 00:09:34.040 ⇒ 00:09:51.550 Vimarsh Patel: And, like, I have tried to, like, build myself as an AI software engineer. Mainly, I’m, like, much more stronger on the backend side, and recently I’ve also started working on agents, rack pipelines, and automations, like, using NATN or Microsoft Power Automate as well.
32 00:09:52.190 ⇒ 00:09:52.759 Pranav: Oh, that’s great.
33 00:09:52.960 ⇒ 00:09:53.670 Vimarsh Patel: Yeah.
34 00:09:53.670 ⇒ 00:10:10.430 Pranav: I think that’s a good segue to kind of talk about maybe some of the automations and agents that you’ve built at… sounds like with your most recent job being more of, like, in that AI engineering role, you probably had a lot more exposure to that. So yeah, maybe let’s focus on the…
35 00:10:10.640 ⇒ 00:10:18.899 Pranav: Let’s focus on maybe whatever you would say is, like, the project that you had the most involvement in, or maybe the most technically complex project.
36 00:10:18.900 ⇒ 00:10:31.490 Pranav: And then, kind of just give me a very brief, just, like, overview of, like, what the product was, who the customers were, just so I can get an understanding of, like, okay, what is this product? And then we can go into, like, more of the technical stuff after.
37 00:10:31.940 ⇒ 00:10:46.429 Vimarsh Patel: So, I am, like, last month I was working on building a, like, a voice ordering agent for restaurants, like, just to understand voice agents also, and, like, put it out there. So that was, like, I took care of, like, mostly the backend part.
38 00:10:46.430 ⇒ 00:10:53.490 Vimarsh Patel: like, I used… like, I did some research initially of what can be used, to build the voice agent, like, like.
39 00:10:53.810 ⇒ 00:11:08.660 Vimarsh Patel: text-to-speech, and then from speech to text, based on the LLM part as well, and then, what models can be used, like a deep cram and LLMs in as well. But I came across WAPI, and they had, like, a one-stop solution. You just create your own
40 00:11:08.800 ⇒ 00:11:21.900 Vimarsh Patel: like, the template of your agent, and then you plug in with the tools that you want it to call, so you don’t have to build the whole voice agent from scratch. And, like, I think that was the quickest way for me to build an MVP, so I went along with that.
41 00:11:21.900 ⇒ 00:11:37.589 Vimarsh Patel: Apart from that, on the… on the backend side, it was mostly Python and FastAPI, initially, like, while I was building out the tool. For database, I… for, like, the MPP, I used SQLite, and for vector database, I used, like, the ChromaDB.
42 00:11:37.690 ⇒ 00:11:40.020 Vimarsh Patel: And… Cool.
43 00:11:40.020 ⇒ 00:11:48.839 Pranav: Just to make sure I understand, like, what the project was itself, so it was this voice agent that you specifically said, like.
44 00:11:49.220 ⇒ 00:11:54.589 Pranav: I may have missed it, you said it, what type of, like, questions are being asked to the voice agent, and then what type.
45 00:11:54.590 ⇒ 00:11:55.300 Vimarsh Patel: Oh, yes.
46 00:11:55.300 ⇒ 00:11:55.970 Pranav: They’re being…
47 00:11:55.970 ⇒ 00:12:10.280 Vimarsh Patel: Sorry, yeah, sorry about that. So, it’s basically a voice ordering agent for a restaurant, so basically, it will take, like, you are a customer, and you call the restaurant to place order for your meal, or, like, whatever you want. So basically, it will…
48 00:12:10.290 ⇒ 00:12:24.679 Vimarsh Patel: like, if you are an existing customer, based on your number, it will retrieve your, like, your details, your name, and your past orders, if you have had. If it is… if you are not an existing customer, it might ask you for your name to proceed further. So something.
49 00:12:24.680 ⇒ 00:12:25.130 Pranav: Gotcha.
50 00:12:25.130 ⇒ 00:12:27.760 Vimarsh Patel: That was the initial starting point, and then…
51 00:12:28.170 ⇒ 00:12:47.039 Vimarsh Patel: like, I ingested the menu, like, extracted the menu items, created embeddings, and stored it in the vector database, and I also made, like, a row, like, for each menu item in the SQL as well, so if the restaurant wants to edit their menu items, maybe change the price, or put in an offer, or something like that, so they can do that as well.
52 00:12:47.630 ⇒ 00:12:54.770 Pranav: Yeah, that’s great. So, Vappy’s part of this was integrating with the phone number.
53 00:12:55.210 ⇒ 00:13:10.300 Vimarsh Patel: Yes, so WAPI has, like, an option where you can choose your provider, like, if you have your own phone number, or if you want to connect it with, like, a Twillo, or, like, if you… like, WAPI also gives you test numbers as well, if you want to take their own, like, on the free account, you can, like, it gives you 3 or 4
54 00:13:10.360 ⇒ 00:13:15.850 Vimarsh Patel: Or, like, 5 test numbers that you can take, depending on your, like, your region, and you can test it out.
55 00:13:16.480 ⇒ 00:13:18.170 Pranav: Okay. Yeah, very interesting.
56 00:13:19.870 ⇒ 00:13:26.399 Pranav: And so… this is something that you worked at just, like, at your previous role, or is this, like, a previous experience to.
57 00:13:26.400 ⇒ 00:13:29.720 Vimarsh Patel: kind of a side project that I’m doing, like, something to… Side project, yeah.
58 00:13:29.720 ⇒ 00:13:32.320 Pranav: It sounds like you had a lot of ownership with that then, which is really cool.
59 00:13:32.320 ⇒ 00:13:43.660 Vimarsh Patel: Yeah, it’s just, like, it is me and one other person, like, it’s one of my friends back in Canada, the two of us are doing it, like, he’s mainly on the front-end side, and he has a couple of restaurants that we are targeting to give our tool to.
60 00:13:43.690 ⇒ 00:13:56.210 Vimarsh Patel: Yeah. So, like, he made the websites for them already, so now we want to give our ordering agent to them as well, so I’m working on the back end. He’s more focused with the front end and getting the customers. Like, getting the restaurants to onboard.
61 00:13:56.550 ⇒ 00:13:57.330 Pranav: Gotcha.
62 00:13:57.490 ⇒ 00:14:05.060 Pranav: Okay, so that’s, that’s really interesting. I think you kind of already walked me through a little bit of, like, the technical implementation as well.
63 00:14:05.890 ⇒ 00:14:23.969 Pranav: One question that I have is, this is not a fixed data set, right? This knowledge base is ever-changing, as you mentioned, as it changes with, maybe new menu items being added, or new deals being, present in the… for the week, or something like that.
64 00:14:24.340 ⇒ 00:14:25.780 Pranav: Have you noticed
65 00:14:26.730 ⇒ 00:14:40.700 Pranav: any… or actually, how are you tracking regressions? Or if you’re not, how would you potentially track regressions from happening, when you’re changing the baseline data, every so often?
66 00:14:41.360 ⇒ 00:14:48.360 Vimarsh Patel: So, like, regression as a new mean, like, if I change, like, the price of Menwhite, and then how do the embeddings also get affected, or something like that?
67 00:14:48.720 ⇒ 00:14:56.760 Pranav: Yeah, so, well, I’m assuming, like, in the beginning, you do a bunch of testing to make sure that it’s properly retrieving the information.
68 00:14:56.760 ⇒ 00:15:09.239 Pranav: But if you’re constantly updating the SQL database with new prices, new menu items, how can you ensure that going forward that it’s still going to be as accurate, if not more accurate?
69 00:15:09.560 ⇒ 00:15:22.810 Vimarsh Patel: Yeah, so I’ll come to two things. One thing was, like, when I was initially ingesting the menu, I was getting issues with not all the items were being extracted and converted into embeddings properly. It was missing out some of the menu items.
70 00:15:22.810 ⇒ 00:15:34.889 Vimarsh Patel: That was one. The other part, like you mentioned, I think we are still working on that, like, the updating part, like, if you change the menu and, like, so that is what I am currently trying to implement. It is, I will be honest, like, it’s not there 100%. I am trying to…
71 00:15:34.890 ⇒ 00:15:43.010 Vimarsh Patel: work on, like, what happens when you change the price, and, like, that is what I’m focusing on currently, because I fixed the first problem. The first problem was, like, it was not,
72 00:15:43.010 ⇒ 00:15:58.299 Vimarsh Patel: ingesting all the menu items, like, because my chunks were not being created properly, it used to break between, like, from every section, like, if it’s an appetizer or if it’s a dessert, the last 2 or 3 menu items were being skipped, and that was because of my chunk size.
73 00:15:58.300 ⇒ 00:16:08.319 Vimarsh Patel: So that was the problem I encountered first, and now I’m focusing on this problem, like you mentioned. So I don’t think I have an outright solution for it, but if I were to think about it, then…
74 00:16:08.360 ⇒ 00:16:23.210 Vimarsh Patel: I think, first of all, would be to, find the embeddings related to that, menu item, and just update that, so none of the other embeddings are affected. That would be the most cleanest path, but I’m not sure if it is optimal. Like, so…
75 00:16:23.550 ⇒ 00:16:34.030 Vimarsh Patel: I’d also have to keep a check, like, because if, like, in the embeddings, it is, like, based on a similarity, so if some… if one… if I’m changing the SQL, items, like, price, then
76 00:16:34.040 ⇒ 00:16:52.749 Vimarsh Patel: based on this embedding similarity, it could be effectively attached to some other item as well, so I will have to keep that in mind as well while the embeddings are being updated. And obviously, updating the whole embedding is going to be a very gruesome task, because then I’ll be calling the OpenAI embeddings, or whatever I’m using, like, a lot, and, like.
77 00:16:52.780 ⇒ 00:16:59.559 Vimarsh Patel: That would increase my token cost as well. So, I think these are, like, the two things that I would have to, like.
78 00:16:59.710 ⇒ 00:17:01.880 Vimarsh Patel: workup one, as you…
79 00:17:02.370 ⇒ 00:17:13.280 Pranav: Gotcha. And so, you mentioned how this… you’re embedding the menu. Is there any other… I’m trying to understand the scale of the data that you’re creating embeddings for?
80 00:17:13.579 ⇒ 00:17:21.149 Pranav: Did you ever calculate, or is there a rough estimate in terms of, like, the number of tokens that all of that data is?
81 00:17:22.619 ⇒ 00:17:23.979 Vimarsh Patel: So I think…
82 00:17:24.179 ⇒ 00:17:35.009 Vimarsh Patel: if we are considering, like, I… I had a couple of test menus for an Indian restaurant, so I’m a, like, an Indian restaurant has a lot of items, as compared to a, like, a normal fast food chain or something like that.
83 00:17:35.009 ⇒ 00:17:46.889 Vimarsh Patel: So, I think based on the kind of restaurant, the menu, like, the menu will be, like, short or long, I think, but, like, since I was testing it out, and but, like, for a moment, I know, like.
84 00:17:46.889 ⇒ 00:17:59.409 Vimarsh Patel: I had put in 2, so… but I don’t know the exact token length.
85 00:17:59.410 ⇒ 00:18:02.119 Pranav: What I’m curious more so for is,
86 00:18:03.250 ⇒ 00:18:16.369 Pranav: how many tokens, like, how many characters… and that can just be, like, a product of the number of characters that are in, like, one of these menus. So I guess, for example, you may know roughly how big one of the menus is, right?
87 00:18:16.370 ⇒ 00:18:17.949 Vimarsh Patel: Like, around 8 to 10 pages of PDF.
88 00:18:17.950 ⇒ 00:18:20.179 Pranav: I’m just gonna do, like, a rough calculation right here.
89 00:18:20.180 ⇒ 00:18:20.840 Vimarsh Patel: Yes.
90 00:18:21.250 ⇒ 00:18:26.029 Vimarsh Patel: like, around… if it is, like, around 8 pages, I’d assume, like, I was the one…
91 00:18:26.030 ⇒ 00:18:32.879 Pranav: No, I’ll just use this up. I won’t make you do, like, the mental math. So, 8 pages, let’s just say it’s a document that’s 8 pages.
92 00:18:33.140 ⇒ 00:18:35.920 Pranav: How many…
93 00:18:39.640 ⇒ 00:18:41.859 Pranav: Just wanna see how many tokens that is.
94 00:18:41.970 ⇒ 00:18:46.740 Pranav: Yeah, so an 8-page document is around, like, 5,000 tokens,
95 00:18:47.360 ⇒ 00:18:52.610 Pranav: So… right now, the context window, right, for these models…
96 00:18:52.860 ⇒ 00:18:57.420 Pranav: Can be more than… there’s much more than 5,000 tokens.
97 00:18:57.420 ⇒ 00:18:57.990 Vimarsh Patel: Yes.
98 00:18:58.180 ⇒ 00:19:05.939 Pranav: Is there other stuff that you’re also embedding, that may require you to have more of, like, a RAG approach versus just…
99 00:19:06.050 ⇒ 00:19:16.099 Pranav: if it’s only an 8-page document, you might as well just throw that into the LLM on top of your prompt as well, right? Was there a reason why you chose RAG versus…
100 00:19:16.320 ⇒ 00:19:18.920 Pranav: just going… just throwing it straight to the LLM.
101 00:19:19.600 ⇒ 00:19:29.860 Vimarsh Patel: So, I think one approach would be, like, if a customer calls up the restaurant, they might not know the exact name of what they want. They might even just say, like, I want something with, like, chicken or something with mutton.
102 00:19:29.980 ⇒ 00:19:43.180 Vimarsh Patel: it is not necessary that they will come up with, like, I want… like, I want this specific dish. Like, someone who might… who’s might call in the restaurant for the first time, they might not exactly know what the restaurant offers. They just know that it’s an Indian restaurant, so they might just
103 00:19:43.200 ⇒ 00:19:54.800 Vimarsh Patel: come up and ask, like, I want something that is spicy and something that is vegetarian, or something like that. So I think it should, based on that, and like, in the menu items, like, they put in the description of the
104 00:19:55.180 ⇒ 00:20:12.739 Vimarsh Patel: of what the dish is about, like, and so I think, because based on that approach, I chose embedding, so that if a customer comes and talks about it, then based on the similarity, the voice agent can suggest something based on the customer’s preferences or what they want. That is why I think I chose the embedding approach.
105 00:20:13.440 ⇒ 00:20:16.590 Pranav: Okay, interesting. Yeah, I wonder,
106 00:20:17.030 ⇒ 00:20:22.669 Pranav: It would be interesting to me, like, if you did, like, some testing to see, like, the difference, between just, like.
107 00:20:23.010 ⇒ 00:20:40.909 Pranav: because the… what you’re kind of talking about is more like semantic search, right? Not just on a keyword. If a customer wants something with chicken, it should be able to assess, like… I guess for that, you could probably do keyword, but if you’re saying something spicy, then…
108 00:20:41.000 ⇒ 00:20:49.600 Pranav: A semantic search will be able to assess, okay, these are the ingredients, these are going to be ingredients that are going to increase spice, and so based on that, it’s going to…
109 00:20:50.100 ⇒ 00:20:54.060 Pranav: recommends certain… Dishes?
110 00:20:54.290 ⇒ 00:20:58.169 Pranav: So, that’s all something that you can probably just ask ChatGPT.
111 00:20:58.170 ⇒ 00:20:58.890 Vimarsh Patel: Yeah.
112 00:20:58.890 ⇒ 00:21:06.050 Pranav: So, it’s a… it’s something interesting to look into, and I think this is, like, a… I bring this up because these are problems that we see every day.
113 00:21:06.050 ⇒ 00:21:06.460 Vimarsh Patel: Yeah.
114 00:21:06.460 ⇒ 00:21:09.480 Pranav: Realizing the solution actually gets easier and easier.
115 00:21:09.480 ⇒ 00:21:10.179 Vimarsh Patel: No, absolutely.
116 00:21:11.020 ⇒ 00:21:11.670 Pranav: Amanda?
117 00:21:11.950 ⇒ 00:21:20.720 Vimarsh Patel: At this level, you need to look at different approaches, and not just one. Like, for my approach, like, from my task, it’s like, I’m just working out, I’m just experimenting, but when you are looking at a comp.
118 00:21:20.840 ⇒ 00:21:25.770 Vimarsh Patel: situation like yours, we have to look at all the possibilities, because it is the client who’s going to be paying for it, so…
119 00:21:26.290 ⇒ 00:21:41.370 Pranav: Yeah, yeah. But what you’re working on is really interesting. It’s actually, very kind of similar in problems that are gonna arise in terms of, like, one of the clients that we’re working with. And so…
120 00:21:41.820 ⇒ 00:21:44.170 Pranav: Yeah, I’m happy to kind of just, like…
121 00:21:44.830 ⇒ 00:21:49.800 Pranav: Actually, maybe before I go into that, like, we can talk about this project a little bit more,
122 00:21:49.970 ⇒ 00:21:56.809 Pranav: What other types of, what, like, types of alerting,
123 00:21:58.640 ⇒ 00:22:08.739 Pranav: frameworks did you set up, or just alerts in general, just to… because if this is already in production for a restaurant, right, like, you should have certain alert mechanisms in place to…
124 00:22:08.740 ⇒ 00:22:21.709 Pranav: to know, like, okay, if things are down, what needs to be patched, if, you know, things are throttling, things of that nature. Were there anything that you put into place, or is that… is that kind of still just, being worked on?
125 00:22:22.540 ⇒ 00:22:35.320 Vimarsh Patel: So, one thing was, like, adding a human, like, if the agent is, at some point, gets stuck, or, like, anything, then we obviously added a human fallback. That was one thing we… I realized that needs to happen.
126 00:22:35.320 ⇒ 00:22:51.569 Vimarsh Patel: So, we need to have a mechanism that will lead us, like, that will make sure that the customer doesn’t go unsatisfied or gets annoyed with what the agent is saying, because, like, the agent can hallucinate, or, like, it can just keep saying that I am not able to find this, I’m not able to find this, and that can annoy the person. I think one learning was.
127 00:22:51.570 ⇒ 00:22:57.929 Vimarsh Patel: Obviously, like, a human fallback is definitely important in such… like, in almost every situation that… that involves agents.
128 00:22:57.930 ⇒ 00:22:58.400 Pranav: North.
129 00:22:58.400 ⇒ 00:23:16.489 Vimarsh Patel: the other part, like, like you mentioned, alerting and that, I think I… I’m… we are there. I mean, we are not there 100%, but we are getting there of alerting and involving, but I think the one thing is, like, logging everything, making sure, like, what, whatever tool call the agent is doing,
130 00:23:16.740 ⇒ 00:23:31.949 Vimarsh Patel: what response it gave, and everything needs to be logged so we can trace it back if we have… when the error comes up. So I think that that was one of the learnings that I also implemented, like, making sure that at every stage, whatever, input the…
131 00:23:32.510 ⇒ 00:23:45.699 Vimarsh Patel: is going to the LLM, and, like, what it gave back also needs to be added, and at what stage, like, what, like, the reasoning and whatever it is. Everything needs to be logged for us to understand what actually happened in that request.
132 00:23:45.730 ⇒ 00:24:04.479 Vimarsh Patel: So, that is what I have been… like, it has happened twice that I am focused, like, in different scenarios, that logging is very important, especially when it is considering when you’re choosing agents and LLMs. And, like, or even in general, like, even if you’re building a basic software, like, you need to know your failures need to be loud and not silent, like, you need to know.
133 00:24:04.480 ⇒ 00:24:15.259 Pranav: Yeah, that’s a good point. I’m also wondering, are you, so logging for code and understanding, like, where, like you said, like, where things are failing is really important.
134 00:24:15.630 ⇒ 00:24:22.519 Pranav: Also, in this case, doing analysis on transcripts, and how… doing…
135 00:24:23.040 ⇒ 00:24:28.109 Pranav: Having some type of reporting system in place, so…
136 00:24:28.280 ⇒ 00:24:34.839 Pranav: These are all the transcripts of all the calls that we had for the past week, past month, whatever, and then generating a…
137 00:24:35.260 ⇒ 00:24:37.120 Pranav: A categorization.
138 00:24:37.240 ⇒ 00:24:38.870 Pranav: of…
139 00:24:39.050 ⇒ 00:24:45.119 Pranav: calls that went well, went bad, or just a categorization in terms of, like, what was being asked. So you can…
140 00:24:45.120 ⇒ 00:24:45.840 Vimarsh Patel: Yancy.
141 00:24:45.840 ⇒ 00:24:46.880 Pranav: then…
142 00:24:47.180 ⇒ 00:25:01.950 Pranav: build a better system, is really interesting. It’s definitely something that a lot of clients look for. Is that something that you thought of? And if not, that’s totally fine. Like, how would you design something like that for this product?
143 00:25:02.550 ⇒ 00:25:14.240 Vimarsh Patel: So, I’ve definitely, like, we have not worked on it, but it’s been in our mind, and, like, obviously, I think it’s important to, I think it can help in multiple ways, not just, like, understanding how are
144 00:25:14.780 ⇒ 00:25:32.810 Vimarsh Patel: how our system works, but also in understanding what the clients are coming in for, and, like, how they’re… like, how, satisfied they were, it can help with that also, like, client satisfaction with the customer satisfaction, I think that way. So, I think… and understanding transcripts, again, I think,
145 00:25:32.810 ⇒ 00:25:40.990 Vimarsh Patel: I think the main purpose is to understand the intent from the transcript, like, what exactly, how the customer felt, or like,
146 00:25:41.300 ⇒ 00:25:50.190 Vimarsh Patel: And what the agent also did. I think it is… I think for extracting intent, I would say maybe, we can again… like, I’m…
147 00:25:51.160 ⇒ 00:25:54.869 Vimarsh Patel: Obviously, manually going through all the transcript is going to be a bit of a task.
148 00:25:55.020 ⇒ 00:26:14.460 Vimarsh Patel: And I think, obviously, the LLM can help here as well, like, and it is very good at summarizing and extracting intent, so I think we can do that, and maybe we can make it in the prompt, like, to categorize, like, what does it think of, like, out of 10 transcripts it can give us, and, like, how many went back to the human, how many…
149 00:26:14.460 ⇒ 00:26:23.919 Vimarsh Patel: were completed in terms of orders, and how many were unsatisfied. Maybe it can help us classify such things, and then we can look at this data and then improve
150 00:26:23.920 ⇒ 00:26:28.630 Vimarsh Patel: our agent or the tool calling based on that. That is… that would be my approach, I guess.
151 00:26:28.940 ⇒ 00:26:40.969 Pranav: Yeah, yeah, I think that’s a… that sounds solid. Yeah, I kind of want to talk about a similar project that, I’ve been working on here at Brainforge, and then just understanding, like.
152 00:26:41.210 ⇒ 00:26:46.859 Pranav: how you would think about certain problems that we’ve had, in our past.
153 00:26:48.070 ⇒ 00:26:59.860 Pranav: And so, yeah, one problem that we had was with a chatbot that we’re working with, and it had integrations with different MCP servers.
154 00:27:00.130 ⇒ 00:27:16.270 Pranav: And then, yeah, so to give you just, like, full clarity, like, to chatbot for an e-commerce company, and one of the MCP servers was for Shopify. One of the issues we had was that when the user would ask, what was my revenue from yesterday.
155 00:27:16.370 ⇒ 00:27:23.690 Pranav: every once in a while, it would generate fake data, and then give that to the user.
156 00:27:24.820 ⇒ 00:27:27.420 Pranav: So… what…
157 00:27:27.640 ⇒ 00:27:39.139 Pranav: I kind of told you a little bit about, like, the application. You know, it has, like, the… you know, has the natural language processor, but then it also has, like, the MCP server to…
158 00:27:39.390 ⇒ 00:27:44.910 Pranav: you know, pull in the data from the different sources, like Shopify.
159 00:27:45.610 ⇒ 00:27:59.419 Pranav: given, you know, the problem that I just, presented to you, what would be… how would you try to diagnose this issue? What comes to mind? Like, what is the issue, just, like, off the bat? And then we can ask some follow-ups from there.
160 00:28:00.130 ⇒ 00:28:01.820 Vimarsh Patel: I… can I ask a follow-up before I.
161 00:28:01.820 ⇒ 00:28:02.970 Pranav: Definitely, definitely, yeah.
162 00:28:02.970 ⇒ 00:28:15.629 Vimarsh Patel: Yeah, so basically, the situation is, like, I am a customer, I am using a chatbot that is connected to my Shopify account, and I am asking what my revenue was from that Shopify account, and then it will retrieve the data and then give it back to me.
163 00:28:15.950 ⇒ 00:28:16.740 Pranav: Exactly.
164 00:28:17.020 ⇒ 00:28:23.610 Vimarsh Patel: So, and I think, basically, the chatbot used fake data or, like, hallucinated, or whatever, and it gave me wrong…
165 00:28:23.910 ⇒ 00:28:25.700 Vimarsh Patel: Details about my revenue.
166 00:28:27.100 ⇒ 00:28:35.679 Vimarsh Patel: So, first thing is to, obviously, to see what went wrong, and so for that, obviously, our logging needs to be perfect, like, to see what data it pulls from Shopify MCP server.
167 00:28:35.800 ⇒ 00:28:54.260 Vimarsh Patel: And how… what… what it sent to LLM for… for generating the response. That would be the first step, like, the input and output would be the first step to see, and understand, like, maybe it… from my… from the initial, or, like, what we… what we’re discussing, like, what you told me. I think one approach would be that I think
168 00:28:54.260 ⇒ 00:29:10.999 Vimarsh Patel: the server response was something… like, it might have not given it the exact data that the chatbot was looking for, and since it missed out on… like, it… what it expected was not there, and then I think then it just used fake data, or maybe just hallucinated, and gave the response back to
169 00:29:11.050 ⇒ 00:29:28.949 Vimarsh Patel: the user. I think I… I read something similar recently, where a person used the ChatGPT’s voice feature, and it asked to count its time for running the mile, and it just topped in 5 seconds, and the response was that it took
170 00:29:29.170 ⇒ 00:29:32.830 Vimarsh Patel: 10 minutes for the runner to complete a mile. So I think…
171 00:29:33.180 ⇒ 00:29:48.799 Vimarsh Patel: I think this is kind of a similar situation, where the data was not there, and it just gave a, like, a wrong answer. I think one was, like, we need to add better guardrails as well, like, if the data is not there, or, like, if it is not able, instead of, you know, coming up with an answer, it can just say that.
172 00:29:48.800 ⇒ 00:29:57.410 Vimarsh Patel: I’m sorry, I don’t have the data at hand, or, like, I’m not able to process a request. That would be a much better response for the user than a fake revenue…
173 00:29:57.700 ⇒ 00:30:03.350 Vimarsh Patel: Like, a percentage, or whatever output it is. The second thing would also be, like, to…
174 00:30:03.500 ⇒ 00:30:16.430 Vimarsh Patel: to understand the tool calling, like, when the chatbot is calling Shopify’s MTP server, maybe we can add certain parameters that can help the… so, like, since we are defining, like, the MCP server will have a…
175 00:30:16.460 ⇒ 00:30:35.850 Vimarsh Patel: like, a list of the tools that it can call, and, like, it will have a certain schema, and, like, if it is… so, like, we can keep a certain tool for just for revenue calling, and, like, or, like, something similar that can help retrieve that data better. So, like, if the input is better, the output might also be better, and so that would be my initial approach to figuring out, or going forward.
176 00:30:35.850 ⇒ 00:30:41.030 Pranav: I think you nailed that, yeah. I mean, that’s basically what happened there.
177 00:30:41.170 ⇒ 00:30:50.979 Pranav: The tool call failed, and then it just… we allowed the model to think too much, basically, and then when you allow… allow for that thinking, like.
178 00:30:50.990 ⇒ 00:31:04.210 Pranav: for that deep reasoning, it’s going to generate data. So I guess, you mentioned how we can add some guardrails, right? You know, guardrails can come in different forms,
179 00:31:04.390 ⇒ 00:31:22.099 Pranav: all of these models allow for various parameters for you to modulate exactly how it’s going to give you a response. What are some parameters here that you could modulate, or just things that you can add into that API request to the LLM to
180 00:31:22.640 ⇒ 00:31:27.829 Pranav: As, kind of guardrails that you just mentioned, like, what would be useful as guardrails?
181 00:31:28.670 ⇒ 00:31:38.899 Vimarsh Patel: So, one would be the time range, like, what time range the person is asking the revenue for. Like, is it asking for the 1 month, or, like, 2 days, or, like, 3 days? So, maybe that would help narrow the response down.
182 00:31:38.980 ⇒ 00:31:45.539 Vimarsh Patel: So… and then it would… so, like, if there’s a time range, then it… we can send, like, that to the Shopify API.
183 00:31:45.600 ⇒ 00:32:03.300 Vimarsh Patel: So, that can help, like, at least get some better response, instead of… maybe the request was very vague, and then Shopify couldn’t understand, like, what time range the revenue is being asked for, and then it just sent garbage data, and then the chatbot also gave garbage out. That could be… so, like, I think maybe the time range.
184 00:32:03.300 ⇒ 00:32:07.319 Vimarsh Patel: Yeah, another thing would also be… So I guess focusing.
185 00:32:07.320 ⇒ 00:32:09.890 Pranav: less on the request happening to Shopify?
186 00:32:09.890 ⇒ 00:32:10.480 Vimarsh Patel: Okay.
187 00:32:10.480 ⇒ 00:32:12.519 Pranav: Because that’s kind of just, you know.
188 00:32:12.940 ⇒ 00:32:23.270 Pranav: server, like, invoking certain tools based on the natural language. Instead, think about the API request that’s happening to the LLM.
189 00:32:23.270 ⇒ 00:32:23.590 Vimarsh Patel: Okay.
190 00:32:23.950 ⇒ 00:32:28.800 Pranav: So, the LLM is also processing this to then assess which
191 00:32:29.090 ⇒ 00:32:33.309 Pranav: tools to invoke, right? And then also.
192 00:32:33.590 ⇒ 00:32:44.099 Pranav: actually more accurately, like, with the data that is coming in from Shopify, what to do with that data goes through an LLM, right?
193 00:32:44.600 ⇒ 00:32:46.000 Pranav: And so…
194 00:32:46.450 ⇒ 00:33:00.400 Pranav: with that LLM request, you know, it has the Shopify data. What are the parameters that you can modulate, in that API request to, let’s say, you know, OpenAI to decrease the chance of,
195 00:33:00.880 ⇒ 00:33:02.990 Pranav: Of, hallucination.
196 00:33:07.240 ⇒ 00:33:12.849 Vimarsh Patel: So, one thing that comes to my mind is putting checks, like, once we get the response from the Shopify API.
197 00:33:12.900 ⇒ 00:33:31.720 Vimarsh Patel: So, you, like, when we are testing it out, we can just look at some of the responses that we have got from the Shopify API, and we can keep checks, like, once we know for certain, or like, maybe the Shopify document can also help us, like, once we know for a certain fact that when we ask for revenue on the Shopify API, not through an LLM, but in general, if you just make a normal call.
198 00:33:31.890 ⇒ 00:33:36.839 Vimarsh Patel: Through our code, and we can see what the response structure is, and then we put those checks in our
199 00:33:36.960 ⇒ 00:33:47.870 Vimarsh Patel: in our code as well, before it is… before it goes to the LLM. Like, we… we make sure that if… are these certain, like, variables present or not? And so that can help us
200 00:33:47.870 ⇒ 00:34:03.769 Vimarsh Patel: make sure, like, if certain things are missing, then we don’t send it to LLM at all, like, we stop there. We make… we are… we make the call again, like, this is missing. So go back, make the API call again, like, do a retry, maybe 2 or 3 retries might help us. If it is… if it is still not happening, then we get a human in the loop.
201 00:34:03.790 ⇒ 00:34:13.659 Vimarsh Patel: But, so, one… one garter would be to make sure the data has… is, up to the mark of that, what we want. If it is not, then do something else, like.
202 00:34:13.840 ⇒ 00:34:18.609 Vimarsh Patel: then that would be the one guardrail. Another,
203 00:34:18.860 ⇒ 00:34:28.040 Vimarsh Patel: would be, like, for… let us assume that even if after checks, it still went and it still gave us garbage out. So, even at that point,
204 00:34:28.210 ⇒ 00:34:39.530 Vimarsh Patel: Once… before it tells the user, like, this is your revenue, or whatever, maybe do one more, cross-check with the data that we got, and if it is aligning.
205 00:34:39.550 ⇒ 00:34:53.699 Vimarsh Patel: with it or not, something like that, to make sure, like, one final check before it passes through to the user. I think maybe it can help. Like, it is just hardening what we have. Like, this is what is coming to the top of my mind right now, like…
206 00:34:53.860 ⇒ 00:34:58.200 Vimarsh Patel: So, like, maybe if you have some, like, a track of that, then maybe I can go in certain things.
207 00:34:58.800 ⇒ 00:35:05.680 Pranav: Yeah, no, no, I think that’s, that’s great, like, the way that you’re thinking about problems. I guess here, too, like, specific…
208 00:35:06.150 ⇒ 00:35:23.140 Pranav: parameters that you can update on an LLM API request, just in general, right, are temperature, output, tokens, system prompt as well. Yeah, you mentioned how some things, like with the system prompt, like, which was completely right, like, we could…
209 00:35:23.140 ⇒ 00:35:29.019 Pranav: Mention, hey, do not… like, if you could even explicitly say, right, just, like, do not generate test data.
210 00:35:29.020 ⇒ 00:35:42.220 Pranav: Things like that make a difference. But specifically for this case, and I’ll, I’ll wrap up soon and give you some time for questions as well. But yeah, specifically for this case.
211 00:35:43.360 ⇒ 00:35:50.009 Pranav: What was also causing some issues is that we were allowing the model, reasoning.
212 00:35:50.010 ⇒ 00:35:51.270 Vimarsh Patel: So…
213 00:35:51.270 ⇒ 00:36:01.370 Pranav: for something like this, you don’t necessarily need reasoning. We’re not asking you to do analysis on the data, we’re just asking to, you know, create a pretty table based on a certain time range of…
214 00:36:01.370 ⇒ 00:36:13.269 Pranav: Shopify revenue, right? So, it’s really important to disable reasoning, because if you have it enabled, it actually increases, or it doesn’t let you modulate temperature.
215 00:36:13.270 ⇒ 00:36:18.990 Pranav: And so temperature’s something you want to keep pretty low here, so it’s less free-thinking, and it’s more deterministic.
216 00:36:19.050 ⇒ 00:36:20.480 Pranav: And, it’s output.
217 00:36:20.480 ⇒ 00:36:21.310 Vimarsh Patel: Exactly.
218 00:36:21.310 ⇒ 00:36:24.380 Pranav: Yeah, so, yeah, no, that was great,
219 00:36:24.610 ⇒ 00:36:29.070 Pranav: I’m wondering if you have any questions, we can go over them. I know I was a little bit late to get started.
220 00:36:29.070 ⇒ 00:36:30.069 Vimarsh Patel: No, it is, no, no worries.
221 00:36:30.530 ⇒ 00:36:48.440 Vimarsh Patel: Yeah, I do have, like… so, currently, like, what is your… like, what is… what would be your evaluation framework be for, like, any agentic work that you’re doing, or even, like, the automation work? Like, what is your evaluation framework for those? And, like, to understand if the features are actually giving out value to the client or not, so how do you do that?
222 00:36:48.990 ⇒ 00:36:56.369 Pranav: Yeah, so… That’s a really good question. We do it in various ways, and…
223 00:36:56.950 ⇒ 00:37:16.789 Pranav: there’s automated evaluation frameworks, but then there’s also just how we can evaluate just how well the agent is performing. And it’s also based on what are the KPIs that we have signed onto with the client, as well as what is the client’s KPIs for their higher-ups, right?
224 00:37:16.790 ⇒ 00:37:17.280 Vimarsh Patel: Yes.
225 00:37:17.280 ⇒ 00:37:30.340 Pranav: And so, for example, one KPI that’s super simple for one of our clients is just usage. So that’s super easy to evaluate, super easy to track. Something that’s a little bit more complex is…
226 00:37:30.890 ⇒ 00:37:34.039 Pranav: not, I mean, not that much more complex, it’s just,
227 00:37:34.480 ⇒ 00:37:48.119 Pranav: what is the feedback? So, we allow for, you know, each chatbot response to have a thumbs up, thumbs down. For the thumbs up and thumbs down. We then allow for additional evaluation in terms of
228 00:37:48.460 ⇒ 00:37:58.700 Pranav: okay, where’s that feedback going? How are we executing on that feedback? And then, so what we’re working on currently is sending out daily memos to let them know, hey.
229 00:37:58.700 ⇒ 00:38:12.300 Pranav: the chatbot improved based on this feedback that you gave us. Certain feedback was actually even rejected because it conflicts with other feedback that we’ve gotten. So, we’re doing things like that.
230 00:38:12.840 ⇒ 00:38:24.139 Pranav: it’s really… it ends up being kind of, you’re getting so much data, right? It’s, like, similar to your restaurant example, where you’re getting so much data, there’s so many insights to gain from that.
231 00:38:24.630 ⇒ 00:38:30.639 Pranav: So, yeah, you have the… you can easily just, like, send them over a,
232 00:38:31.280 ⇒ 00:38:43.849 Pranav: a recommendation on a menu item, but then analyzing, okay, what do all these transcripts say? What… what can we gain from, what can we gain from all these transcripts in terms of, okay, what are they asking for?
233 00:38:44.290 ⇒ 00:38:52.890 Pranav: That’s with AI, where, like you said, it does a really good job of summarizing, so it can really get you to… from, like, point of raw data to insights very quickly.
234 00:38:53.480 ⇒ 00:38:54.500 Vimarsh Patel: No, yes.
235 00:38:54.500 ⇒ 00:38:56.510 Pranav: But, yeah, no, that’s a great question.
236 00:38:57.320 ⇒ 00:39:10.680 Vimarsh Patel: And one would be, like, how do you balance, like, shipping fast with, like, maintaining quality as well for your clients? Like, because I’m assuming, like, I was told that sometimes you have to work with multiple clients, and then I think you have a specific,
237 00:39:10.680 ⇒ 00:39:19.870 Vimarsh Patel: like, time or deadlines to maintain, so, like, I think in that terms, past and quality have to go side by side, so, like, how does the team, like, work on that and, like, balance that?
238 00:39:20.590 ⇒ 00:39:27.740 Pranav: Yeah, I mean, it depends on your role at Brainforge, and I think it’s a healthy push and pull.
239 00:39:27.840 ⇒ 00:39:31.000 Pranav: For me, as kind of the…
240 00:39:31.330 ⇒ 00:39:44.550 Pranav: we have different, like, terms internally, but you can kind of think of it in, like, a consulting… in a consulting world, it’s kind of called, like, the account manager, or even just, like, the product owner, in the just, like, regular, you know, tech fields.
241 00:39:44.690 ⇒ 00:40:03.870 Pranav: I… my kind of goal and my motivation is to ship as quick as possible, because that’s what makes the client happy. However, we also have… I work very closely with Sam, and his KPIs are more closely tied to how do we make sure we’re estimating things properly, and how do we make sure we’re
242 00:40:03.880 ⇒ 00:40:20.280 Pranav: reducing the amount of bug fix tickets that are being created. And so, we also, you know, we work together, we don’t push each other too much, right? We’re still a team here. What we do is,
243 00:40:20.510 ⇒ 00:40:25.100 Pranav: We really… and individually, you know, sometimes you have… you wear multiple hats.
244 00:40:25.310 ⇒ 00:40:32.950 Pranav: I think it’s really important to figure out what exactly the customer wants, the client wants in this case,
245 00:40:33.560 ⇒ 00:40:39.249 Pranav: And then, not worrying about all the bells and whistles outside of that.
246 00:40:39.250 ⇒ 00:40:39.569 Vimarsh Patel: Me too.
247 00:40:39.570 ⇒ 00:40:55.040 Pranav: It’s really easy to ink, like, over-complicate the technical approach on something when, you know, just a simple, maybe, LLM request, or just, like, a system of LLM requests is all you needed.
248 00:40:55.480 ⇒ 00:41:04.909 Pranav: And so, yeah, we’re noticing things change, like, week… like, month to month, you know? Like, these models, their context windows get bigger and bigger, they get cheaper and cheaper,
249 00:41:05.260 ⇒ 00:41:11.290 Pranav: And so, it really makes you… like, one kind of decision we’re making a lot is kind of what we discussed in the beginning of this interview, which is…
250 00:41:11.290 ⇒ 00:41:11.630 Vimarsh Patel: Yes.
251 00:41:11.630 ⇒ 00:41:21.140 Pranav: Hey, when do we actually need RAG? When do we actually need data warehouses? And when can we just use maybe a CLI or an MCP server, instead?
252 00:41:21.530 ⇒ 00:41:22.590 Pranav: So…
253 00:41:22.720 ⇒ 00:41:32.520 Pranav: It’s really about reducing the technical complexity as much as you can, but of course not too much, and then also,
254 00:41:33.070 ⇒ 00:41:45.349 Pranav: AI-assisted development is extremely important. So, it’s, it’s basically, like, required here. You… because it… and it really does maximize your efficiency. You can really, like…
255 00:41:45.750 ⇒ 00:41:52.369 Pranav: ripped through, you know, huge features that used to take Teams just, like, a couple years ago, like, months and, like, days.
256 00:41:52.750 ⇒ 00:41:55.520 Vimarsh Patel: It can write thousand lines of codes like that, so…
257 00:41:55.520 ⇒ 00:41:56.670 Pranav: Yeah, definitely.
258 00:41:57.100 ⇒ 00:42:16.180 Vimarsh Patel: And I think one last question, if you don’t mind, like, if I am supposed to move forward, like, then from the third round, like, how should I, you know, prepare for the panel, and, like, what they would be looking for? Like, is it just my decision-making, or, like, they also… they would also want, like, a correct output from the assessment, or, like, how would… how should my approach be, basically, for that?
259 00:42:16.180 ⇒ 00:42:30.460 Pranav: Yeah, I think it’ll be, you know, kind of, like, it’ll be discussion-based as well. You’ll get a lot more information, from… I’m not sure if you’ve been in touch with, like, who you’ve been in touch with, but you’ll get more information from them.
260 00:42:30.570 ⇒ 00:42:32.400 Pranav: There… there’s, like, a…
261 00:42:32.950 ⇒ 00:42:40.939 Pranav: It’ll be, like, kind of, you know, you’ll meet with a new person, maybe meet with a couple people for, like, a final round,
262 00:42:41.940 ⇒ 00:42:51.289 Pranav: And then, yeah, there may be… there’ll be other components to it as well. You know, every one of these conversations, you try to make a little bit different, so you’ll get more information about that.
263 00:42:51.660 ⇒ 00:42:55.880 Vimarsh Patel: Got it. All right. But yeah, that was all that I had at the top of my mind, and…
264 00:42:56.020 ⇒ 00:42:56.370 Pranav: Awesome.
265 00:42:56.370 ⇒ 00:42:57.279 Vimarsh Patel: Thank you, thank you, thank you.
266 00:42:57.280 ⇒ 00:42:59.200 Pranav: Yeah, it was great talking to you.
267 00:42:59.200 ⇒ 00:42:59.719 Vimarsh Patel: Yes. I’ll talk to you.
268 00:42:59.720 ⇒ 00:43:00.330 Pranav: Excellent.
269 00:43:00.530 ⇒ 00:43:03.690 Vimarsh Patel: Yeah, thank you for giving up. Yeah, have a good day. Bye-bye.
270 00:43:03.690 ⇒ 00:43:04.670 Pranav: Totally. Bye.