Meeting Title: AI Service Standup Date: 2026-03-10 Meeting participants: Pranav, Mustafa Raja, Brylle Girang, Casie Aviles, Uttam Kumaran, Samuel Roberts
WEBVTT
1 00:00:21.710 ⇒ 00:00:22.740 Brylle Girang: Hello?
2 00:00:24.200 ⇒ 00:00:26.210 Pranav: Hello, hello! How’s it going?
3 00:00:28.450 ⇒ 00:00:30.360 Brylle Girang: Pretty crazy Tuesday.
4 00:00:31.840 ⇒ 00:00:32.930 Pranav: Already?
5 00:00:33.840 ⇒ 00:00:36.949 Brylle Girang: It is already. We’re, we’re half in the month on that.
6 00:00:37.310 ⇒ 00:00:43.549 Brylle Girang: So many things happening, and I can’t imagine how it’s going in with you, Pranav. How’s it going with ABC?
7 00:00:44.350 ⇒ 00:00:46.590 Brylle Girang: ABC’s good.
8 00:00:46.590 ⇒ 00:00:49.220 Pranav: Yeah, it is kind of like, you know…
9 00:00:49.810 ⇒ 00:00:57.869 Pranav: there’s… this is a period where you go from, like, knowing nothing last week to then, like, a week later, just, like, knowing way more stuff, right?
10 00:00:58.510 ⇒ 00:00:59.590 Pranav: So…
11 00:00:59.860 ⇒ 00:01:06.730 Pranav: Yeah, I feel a lot more confident on this project first. At first, when it’s just like, you know nothing, it’s just like, shoot, okay, keep on just…
12 00:01:07.250 ⇒ 00:01:16.739 Pranav: asking questions, like, running around, looking for information. But yeah, now I feel like at this point, I understand, like, everything that’s going on.
13 00:01:16.920 ⇒ 00:01:18.560 Pranav: Now it’s really just about, like.
14 00:01:19.270 ⇒ 00:01:26.119 Pranav: figuring out exactly if we’re working on the highest impact items, because I know now very well, like.
15 00:01:26.340 ⇒ 00:01:28.109 Pranav: What the client wants.
16 00:01:28.510 ⇒ 00:01:40.670 Pranav: like, what are they looking for in terms of what does Andy provide? And so now I just… and we have a session later today for me to just basically align the team on, like, what those things are. And, like.
17 00:01:41.690 ⇒ 00:01:46.920 Pranav: Take those, like, problems that the client has and, like, turn them into, like, things we can work on.
18 00:01:47.160 ⇒ 00:01:47.970 Pranav: So…
19 00:01:47.970 ⇒ 00:01:52.440 Brylle Girang: Can you tell me more about the client? I feel like I only have, like, a breadcrumb.
20 00:01:52.810 ⇒ 00:01:53.650 Pranav: Totally.
21 00:01:55.040 ⇒ 00:02:02.620 Pranav: Yeah, I can… I can go into however deep you want me to. So, basically, the product that we built for them.
22 00:02:02.770 ⇒ 00:02:08.969 Pranav: the main product that we built for them is this thing called Andy, which is a Google Chat Bot.
23 00:02:09.389 ⇒ 00:02:09.769 Brylle Girang: if you don.
24 00:02:09.770 ⇒ 00:02:15.339 Pranav: Just go onto your Google Chat, you can speak to Andy just like how you’d speak to anybody else in your Google Workspace.
25 00:02:15.930 ⇒ 00:02:23.399 Pranav: What it is, essentially, the most, just kind of bare-bones explanation, is that it’s a RAG app on top of a Google Doc.
26 00:02:23.760 ⇒ 00:02:25.960 Pranav: So…
27 00:02:26.250 ⇒ 00:02:38.649 Pranav: that Google Doc is then, you know, we create embeddings on it, and that Google Doc, for just one department is over 200 pages. And so we’re supporting, I think, 5 different departments.
28 00:02:39.230 ⇒ 00:02:44.780 Pranav: As of right now. So… The complexity comes from just, like.
29 00:02:45.310 ⇒ 00:02:51.179 Pranav: Of course, being as quick as possible, as well as being as accurate as possible, and…
30 00:02:51.500 ⇒ 00:02:57.650 Pranav: An added element to this is that the people at ABC that are using this product are using it when they’re on
31 00:02:57.960 ⇒ 00:03:00.950 Pranav: The… on the phone with a customer.
32 00:03:01.080 ⇒ 00:03:15.740 Pranav: And so, they get a query from a customer, and then they type into this chatbot. And so, that’s why latency is super important, right? Like, the matter of seconds, like, if you want to sound like a real human being, like, you can’t just be waiting 20 seconds.
33 00:03:15.900 ⇒ 00:03:23.409 Pranav: So, yeah, it’s a pretty interesting problem, but it’s… when I think about it that way, it’s actually pretty simple.
34 00:03:24.100 ⇒ 00:03:28.519 Pranav: And so, yeah, just kind of, right now, what we’re working on is, like.
35 00:03:28.690 ⇒ 00:03:34.699 Pranav: some migration stuff. We used to have the whole, like, backend built on N8N, now it’s,
36 00:03:35.040 ⇒ 00:03:38.669 Pranav: We’re building it with, using Mastra.
37 00:03:38.910 ⇒ 00:03:44.119 Pranav: And yeah, so that’s kind of where we’re at right now.
38 00:03:45.090 ⇒ 00:03:55.180 Brylle Girang: That’s super cool. My role before Brainforge is actually the leader of a help desk team, where we were the ones answering questions for our customers.
39 00:03:55.380 ⇒ 00:04:11.370 Brylle Girang: One of the main problems that we have there is that there’s no singular knowledge base where agents can just get context right away, right? We have tried so many things. We have tried, like, Google Gemini when it comes to parsing the Google Doc. We have tried
40 00:04:11.620 ⇒ 00:04:13.970 Brylle Girang: Built-in products.
41 00:04:14.320 ⇒ 00:04:18.650 Brylle Girang: And I don’t believe that we have succeeded on that part, because
42 00:04:18.779 ⇒ 00:04:29.410 Brylle Girang: AI was pretty young during those days, and then REG was not… it was not a well-known method within the… within the business, so this is super cool.
43 00:04:30.510 ⇒ 00:04:35.760 Brylle Girang: What opportunities… what expansion opportunities are you seeing right now with ABC?
44 00:04:36.470 ⇒ 00:04:43.020 Pranav: Yeah, so, right after this, once we have an application, or once we have a…
45 00:04:43.080 ⇒ 00:04:57.519 Pranav: an AI system that actually, like, performs exactly where we want it to perform in terms of latency and accuracy. The very next step is a dashboard where they can, admins, like, I guess managers at the company can just, like, assess,
46 00:04:58.230 ⇒ 00:04:59.210 Pranav: just…
47 00:04:59.500 ⇒ 00:05:06.000 Pranav: They mentioned Dashboard as something that even Utam talked to them about, like, early when they were scoping Andy.
48 00:05:06.100 ⇒ 00:05:12.359 Pranav: the exact requirements of what this dashboard looks like, I’m not exactly sure what would be useful for them.
49 00:05:14.250 ⇒ 00:05:23.730 Pranav: probably, like, it would be a place where they can log certain calls, and just, like, what went well, is my guess.
50 00:05:24.430 ⇒ 00:05:29.810 Pranav: we already have, like, a dashboard that pulls in, Andy…
51 00:05:30.540 ⇒ 00:05:50.070 Pranav: messaging data. So, like, whatever messages were sent to Andy, by who, what those questions were, what the execution time was, like, all that information we have, and we can expose it, and I think that would all be, like, super, like, relevant data for, a dashboard to be on.
52 00:05:50.290 ⇒ 00:05:58.000 Pranav: And I think maybe there, too, like, it could be an assessment tool for their internal, customer service reps.
53 00:05:58.550 ⇒ 00:06:02.309 Pranav: So, I think that’s probably next.
54 00:06:03.910 ⇒ 00:06:12.910 Pranav: Yeah, and that’s… in conversations that I have with them, I’ll probably start bringing that up with them, like, okay, like, we’re getting close on this latency and accuracy stuff, like…
55 00:06:13.100 ⇒ 00:06:17.340 Pranav: You guys mentioned how, like, a dashboard would be cool, like, let’s start defining, like, what that looks like.
56 00:06:18.360 ⇒ 00:06:24.910 Brylle Girang: Okay, okay. I feel like we’re in a really good place with ABC, because the system that we’re building for them is…
57 00:06:25.330 ⇒ 00:06:29.779 Brylle Girang: Needs to be long-term. Like, it’s not… they can’t just…
58 00:06:29.970 ⇒ 00:06:37.530 Brylle Girang: get out of our engagement in one time, right? Have they… So…
59 00:06:37.680 ⇒ 00:06:45.589 Brylle Girang: have they explored, like, actually applying the AI stuff that we’re building directly to the external customers?
60 00:06:45.930 ⇒ 00:06:49.309 Brylle Girang: I’m saying this because in my previous role.
61 00:06:49.410 ⇒ 00:06:55.790 Brylle Girang: We have tried, like, providing all the tools that we can to our internal agents.
62 00:06:55.960 ⇒ 00:07:04.899 Brylle Girang: But at the end of the day, the more important part is, like, providing the tools that we can to the actual customers, so that
63 00:07:05.010 ⇒ 00:07:07.069 Brylle Girang: They don’t need to go through the agents.
64 00:07:07.560 ⇒ 00:07:08.980 Pranav: Yeah, that’s a…
65 00:07:09.090 ⇒ 00:07:17.089 Pranav: One thing that the agents do that we haven’t been thinking about with Andy is just, that customer relationship.
66 00:07:17.170 ⇒ 00:07:18.180 Brylle Girang: Yeah.
67 00:07:18.210 ⇒ 00:07:24.669 Pranav: And so, for… for them, that’s important. That’s, like, that’s something that they mentioned on multiple occasions.
68 00:07:25.590 ⇒ 00:07:38.909 Pranav: So, potentially, though, that is something that they, like, I mean, Andy’s definitely not at that point right now, right? Like, they would lose a lot of business, I’m sure, if, like, they were using Andy exclusively.
69 00:07:39.230 ⇒ 00:07:49.439 Pranav: However, yeah, down the line, like, maybe they just need to see how well Andy performs, and that’s just gonna be, like, more than just, like, what we say, but, like, what is it actually doing?
70 00:07:49.830 ⇒ 00:08:08.049 Pranav: once they… once that happens, I could see, like… it seems like a very natural progression, right? Like, okay, Andy’s doing everything, like, if it really gets to the point where, like, the customer service reps are just, like, taking in the question that the customer’s asking and just, like, verbatim, just pasting it in there.
71 00:08:08.050 ⇒ 00:08:15.519 Pranav: That’s a pretty good sign that, like, you don’t… The customer service reps can probably be, like, taken out of that, like, that process.
72 00:08:15.970 ⇒ 00:08:18.790 Brylle Girang: Yeah, might as well remove the middleman, right?
73 00:08:18.940 ⇒ 00:08:19.590 Pranav: Yeah.
74 00:08:19.940 ⇒ 00:08:28.269 Brylle Girang: Okay, okay. So, are there any… before I dive into, like, ABC and Anti-Mustafa, Casey, do you have anything that you want to share?
75 00:08:33.759 ⇒ 00:08:43.349 Casie Aviles: No, not really. I mean, we’re just, you know, working on… and at the moment, we’re just, making sure that… so, yeah, like what Branov mentioned, we were just…
76 00:08:43.679 ⇒ 00:08:48.839 Casie Aviles: Making sure, like, right now, our focus right now is just to get the latency
77 00:08:48.979 ⇒ 00:09:02.909 Casie Aviles: better than where it was, and yeah, and then right now we’re also making sure that the accuracy is good, although we don’t really have, like, a very good measure in place right now for the accuracy. We’ve tried in the past, like.
78 00:09:03.429 ⇒ 00:09:08.199 Casie Aviles: Setting up the evaluation… Metrics, but…
79 00:09:08.759 ⇒ 00:09:11.499 Casie Aviles: Yeah. Right now, we don’t have, like…
80 00:09:11.669 ⇒ 00:09:15.479 Casie Aviles: One in place, like, a very solid one right now, but…
81 00:09:15.769 ⇒ 00:09:18.549 Casie Aviles: Yeah, that’s something we can figure out as we go.
82 00:09:19.180 ⇒ 00:09:30.049 Brylle Girang: And, I don’t know, based on our previous discussions, is that supposed to be under our scope? Like, measuring how effective Andy is, or is that… is that supposed to be?
83 00:09:30.460 ⇒ 00:09:32.210 Brylle Girang: under ABC.
84 00:09:35.930 ⇒ 00:09:37.980 Casie Aviles: I think… yeah, go ahead, go ahead.
85 00:09:38.810 ⇒ 00:09:43.490 Pranav: Yeah, I would say… and I… Casey, too, like, you chime in too, but
86 00:09:43.600 ⇒ 00:09:49.050 Pranav: I would say it is part of our scope, just because… we’ve been…
87 00:09:49.450 ⇒ 00:09:54.900 Pranav: I mean, it’s a good question, actually, like, what does the SOW exactly say?
88 00:09:56.000 ⇒ 00:09:59.109 Pranav: We’ve definitely been operating under the…
89 00:09:59.520 ⇒ 00:10:08.670 Pranav: the sense that it is part of scope, that reducing accuracy, I mean, sorry, increasing accuracy, reducing execution time, so…
90 00:10:09.930 ⇒ 00:10:11.500 Pranav: I don’t think we…
91 00:10:12.570 ⇒ 00:10:22.239 Pranav: I need to look at the SOW to see, like, okay, do we set, like, a certain bar for, like, what that looks like? And then after that point, then it’s out of scope? I’ll look into that.
92 00:10:23.180 ⇒ 00:10:29.879 Brylle Girang: Okay, I’m actually asking Chrysler right now, and let’s see if it… Answers this, but yeah,
93 00:10:30.050 ⇒ 00:10:34.419 Brylle Girang: If that’s going to be under our scope, and they push for that to be under our scope.
94 00:10:35.560 ⇒ 00:10:42.319 Brylle Girang: do we have any plans on how we can do that? And based on Casey’s answer earlier, we don’t have one.
95 00:10:42.320 ⇒ 00:10:42.830 Pranav: We’re accurate.
96 00:10:42.830 ⇒ 00:10:43.700 Brylle Girang: Are we…
97 00:10:43.700 ⇒ 00:10:44.819 Pranav: Yeah. Yeah.
98 00:10:45.530 ⇒ 00:10:52.659 Brylle Girang: I think we can start with, like, checking the feedback tickets that’s coming in. We do have that, we do have those, right?
99 00:10:52.910 ⇒ 00:10:53.480 Pranav: Yep.
100 00:10:54.050 ⇒ 00:10:56.690 Brylle Girang: And do we have logs on, like, how…
101 00:10:58.040 ⇒ 00:11:01.079 Brylle Girang: how many uses Andy, how…
102 00:11:01.080 ⇒ 00:11:02.129 Casie Aviles: Yeah, yeah, we do, we do.
103 00:11:02.130 ⇒ 00:11:03.860 Brylle Girang: triggers the conversations, okay.
104 00:11:04.900 ⇒ 00:11:05.890 Brylle Girang: Gotcha.
105 00:11:06.900 ⇒ 00:11:12.540 Pranav: It’s just that… Yeah, we actually have a monologue here. I think we could probably take advantage of all the disk…
106 00:11:12.690 ⇒ 00:11:14.310 Pranav: We can take advantage of, like…
107 00:11:14.540 ⇒ 00:11:33.349 Pranav: the human elements that kind of have, like, labeled the data as, like, good versus bad. We’ve gotten a lot of thumbs up and thumbs down. That can be used as, like, a dataset for evaluating how are these future progressions of Andy operating. They’re like, we need to make sure there’s no regression there.
108 00:11:35.130 ⇒ 00:11:47.049 Pranav: Yeah, so creating that data set, I feel like, can be a good measurement of just, like, how accurate is Andy getting? Is it getting more accurate? In terms of latency, I came up with quite, like, a few solutions,
109 00:11:47.270 ⇒ 00:11:54.169 Pranav: some high impact, that could be pretty, like, low time for implementation. Like, we could probably even get them out today.
110 00:11:54.290 ⇒ 00:11:59.309 Pranav: So, yeah, we can talk a little bit more about that, too.
111 00:11:59.800 ⇒ 00:12:00.360 Brylle Girang: Okay.
112 00:12:00.650 ⇒ 00:12:01.970 Brylle Girang: Hi, Autumn. Hi, Sam.
113 00:12:02.530 ⇒ 00:12:03.759 Brylle Girang: Right now, what…
114 00:12:04.350 ⇒ 00:12:09.179 Brylle Girang: Britta was just walking me through ANDI and what we have been doing so far, and currently.
115 00:12:09.180 ⇒ 00:12:09.650 Samuel Roberts: Okay.
116 00:12:09.650 ⇒ 00:12:10.990 Brylle Girang: We’re just thinking…
117 00:12:11.660 ⇒ 00:12:18.319 Brylle Girang: the evaluation stuff that we’re doing. I asked Pranov, and I’m asking Kursa right now.
118 00:12:18.660 ⇒ 00:12:26.500 Brylle Girang: In the SOW, whose responsibility is it to measure, like, Andy’s performance? And I just wanted to make sure that
119 00:12:26.980 ⇒ 00:12:29.059 Brylle Girang: If… whatever happens.
120 00:12:29.230 ⇒ 00:12:36.250 Brylle Girang: Andy or ABC doesn’t get back to us and say, hey, Andy’s not working, we’re not getting the correct answers.
121 00:12:36.350 ⇒ 00:12:40.200 Brylle Girang: So… Who should be responsible for measuring that?
122 00:12:41.720 ⇒ 00:12:44.210 Uttam Kumaran: Yeah, I feel like we’ve measured it to date, right, and we’ve made.
123 00:12:44.210 ⇒ 00:12:45.000 Samuel Roberts: Yeah, sister.
124 00:12:45.000 ⇒ 00:12:45.710 Uttam Kumaran: today.
125 00:12:47.160 ⇒ 00:12:48.859 Samuel Roberts: Yeah, we have the…
126 00:12:49.050 ⇒ 00:12:56.750 Samuel Roberts: the, the dashboards and stuff that we share with them, and I believe they have access to it, but usually that’s presented in the meetings anyway.
127 00:12:56.990 ⇒ 00:13:02.740 Samuel Roberts: Whether or not they’re tracking it, I don’t know, but… Yeah, I don’t know…
128 00:13:03.180 ⇒ 00:13:08.810 Samuel Roberts: how much we’ve focused on… I mean, we’re focusing now on a lot more valuation, time…
129 00:13:08.980 ⇒ 00:13:11.740 Samuel Roberts: Or not evaluation time, like, inference time, total…
130 00:13:11.920 ⇒ 00:13:21.139 Samuel Roberts: kind of that total round trip, I think, Casey, I saw your post. Yeah, good. So that was, that was related to the migration stuff as well, as well as just, like, ability to…
131 00:13:21.570 ⇒ 00:13:33.969 Samuel Roberts: refine… I think Amber was mentioning stuff about, follow-up questions and things, like, things that N8N wasn’t able to do. So, like, that might change a little bit how we measure things, because now the conversations are a little more back and forth, but yeah, it’s definitely more on us, I think, than…
132 00:13:34.230 ⇒ 00:13:36.070 Samuel Roberts: You know, they have their…
133 00:13:36.330 ⇒ 00:13:45.649 Samuel Roberts: thumbs up, thumbs down feedback, and so they might have a sense of maybe how they think things are, but then the dashboards show, like, the percentages and stuff there, so…
134 00:13:47.810 ⇒ 00:13:50.500 Brylle Girang: We’ll do… Yeah, sorry, go ahead.
135 00:13:51.040 ⇒ 00:13:56.930 Brylle Girang: So, like, on our vantage point, like, is Andy effective in its current state?
136 00:13:58.780 ⇒ 00:14:00.930 Samuel Roberts: Danny, effective, yeah, yeah, right, okay.
137 00:14:04.270 ⇒ 00:14:09.949 Brylle Girang: Gotcha. I’m really interested in this, because I came from a customer service background, and…
138 00:14:09.950 ⇒ 00:14:10.690 Samuel Roberts: Oh, yeah.
139 00:14:10.690 ⇒ 00:14:15.119 Brylle Girang: was similar to one of the last projects that I have built for them.
140 00:14:15.300 ⇒ 00:14:21.459 Brylle Girang: I would love to look at the dashboards just for my own curiosity, if that’s okay.
141 00:14:23.230 ⇒ 00:14:24.919 Samuel Roberts: Yeah, I think we can probably get that.
142 00:14:25.330 ⇒ 00:14:26.140 Samuel Roberts: Okay.
143 00:14:26.370 ⇒ 00:14:38.679 Samuel Roberts: Is that overview? There’s something else I was gonna say there, and I forgot what. Oh, yeah, the other side of this is, like, even when Andy is not necessarily, like, the fastest thing, it’s probably still a little faster than the, like, especially certain CSRs, searching the…
144 00:14:39.170 ⇒ 00:14:58.639 Samuel Roberts: the docs that were already, like, the work, I think, this was definitely pre-me a little bit. The central docs came from everything being spread across tons of other places. So, like, even that work of, like, consolidating, and the ZipsDB is another one that is definitely more consolidating their data, because it was a little…
145 00:14:59.600 ⇒ 00:15:08.050 Samuel Roberts: chaotic, I think, beforehand. So, like, yeah, getting Andy… optimizing Andy, getting Andy faster is definitely good, but even the ability to just say, like.
146 00:15:08.390 ⇒ 00:15:11.359 Samuel Roberts: You know, what do we normally do for this, and have it spit back the…
147 00:15:11.540 ⇒ 00:15:23.900 Samuel Roberts: the procedure, and the zips when it works, certainly, but there’s also the back and forth, like, keeping the data fresh and everything, so… I think it’s still a… it’s still been a value add to them in general, I would say, but now there’s optimization stuff to do, yeah.
148 00:15:26.170 ⇒ 00:15:33.720 Samuel Roberts: Cool. Other… anything else we missed this morning, just now, or you guys… Okay, I guess let’s jump.
149 00:15:33.720 ⇒ 00:15:37.150 Brylle Girang: I think… I just think that we can focus on the wrist.
150 00:15:37.310 ⇒ 00:15:43.639 Brylle Girang: For this session, and do you want to go through the linear stuff right now with them, or we can scrap that?
151 00:15:44.340 ⇒ 00:15:52.059 Uttam Kumaran: Yeah, I mean, I guess, like, up to you guys, Pranav, like, do you feel good on… on ABC stuff, or what else can we help with?
152 00:15:52.060 ⇒ 00:15:54.569 Samuel Roberts: We do have a working session later as well.
153 00:15:54.570 ⇒ 00:16:06.770 Pranav: Yeah, before that working session, I’m gonna just fully, like, hash out, like, what is the current state of, like, these three different workflows, and just, like, making sure, like, that plan,
154 00:16:06.980 ⇒ 00:16:14.300 Pranav: looks like the right plan in order to execute on what the problems Yvette and Janice have brought up to me. Okay.
155 00:16:14.600 ⇒ 00:16:21.499 Pranav: So, yeah, basically I’ll come into that working session just kind of, like, being like, okay, I like this plan, and Sam, like.
156 00:16:21.650 ⇒ 00:16:28.950 Pranav: like, I’ll come up with… if I’m, like, okay, we should change up this plan, Sam, like, I’ll probably, like, look for you to, like, refine, like, what I think
157 00:16:29.140 ⇒ 00:16:34.290 Pranav: could be the things to fix these problems, and so I think that’s probably the best way to use that working session.
158 00:16:34.860 ⇒ 00:16:35.420 Samuel Roberts: Okay.
159 00:16:35.470 ⇒ 00:16:36.470 Pranav: Yeah.
160 00:16:36.720 ⇒ 00:16:37.909 Samuel Roberts: Yeah, those are the others…
161 00:16:38.120 ⇒ 00:16:42.010 Pranav: Like, looking at linear, probably we don’t need to,
162 00:16:42.210 ⇒ 00:17:00.979 Pranav: there’s just, like, a few things top of mind, like, I just did, like, a whole, just, like, cursor report, just, like, on, like, the master app, just, like, on the migration progress, branch right now, and there seems like there’s a few things here that could just be, like, easy wins to just, like, reduce the execution time.
163 00:17:01.110 ⇒ 00:17:02.070 Pranav: So…
164 00:17:02.210 ⇒ 00:17:10.030 Pranav: Yeah, that would be great. Casey, too, like, I was looking at the… the sheet that you provided in,
165 00:17:10.170 ⇒ 00:17:20.840 Pranav: In Slack yesterday, which is… yeah, that was awesome. I just want to make sure, like, I’m reading this right, too. So, like, Master is taking a little bit longer than, NNN.
166 00:17:21.069 ⇒ 00:17:22.079 Pranav: Is that right?
167 00:17:22.540 ⇒ 00:17:32.069 Casie Aviles: Yeah, that was the initial test when I didn’t… when it was still, like, using a full agent framework, so I created, like, a more lightweight version of that.
168 00:17:32.280 ⇒ 00:17:45.110 Casie Aviles: That just focuses on classification, and that performed a lot better, so it was around, around 5 seconds when it’s not, like, related to the text-to-SQL querying.
169 00:17:45.450 ⇒ 00:17:49.880 Casie Aviles: But, yeah, like I also mentioned there that… The, the, the querying part.
170 00:17:50.300 ⇒ 00:17:55.540 Casie Aviles: Does take a little bit more time, so that’s another area that we can optimize for.
171 00:17:56.330 ⇒ 00:18:00.830 Pranav: Okay, which, Google Sheets should I look at to see, like, those times?
172 00:18:01.090 ⇒ 00:18:06.300 Uttam Kumaran: Why… is this not in Snowflake? Or is this not in Reel already? Like, why are we doing this in…
173 00:18:08.610 ⇒ 00:18:09.230 Samuel Roberts: The monster stuff?
174 00:18:10.160 ⇒ 00:18:11.040 Uttam Kumaran: Yeah, like…
175 00:18:11.040 ⇒ 00:18:12.250 Pranav: The…
176 00:18:12.390 ⇒ 00:18:13.520 Uttam Kumaran: Runtimes.
177 00:18:16.540 ⇒ 00:18:18.929 Samuel Roberts: I mean, I think… Go ahead, Casey. Yeah, sorry.
178 00:18:18.930 ⇒ 00:18:27.049 Casie Aviles: Yeah, I was… I was getting, like, execution times from Snowflake, and I was… I have, like, a test script that will run.
179 00:18:27.680 ⇒ 00:18:28.370 Uttam Kumaran: I see.
180 00:18:28.620 ⇒ 00:18:31.840 Casie Aviles: And it will log, yeah, so that’s how it’s working.
181 00:18:34.380 ⇒ 00:18:43.840 Pranav: So for that, like, architecture that was using… that you said that was, like, comparable… comparable to N8N, which Google Sheet is that?
182 00:18:43.950 ⇒ 00:18:48.429 Pranav: And, like, which, like, sheet number or sheet title is that?
183 00:18:48.860 ⇒ 00:18:55.349 Casie Aviles: Yeah, that was in the ABC Spreadsheet hub. It’s the three-way comparison sheet.
184 00:18:56.820 ⇒ 00:18:59.130 Pranav: Three-way comparison, okay, gotcha.
185 00:19:00.420 ⇒ 00:19:05.220 Pranav: Okay, so yeah, here there’s only, like… what is it, like, 15 rows?
186 00:19:06.280 ⇒ 00:19:08.189 Pranav: We only ran it on, like, 20 questions.
187 00:19:10.100 ⇒ 00:19:13.119 Pranav: Okay, yeah, so you probably need to run this on a lot more.
188 00:19:13.920 ⇒ 00:19:16.549 Casie Aviles: Yeah, bro, okay. There are also other, like.
189 00:19:16.720 ⇒ 00:19:25.800 Casie Aviles: tests I added here, where I checked if, you know, if the master agent could improve on, like, the longest execution times that we had.
190 00:19:26.270 ⇒ 00:19:31.390 Pranav: Right, yeah, I saw that, and that looked amazing. Like, it was way less, which was really good.
191 00:19:33.920 ⇒ 00:19:34.530 Casie Aviles: No.
192 00:19:35.190 ⇒ 00:19:36.350 Pranav: Yeah, so that was good.
193 00:19:36.350 ⇒ 00:19:37.709 Samuel Roberts: Yeah, that looks much better.
194 00:19:38.040 ⇒ 00:19:42.110 Pranav: Yeah, so for that 3-way comparison, let’s just probably… like…
195 00:19:42.370 ⇒ 00:19:54.020 Pranav: expand the dataset that we’re testing against, just so we can, like, more confidently say that Master’s performing better. Because, yeah, right now, I think it’s only, like, 13 records, so…
196 00:19:54.530 ⇒ 00:20:03.159 Samuel Roberts: Mustafa, do we have the… or Casey, I’m not sure who had looked at it, but the… feeding the current questions to the Mastra app as well?
197 00:20:05.220 ⇒ 00:20:08.129 Mustafa Raja: Yes, I think, Jesse did build that.
198 00:20:08.490 ⇒ 00:20:09.030 Samuel Roberts: Okay.
199 00:20:09.030 ⇒ 00:20:19.970 Mustafa Raja: This is the workflow where a question gets asked from Annette and Andy, and then it also gets fed into Master Andy to see how they’re comparing with each other right now.
200 00:20:24.350 ⇒ 00:20:26.640 Mustafa Raja: Yeah, yeah, so Casey built that. Yeah.
201 00:20:26.810 ⇒ 00:20:28.780 Samuel Roberts: Okay, so we should have some…
202 00:20:28.890 ⇒ 00:20:37.399 Samuel Roberts: data that’s been accruing then. Maybe it’s… maybe it needs to get tweaked, because, if Casey reworked the classification, but,
203 00:20:37.560 ⇒ 00:20:40.840 Samuel Roberts: There should be something there that we can at least see, like, the trends over the last…
204 00:20:40.970 ⇒ 00:20:44.730 Samuel Roberts: few days of live questions through Mastra.
205 00:20:45.640 ⇒ 00:20:47.379 Samuel Roberts: Not with feedback or anything, but…
206 00:20:48.070 ⇒ 00:20:50.320 Samuel Roberts: At least we could maybe see the execution times.
207 00:20:51.680 ⇒ 00:21:01.400 Pranav: Oh, so basically all of the questions that are being fed to, like, Andium production by the CSRs, like, that is now going through the QA. Okay, cool.
208 00:21:01.400 ⇒ 00:21:04.389 Samuel Roberts: Yeah, yeah, I wanted to have something to test the, like…
209 00:21:04.650 ⇒ 00:21:20.050 Samuel Roberts: master on a little… like, this… rather than, like, pull a dataset and run that, and then keep refining… I mean, which I think we should still do. I wanted something that was just gonna take their real questions and just feed it through, and at least see, like, execution times, and maybe be able to compare outputs over time, but I figured that was a good way to just…
210 00:21:20.310 ⇒ 00:21:22.990 Samuel Roberts: That’s a good one. Test it without them having to, like, test it.
211 00:21:23.410 ⇒ 00:21:24.549 Pranav: I like that, yeah.
212 00:21:24.690 ⇒ 00:21:26.219 Mustafa Raja: cases that are alive?
213 00:21:26.440 ⇒ 00:21:29.409 Mustafa Raja: a live CSV somewhere that we can take a look at?
214 00:21:30.430 ⇒ 00:21:34.149 Casie Aviles: Oh yeah, it’s in, it’s also in the spreadsheet, probably.
215 00:21:34.150 ⇒ 00:21:34.720 Samuel Roberts: Okay.
216 00:21:35.020 ⇒ 00:21:39.550 Casie Aviles: It’s a bit noisy at the moment, or yeah, it’s a bit messier now.
217 00:21:40.760 ⇒ 00:21:47.479 Casie Aviles: Because, when I attach it to N8, like, NAT’s not the best, like, at, like, you know…
218 00:21:47.770 ⇒ 00:21:48.240 Samuel Roberts: Yeah.
219 00:21:48.240 ⇒ 00:21:50.250 Casie Aviles: Being able to log everything, so…
220 00:21:51.250 ⇒ 00:21:54.020 Casie Aviles: This might not be, like, the most accurate.
221 00:21:54.020 ⇒ 00:21:54.920 Samuel Roberts: Was it.
222 00:21:57.350 ⇒ 00:21:58.329 Pranav: Oh, you’re muted.
223 00:21:59.470 ⇒ 00:22:04.480 Samuel Roberts: Damn it, okay. Sorry, I keep doing that when I move with the mic, and I will not get used to it for some reason.
224 00:22:05.170 ⇒ 00:22:10.590 Samuel Roberts: what was I saying? Oh, the master is getting logged. Is there a way to…
225 00:22:11.570 ⇒ 00:22:14.939 Samuel Roberts: pass the N8N stuff through as well? Maybe not, I don’t know.
226 00:22:17.840 ⇒ 00:22:28.110 Casie Aviles: Yeah, essentially what I did was I just created, you know, new testing scripts that will basically call the Mastra endpoint that we have on Google Cloud.
227 00:22:28.250 ⇒ 00:22:30.649 Samuel Roberts: Okay. And it’ll get, like, the output there.
228 00:22:30.860 ⇒ 00:22:34.999 Casie Aviles: But we’re… but what I’m comparing it against are historical…
229 00:22:35.560 ⇒ 00:22:39.310 Casie Aviles: logs from Snowflake, which is from the NA10 executions.
230 00:22:39.540 ⇒ 00:22:41.410 Samuel Roberts: Right, right, okay.
231 00:22:41.560 ⇒ 00:22:46.930 Casie Aviles: So the live one is… It may not look like it’s, you know, it’s…
232 00:22:47.120 ⇒ 00:22:56.779 Casie Aviles: Giving good results, because there were times here where the master app was still unoptimized, and that’s why it’s showing, like, really bad scores.
233 00:22:57.120 ⇒ 00:22:57.450 Samuel Roberts: Okay.
234 00:22:57.450 ⇒ 00:23:02.040 Casie Aviles: And that… that might be, like, might not be, like, the best way to see.
235 00:23:02.630 ⇒ 00:23:04.030 Casie Aviles: Okay. Unfortunately. Okay.
236 00:23:04.220 ⇒ 00:23:08.830 Samuel Roberts: Has… have we updated that live Moscow with the new classification and the optimizations?
237 00:23:10.970 ⇒ 00:23:23.480 Casie Aviles: Yeah, the one on Google Cloud should have… well, there’s, like, two deployments that I tested with, but yeah, I do have, like, the lightweight version there, and then the one that’s not lightweight.
238 00:23:23.480 ⇒ 00:23:24.280 Samuel Roberts: Okay.
239 00:23:24.980 ⇒ 00:23:26.430 Casie Aviles: Well, two deployments.
240 00:23:27.560 ⇒ 00:23:32.070 Samuel Roberts: Okay, so the lightweight one, is that being fed the live question still?
241 00:23:33.310 ⇒ 00:23:34.679 Casie Aviles: No, no, no.
242 00:23:35.100 ⇒ 00:23:38.909 Samuel Roberts: Okay, we may want to set that back up then, or set that up.
243 00:23:39.630 ⇒ 00:23:40.290 Casie Aviles: Okay.
244 00:23:40.730 ⇒ 00:23:42.759 Samuel Roberts: Maybe we can chat after this, just so I can…
245 00:23:43.110 ⇒ 00:23:46.000 Samuel Roberts: Understand, maybe, if there’s a good way to, like.
246 00:23:46.910 ⇒ 00:23:49.570 Samuel Roberts: I don’t know, I’m not sure. Okay.
247 00:23:51.640 ⇒ 00:23:53.840 Pranav: So we have two master apps up right now.
248 00:23:55.000 ⇒ 00:23:56.370 Casie Aviles: ES on Google.
249 00:23:56.780 ⇒ 00:23:57.470 Pranav: Gotcha.
250 00:23:59.920 ⇒ 00:24:05.499 Samuel Roberts: Do we need the old one anymore? Like, is the lightweight one… or are we still trying to, like, compare them?
251 00:24:07.660 ⇒ 00:24:12.420 Casie Aviles: It’s mainly just for comparison, but… Okay. I think… Okay.
252 00:24:12.700 ⇒ 00:24:18.309 Pranav: I’m happy to keep them up for now, so we can have the report, and then… I think what would be great is…
253 00:24:19.080 ⇒ 00:24:34.299 Pranav: probably, like, today in our working session, we can look at all this data that’s getting pulled into the Google Sheet. Casey, you said, like, it needs, like, some refining, so we can do that. And then we can just see, like, okay, what does the output look like, in terms of execution time?
254 00:24:34.870 ⇒ 00:24:36.120 Pranav: And…
255 00:24:36.370 ⇒ 00:24:43.310 Pranav: on Thursday or Friday, I’d like to just, like, send them over a report of, like, okay.
256 00:24:43.450 ⇒ 00:24:48.229 Pranav: this is, like, the updates that we’re seeing. These aren’t… obviously not in production yet.
257 00:24:48.990 ⇒ 00:24:52.629 Pranav: But, like, preliminary, like, tests are looking really good.
258 00:24:54.650 ⇒ 00:24:55.300 Casie Aviles: Okay.
259 00:24:57.070 ⇒ 00:25:02.460 Samuel Roberts: And then, the other thing is, Casey, you’re out starting tomorrow, right?
260 00:25:03.390 ⇒ 00:25:04.180 Casie Aviles: Yes.
261 00:25:04.600 ⇒ 00:25:07.029 Casie Aviles: So we want to make sure that… I’ll wrap up with everything, yeah.
262 00:25:07.030 ⇒ 00:25:12.010 Samuel Roberts: Okay, yeah, I mean, either wrap up, or make sure we can pass off, or I can, you know, one of us can grab something.
263 00:25:12.010 ⇒ 00:25:14.950 Casie Aviles: If it’s needed. But just make sure we have a plan for that.
264 00:25:15.340 ⇒ 00:25:18.329 Samuel Roberts: Through the working session, at least. Or at the end of the working session, probably.
265 00:25:18.720 ⇒ 00:25:19.360 Pranav: Yeah.
266 00:25:19.590 ⇒ 00:25:20.210 Samuel Roberts: Okay.
267 00:25:21.880 ⇒ 00:25:23.640 Samuel Roberts: Cool, anything else?
268 00:25:24.510 ⇒ 00:25:25.810 Samuel Roberts: Before that, then?
269 00:25:30.670 ⇒ 00:25:33.339 Samuel Roberts: Alright, we’ll be good then.
270 00:25:35.300 ⇒ 00:25:43.340 Samuel Roberts: I think, on ABC them were good. So Utam had to drop a, he was mentioning some stuff about the open work stuff, Mustafa.
271 00:25:44.130 ⇒ 00:25:45.949 Mustafa Raja: Yeah, are you familiar with it?
272 00:25:46.380 ⇒ 00:25:57.939 Samuel Roberts: I know about… I know of it, I haven’t really dug into it at all, and that’s what… I had asked him about that on the previous call, and he said, like, we would discuss it here, but he had to bounce, so I… I just was curious what you guys talked about last night.
273 00:25:58.610 ⇒ 00:26:00.760 Mustafa Raja: Yeah, so,
274 00:26:02.130 ⇒ 00:26:25.910 Mustafa Raja: So we are, we are trying to build a new, new app, and that, that, we would want to, you know, ship to our, clients also. What it really is, is just, open claw, but with, a lot more security, right? So it would have… this app would have, all of the access in our.
275 00:26:26.000 ⇒ 00:26:37.330 Mustafa Raja: OS, right? But, so the version that, Clarence came up with, it’s, it’s a desktop application.
276 00:26:37.850 ⇒ 00:26:46.800 Mustafa Raja: really wants a, a web application, but what that is, that would not have a… that would… of course, that would not have
277 00:26:47.300 ⇒ 00:26:52.499 Mustafa Raja: access to our OS, right? Because, you know, it wouldn’t be able to get out of our…
278 00:26:53.280 ⇒ 00:27:02.070 Mustafa Raja: browser. So, what we will have to do, if we want to do that, we will have to stim down some features that are related to OS.
279 00:27:02.180 ⇒ 00:27:05.930 Mustafa Raja: I’ll create… create some… something up, and then…
280 00:27:06.690 ⇒ 00:27:24.820 Mustafa Raja: So Utham… Utham is open to either… either looking to a desktop app or a web app, but, you know, we need to, you know, come up with, what we will be losing if we do a web app. I see.
281 00:27:24.970 ⇒ 00:27:27.669 Samuel Roberts: Okay, yeah, like, what the browser restrictions are, okay.
282 00:27:28.450 ⇒ 00:27:29.720 Mustafa Raja: Yeah.
283 00:27:29.980 ⇒ 00:27:38.409 Mustafa Raja: And… That’s pretty much it. I see that, the open work, the new open work.
284 00:27:39.170 ⇒ 00:27:43.660 Mustafa Raja: repo that the OpenWork has. They now have a web,
285 00:27:44.070 ⇒ 00:27:49.780 Mustafa Raja: web distribution. The one that, Clarence has forked does not have that.
286 00:27:52.070 ⇒ 00:27:53.310 Samuel Roberts: Oh, okay.
287 00:27:53.570 ⇒ 00:27:56.330 Mustafa Raja: Yeah, so we might want to also take a look at that.
288 00:27:57.170 ⇒ 00:27:57.550 Samuel Roberts: Okay.
289 00:27:57.550 ⇒ 00:27:59.279 Mustafa Raja: That’s pretty much it.
290 00:27:59.470 ⇒ 00:28:03.119 Samuel Roberts: Alright, cool, yeah, I’ll try to look at that a little bit, maybe we can talk about it on Slack and…
291 00:28:03.660 ⇒ 00:28:07.090 Samuel Roberts: Go from there, okay.
292 00:28:07.340 ⇒ 00:28:09.699 Samuel Roberts: Thank you for that context.
293 00:28:11.820 ⇒ 00:28:16.289 Samuel Roberts: I think that’s it, then. Anyone else have anything they want to bring up? We all good?
294 00:28:17.360 ⇒ 00:28:17.920 Samuel Roberts: Alright.
295 00:28:17.920 ⇒ 00:28:19.170 Brylle Girang: All good. Thank you.
296 00:28:19.170 ⇒ 00:28:26.570 Samuel Roberts: I’ll see you guys later. I have to run some errands today, so I may be not the keyboard, but I’ll have my phone if you need anything, so I’ll see Slack, so…
297 00:28:27.190 ⇒ 00:28:28.000 Samuel Roberts: Alright.
298 00:28:28.490 ⇒ 00:28:29.680 Samuel Roberts: Catch y’all later.
299 00:28:29.680 ⇒ 00:28:30.520 Pranav: Thanks, guys.