Meeting Title: LangFuse Architecture Review Kickoff Date: 2025-10-27 Meeting participants: Awaish Kumar, Mustafa Raja, Casie Aviles, Samuel Roberts, Uttam Kumaran, Demilade Agboola
WEBVTT
1 00:02:22.940 ⇒ 00:02:23.320 Mustafa Raja: Hey.
2 00:02:31.960 ⇒ 00:02:32.570 Awaish Kumar: Hello.
3 00:02:33.610 ⇒ 00:02:34.460 Mustafa Raja: Oh, how are you?
4 00:02:36.310 ⇒ 00:02:37.879 Awaish Kumar: I’m good, how about you?
5 00:02:38.310 ⇒ 00:02:39.810 Mustafa Raja: Yeah, yeah, doing good, doing good.
6 00:02:40.090 ⇒ 00:02:42.079 Mustafa Raja: How’s the weather in Karachi?
7 00:02:43.120 ⇒ 00:02:44.353 Awaish Kumar: Very slow.
8 00:02:45.970 ⇒ 00:02:51.180 Mustafa Raja: Nice. It’s getting a little bit… a little bit colder here, but… yeah.
9 00:02:51.180 ⇒ 00:02:53.360 Awaish Kumar: No, AQI is very bad.
10 00:02:54.380 ⇒ 00:02:54.910 Mustafa Raja: stop.
11 00:02:56.310 ⇒ 00:02:57.849 Awaish Kumar: 190 plus.
12 00:02:59.250 ⇒ 00:03:09.939 Mustafa Raja: Oh, I think Lahore is even worse, worse than 119, I’m not sure, though. I thought I saw somewhere that it’s about 400 or something.
13 00:03:10.700 ⇒ 00:03:11.800 Mustafa Raja: For a hole.
14 00:03:12.360 ⇒ 00:03:13.719 Mustafa Raja: And I can see it.
15 00:03:14.420 ⇒ 00:03:17.939 Awaish Kumar: Yeah, one night is… I have lived in Lajo, so I know what…
16 00:03:18.370 ⇒ 00:03:21.169 Awaish Kumar: how bad it is, but in the Kathy, it’s also very bad, right?
17 00:03:21.170 ⇒ 00:03:22.879 Mustafa Raja: Oh, okay, okay.
18 00:03:28.170 ⇒ 00:03:29.260 Samuel Roberts: You know, I’ve…
19 00:03:35.800 ⇒ 00:03:36.839 Samuel Roberts: You’re Monday.
20 00:04:10.460 ⇒ 00:04:17.319 Mustafa Raja: So, for LangFuse, can we do the same, tracing thing for, Copilot, Sam?
21 00:04:18.050 ⇒ 00:04:21.509 Samuel Roberts: Sort of. It’s not,
22 00:04:22.580 ⇒ 00:04:35.420 Samuel Roberts: Yeah, so Copilot does have some things you can tap into, but I ran into some really weird things with it, but I think what we’re really gonna end up doing is putting master agents in place of just the raw LLM calls.
23 00:04:35.750 ⇒ 00:04:36.890 Mustafa Raja: Oh…
24 00:04:36.890 ⇒ 00:04:45.759 Samuel Roberts: And then we’ll be able to use Mastra to do it, because, like, Copilot Kit, when I… even when I emailed them a little bit, they were like, well, we’re not really, like, the… you know, we have some…
25 00:04:46.110 ⇒ 00:04:52.799 Samuel Roberts: some observability hooks, but it’s not the same kind of base as just, like, calling the LLMs like Monstra does.
26 00:04:53.160 ⇒ 00:04:58.290 Samuel Roberts: So I think we’ll be able to, as we start to replace those raw LN or N8N calls.
27 00:04:58.780 ⇒ 00:05:01.220 Samuel Roberts: Then we can do that, and it’ll help a lot more.
28 00:05:01.500 ⇒ 00:05:02.360 Mustafa Raja: Yeah, yeah.
29 00:05:04.340 ⇒ 00:05:09.599 Mustafa Raja: I was thinking… I was thinking the same thing, because it’s now tempted to doing it in, like.
30 00:05:11.300 ⇒ 00:05:18.719 Samuel Roberts: Yeah, yeah, that’s why, like, look, the stuff we were seeing in LangFuse that you added is… is better than what I was seeing from Copilot Kit, so…
31 00:05:19.030 ⇒ 00:05:19.570 Mustafa Raja: Yep.
32 00:05:20.580 ⇒ 00:05:21.450 Uttam Kumaran: Hey guys.
33 00:05:21.970 ⇒ 00:05:22.930 Samuel Roberts: Hello.
34 00:05:22.930 ⇒ 00:05:26.349 Uttam Kumaran: I can tell I’m on an absolute coffee binge today.
35 00:05:27.790 ⇒ 00:05:38.560 Uttam Kumaran: We’ve had some great meetings today. I thought the initial AI meeting was good, delivery meeting was good. Yeah. I’m excited for this meeting. This is our first in a series of…
36 00:05:38.960 ⇒ 00:05:41.810 Uttam Kumaran: Hopefully many future architecture reviews.
37 00:05:42.040 ⇒ 00:05:48.149 Uttam Kumaran: So I’m excited, is Demelade in a different call?
38 00:05:48.710 ⇒ 00:05:52.140 Uttam Kumaran: Or is he in this one? I don’t know if he was gonna attend.
39 00:05:53.050 ⇒ 00:05:58.049 Uttam Kumaran: But if he… If he can, maybe,
40 00:05:58.970 ⇒ 00:06:01.649 Uttam Kumaran: Oh, Ace, do you want to message him, or maybe I’ll ask him.
41 00:06:02.450 ⇒ 00:06:03.870 Awaish Kumar: I can’t…
42 00:06:16.610 ⇒ 00:06:20.320 Uttam Kumaran: Okay, so kind of, I just want to run through, like.
43 00:06:20.990 ⇒ 00:06:26.600 Uttam Kumaran: how I’ve done architecture reviews in the past, and sort of getting, you know,
44 00:06:26.790 ⇒ 00:06:34.859 Uttam Kumaran: feedback on how we should run these, but ideally, we’re running a couple this week. We’re running this one, we’re gonna run one on Insomnia.
45 00:06:35.130 ⇒ 00:06:53.279 Uttam Kumaran: And then I… ideally, like, at the end of this week or next week, I kind of want to start handing some of these meetings off to… to a few of our… you guys, like Sam, Awash, Demolade, to start to run. But I think what you’ll see is sort of, like, how I typically think about these.
46 00:06:53.460 ⇒ 00:07:10.649 Uttam Kumaran: And so roughly, like, the goal of this meeting is to get a full understanding end-to-end of the system that we’re working on, and then be able to poke and identify, like, the issues. So I guess my first question is, Casey, do we have, like, any, existing ABC
47 00:07:11.000 ⇒ 00:07:12.350 Uttam Kumaran: diagrams.
48 00:07:12.530 ⇒ 00:07:18.720 Uttam Kumaran: That we can start. So, first thing I want to do in this meeting is just get a… get, like, a audit of everything that we…
49 00:07:19.340 ⇒ 00:07:20.639 Uttam Kumaran: You know, we have…
50 00:07:21.280 ⇒ 00:07:25.299 Casie Aviles: Sure, we have an existing one, I can share it now.
51 00:07:25.640 ⇒ 00:07:26.270 Uttam Kumaran: Okay.
52 00:07:27.830 ⇒ 00:07:33.239 Casie Aviles: Although this one is… This is outdated now, since we’ve added a few things.
53 00:07:35.640 ⇒ 00:07:43.149 Casie Aviles: In the future, so I just quickly prepared additional… an additional diagram here to also talk about.
54 00:07:44.390 ⇒ 00:07:47.730 Casie Aviles: Like, what was missing, so…
55 00:07:48.660 ⇒ 00:07:50.619 Uttam Kumaran: Would you like to send this in the chat?
56 00:07:52.090 ⇒ 00:07:52.620 Casie Aviles: Yes.
57 00:07:52.620 ⇒ 00:08:11.810 Uttam Kumaran: And then, yeah, I guess, for all… for each of these architecture reviews, I think over time, we will end up batching these, like, we’ll have some day, like Tuesday, where we… we have an architecture review block, and anyone can come with architecture, but I do want to have case… I do want to have Awash and Demolata here. You guys, of course, don’t need to…
58 00:08:12.240 ⇒ 00:08:28.180 Uttam Kumaran: be super active, but I do want to have all of our senior folks looking and poking and asking questions. I’m really excited to have both of you guys here, because this is a new client that you guys haven’t seen, but I think it’s kind of really cool work, and I want to get your guys’ brain involved, so that’s sort of my goal of…
59 00:08:28.420 ⇒ 00:08:36.050 Uttam Kumaran: why I’m looping you guys in here. So, yeah, if everybody can open up that Figma, we can start there.
60 00:08:48.600 ⇒ 00:08:52.209 Casie Aviles: Okay, should I go through, like…
61 00:08:53.000 ⇒ 00:08:56.740 Casie Aviles: Just a brief overview on what all of this is.
62 00:08:57.190 ⇒ 00:09:03.849 Uttam Kumaran: Yeah, please, and even before, if you can just do a short demo of the tool for Demolade and Awash.
63 00:09:04.110 ⇒ 00:09:07.349 Demilade Agboola: Starting her up, but I had to request for access, so I just did…
64 00:09:07.700 ⇒ 00:09:09.080 Casie Aviles: Oh, okay.
65 00:09:09.420 ⇒ 00:09:10.080 Casie Aviles: Fair.
66 00:09:10.980 ⇒ 00:09:12.179 Casie Aviles: Okay, I approve now.
67 00:09:14.910 ⇒ 00:09:22.230 Uttam Kumaran: Yeah, and if you can even do, like, just a 2-minute demo of, like, the problem we’re solving for them, and, like, what the existing tool looks like, and then we can go into it.
68 00:09:23.020 ⇒ 00:09:27.350 Casie Aviles: Yeah, sure, oops.
69 00:09:27.560 ⇒ 00:09:28.530 Casie Aviles: baseless.
70 00:09:29.210 ⇒ 00:09:31.340 Casie Aviles: Should be this one… okay.
71 00:09:31.970 ⇒ 00:09:35.230 Casie Aviles: So, yeah, we have our…
72 00:09:35.820 ⇒ 00:09:38.030 Uttam Kumaran: So, basically, we built this,
73 00:09:38.030 ⇒ 00:09:42.090 Casie Aviles: AI chatbot for… Abc and…
74 00:09:42.770 ⇒ 00:09:51.370 Casie Aviles: Some of the… as you can see here, we have these questions. So this bot is supposed to help their customer service representatives.
75 00:09:52.710 ⇒ 00:09:58.979 Casie Aviles: It… So they have, like, a data source, they have different… Policies, guidelines…
76 00:09:59.150 ⇒ 00:10:03.809 Casie Aviles: And assignment data for their… for their employees, and…
77 00:10:04.410 ⇒ 00:10:14.399 Casie Aviles: That all of that is kind of scattered throughout different documents, so what we’re trying to do here is we’re trying to consolidate it such that it can be
78 00:10:14.570 ⇒ 00:10:20.470 Casie Aviles: So, you know, like, an AI could be asked about those, so we’re feeding all those contexts
79 00:10:20.840 ⇒ 00:10:24.600 Casie Aviles: to this AI bot, and we’ve implemented different
80 00:10:24.860 ⇒ 00:10:30.970 Casie Aviles: kinds of techniques to do that. So, here are just some examples of questions that we have.
81 00:10:32.500 ⇒ 00:10:36.349 Casie Aviles: And… Yeah, for example, we have questions about the zips.
82 00:10:39.210 ⇒ 00:10:45.550 Casie Aviles: And we have questions about costs and policy, so that’s how Andy works.
83 00:10:46.670 ⇒ 00:10:50.310 Casie Aviles: And… yeah, pretty much that’s… that’s how it works, so…
84 00:10:50.410 ⇒ 00:10:55.079 Casie Aviles: We have, so we’ve integrated it to the Google Chat.
85 00:10:55.760 ⇒ 00:11:00.009 Casie Aviles: Interface, because that’s where they are… that’s what they’re using.
86 00:11:01.770 ⇒ 00:11:08.899 Casie Aviles: And then, yeah, I think, yeah, that’s pretty much it for, like, the high-level, overview of Andy.
87 00:11:10.790 ⇒ 00:11:17.490 Uttam Kumaran: So, Demolade Awash, you guys… I don’t know Wase, you may be a… you’re familiar with this, but… Demolati, does that roughly make sense?
88 00:11:18.890 ⇒ 00:11:19.910 Demilade Agboola: Yeah, it does.
89 00:11:20.510 ⇒ 00:11:27.219 Uttam Kumaran: It’s just like a… it’s just a very sophisticated chatbot over a ton of documents that their customer service reps are using.
90 00:11:27.550 ⇒ 00:11:30.050 Uttam Kumaran: When they’re on the phone with clients.
91 00:11:32.480 ⇒ 00:11:33.780 Casie Aviles: Okay, that makes sense.
92 00:11:37.610 ⇒ 00:11:38.890 Casie Aviles: Yeah, so…
93 00:11:39.420 ⇒ 00:11:47.030 Casie Aviles: tools to just, give you an overview of this architecture diagram. So, we have Andy basically in
94 00:11:47.130 ⇒ 00:11:54.649 Casie Aviles: living in NATON, so that’s, like, the tool that we have that allows us to build, like.
95 00:11:56.250 ⇒ 00:12:00.000 Casie Aviles: automations, workflows, and then also AI.
96 00:12:00.740 ⇒ 00:12:04.079 Casie Aviles: We’re AI agents, so that’s what we use.
97 00:12:04.990 ⇒ 00:12:13.200 Casie Aviles: Right now, the data that it’s getting is from… is across different sources, multiple sources, so we have several central documents
98 00:12:13.380 ⇒ 00:12:18.470 Casie Aviles: That are on Google. So, for example, this is one such document that we have.
99 00:12:18.940 ⇒ 00:12:26.069 Casie Aviles: And then they have different departments, so they have… We started with… Just the pest department.
100 00:12:26.470 ⇒ 00:12:29.869 Casie Aviles: This is their… this is kind of, like, the document that they have.
101 00:12:30.280 ⇒ 00:12:38.280 Casie Aviles: So we’ve also… Kind of help… help them structure all of this to make it as… easily…
102 00:12:38.670 ⇒ 00:12:40.800 Casie Aviles: accessible to Andy.
103 00:12:41.490 ⇒ 00:12:44.880 Awaish Kumar: Yeah, so… Are you using Superbase?
104 00:12:47.300 ⇒ 00:12:50.700 Casie Aviles: So, you’re asking if we’re doing… if this is in Supabase?
105 00:12:51.450 ⇒ 00:12:57.039 Awaish Kumar: I’m asking, like, Google Docs, Google Sheets, where you have Arrow, and it is going to N at N.
106 00:12:57.280 ⇒ 00:13:01.150 Awaish Kumar: So it’s, like, is the storage is, the superbase, or…
107 00:13:01.750 ⇒ 00:13:09.159 Casie Aviles: Yeah, so what’s happening is… so before, we used to do… we used to just feed the entire document
108 00:13:09.360 ⇒ 00:13:10.780 Casie Aviles: to the context.
109 00:13:10.960 ⇒ 00:13:18.520 Casie Aviles: And we’ve had some issues with that. So now, what we were… what we implemented is vectorization, or…
110 00:13:18.670 ⇒ 00:13:22.660 Casie Aviles: rug, which is… A way to basically…
111 00:13:23.910 ⇒ 00:13:29.150 Casie Aviles: We’re basically getting all of these texts, and we’re transforming them into numbers.
112 00:13:29.260 ⇒ 00:13:37.429 Casie Aviles: That can be… queried by the AI. So that’s why there… that’s why there’s this step here.
113 00:13:37.750 ⇒ 00:13:43.620 Casie Aviles: So whenever the central document is updated, the workflow is triggered.
114 00:13:43.910 ⇒ 00:13:47.850 Casie Aviles: And then it’s going to be converted into the vector database.
115 00:13:48.200 ⇒ 00:13:54.080 Casie Aviles: Which will be… which will be how Andy queries, you know, the data.
116 00:13:54.660 ⇒ 00:13:58.420 Casie Aviles: So… For example, some of these questions, they are in…