Meeting Title: Interlude Project Case Study Interview Date: 2025-09-05 Meeting participants: Mustafa Raja, Hannah Wang
WEBVTT
1 00:00:08.900 ⇒ 00:00:10.020 Hannah Wang: Whoa.
2 00:00:10.580 ⇒ 00:00:11.940 Mustafa Raja: Hey, how about you?
3 00:00:12.620 ⇒ 00:00:13.370 Hannah Wang: Good.
4 00:00:14.370 ⇒ 00:00:15.349 Hannah Wang: How are you?
5 00:00:15.630 ⇒ 00:00:16.750 Mustafa Raja: Yeah, doing good.
6 00:00:18.020 ⇒ 00:00:23.669 Hannah Wang: I’m still thinking about your answer for the, would you rather? I thought it was hilarious.
7 00:00:24.740 ⇒ 00:00:25.550 Mustafa Raja: Yeah.
8 00:00:25.550 ⇒ 00:00:29.420 Hannah Wang: I’m also introverted too, so… Yeah.
9 00:00:30.540 ⇒ 00:00:31.200 Mustafa Raja: Yeah.
10 00:00:31.410 ⇒ 00:00:41.030 Mustafa Raja: So in university, I have, I had, very, very few friends, like, 6 or 7, and it was because, yeah, I didn’t really…
11 00:00:41.430 ⇒ 00:00:45.540 Mustafa Raja: I didn’t really know how to navigate through people, so yeah.
12 00:00:47.180 ⇒ 00:00:50.150 Hannah Wang: Yeah, I mean, I feel like most people…
13 00:00:51.890 ⇒ 00:00:59.810 Hannah Wang: most people gen… like, generally only have, like, a handful of close friends, like, same for me, too. I don’t…
14 00:01:00.660 ⇒ 00:01:04.870 Hannah Wang: It’s too tiring to maintain all those relationships.
15 00:01:04.879 ⇒ 00:01:08.009 Mustafa Raja: Yeah, but the friends that I have, I’m really close to them.
16 00:01:08.430 ⇒ 00:01:09.389 Hannah Wang: Yeah, exactly.
17 00:01:09.390 ⇒ 00:01:10.990 Mustafa Raja: It’s like a safe space for me.
18 00:01:11.310 ⇒ 00:01:12.840 Hannah Wang: Yeah, I agree.
19 00:01:12.960 ⇒ 00:01:21.559 Hannah Wang: Yeah. Okay, so… I know… well, I forgot if we ever did a…
20 00:01:22.260 ⇒ 00:01:42.390 Hannah Wang: format like this, where I kind of interview you for the case study, because I know in the past you recorded a loom, but basically I’m just going to walk through a bunch of questions and ask you questions, even if the answer’s obvious. Feel free to answer it, because what I do is I take the transcript and I, generate copy.
21 00:01:42.390 ⇒ 00:01:44.949 Hannah Wang: Or, like, content for that, so…
22 00:01:44.950 ⇒ 00:01:47.380 Hannah Wang: Yeah, I know our case study…
23 00:01:47.430 ⇒ 00:01:56.789 Hannah Wang: I know Utah wanted something about, evals, right? That was what we’re gonna talk about today.
24 00:01:58.180 ⇒ 00:02:06.289 Hannah Wang: Or AI Adoption and eval, yeah, eval-focused, and then Sam mentioned something about Brain Trust?
25 00:02:07.280 ⇒ 00:02:09.180 Hannah Wang: Sorry, I don’t really know what these…
26 00:02:09.180 ⇒ 00:02:09.560 Mustafa Raja: Yeah, yeah, yeah.
27 00:02:09.560 ⇒ 00:02:13.299 Hannah Wang: are, so bear with me, but if there’s any, like.
28 00:02:13.890 ⇒ 00:02:32.000 Hannah Wang: Ideally, the case study would be, like, an actual case study that we did for a client or internal work, not just, like, oh, this is what Brain Trust is, and this is how we use it. Like, that’s a little bit too general, so if you could think about, like, a…
29 00:02:32.660 ⇒ 00:02:34.799 Hannah Wang: Client work, or internal work
30 00:02:35.380 ⇒ 00:02:50.299 Hannah Wang: you did before that heavily involved either brain trust or interlude. Ideally, you’d, like, talk about that. So feel free to share your screen, and kind of walk through things. So do you have something in mind? Can I start?
31 00:02:50.300 ⇒ 00:02:50.710 Mustafa Raja: Yes.
32 00:02:50.710 ⇒ 00:02:52.519 Hannah Wang: new questions? Okay, cool.
33 00:02:52.580 ⇒ 00:02:54.179 Mustafa Raja: Give me a moment.
34 00:02:55.150 ⇒ 00:02:55.790 Hannah Wang: Sure.
35 00:02:56.230 ⇒ 00:02:58.299 Mustafa Raja: Let me share my screens.
36 00:03:00.630 ⇒ 00:03:07.480 Mustafa Raja: So, we did evals, but they never really got to production, to be honest. Okay.
37 00:03:08.680 ⇒ 00:03:10.900 Mustafa Raja: This place is just… oh, signing.
38 00:03:12.280 ⇒ 00:03:13.760 Mustafa Raja: Yeah.
39 00:03:15.630 ⇒ 00:03:22.920 Mustafa Raja: Let me walk you through what we did create. Do you have… do you have, idea of what eval’s, would be?
40 00:03:24.060 ⇒ 00:03:30.739 Hannah Wang: No, so just explain to me, because I don’t really know anything. Okay, okay. Just, like, explain the background of what evals are.
41 00:03:30.740 ⇒ 00:03:34.659 Mustafa Raja: Alright, so let’s look into our data sets, too, then.
42 00:03:34.930 ⇒ 00:03:38.440 Mustafa Raja: I think this… this would explain a lot of things.
43 00:03:40.630 ⇒ 00:03:41.340 Mustafa Raja: Hmm.
44 00:03:42.650 ⇒ 00:03:43.770 Mustafa Raja: So…
45 00:03:46.330 ⇒ 00:03:58.859 Mustafa Raja: Yeah, sure. So, let’s first understand what Interlude wants. So, what Interlude, does is they create, decks for their clients, right?
46 00:03:59.160 ⇒ 00:04:24.089 Mustafa Raja: And for that, for those decks to, to be created, they need two things. One is the questionnaire, and the other one is the meeting with the client, which we have the transcript for, and this questionnaire. So they have a set of questions they have, which they put forward to their clients, and their clients would answer those questions, right?
47 00:04:24.160 ⇒ 00:04:42.800 Mustafa Raja: And then they would have an actual meeting with their clients. So these are the two, two things that would, that we would consider as, input to our workflow, right? Okay. And then there’s this expected output thing. Sorry. Sorry. Sorry.
48 00:04:43.100 ⇒ 00:04:47.549 Mustafa Raja: It’s okay. Okay, let’s move to the… yeah.
49 00:04:49.140 ⇒ 00:05:03.759 Mustafa Raja: This, yeah. So, so what this expected output is, the, the output that interlude people generated manually. Meaning, so before, before we came in, this whole process of tech creation was manually done.
50 00:05:03.770 ⇒ 00:05:13.789 Mustafa Raja: By people, right? And for this set of… this set of input, that we have right here, this transcript and questionnaire, they created this deck.
51 00:05:14.920 ⇒ 00:05:15.400 Hannah Wang: I see.
52 00:05:15.400 ⇒ 00:05:16.169 Mustafa Raja: this deck.
53 00:05:16.170 ⇒ 00:05:39.720 Mustafa Raja: Yeah, and what evals do is, they, they run on, the same set of inputs, that, Interlude generated this output for, and, they compare the output of, the AI… sorry, they compare the, AI-generated output with the output that.
54 00:05:39.760 ⇒ 00:05:48.040 Mustafa Raja: interlude people gave us over here. And this, the evals would tell us how close are we to the actual thing.
55 00:05:48.370 ⇒ 00:05:49.510 Hannah Wang: I see.
56 00:05:49.510 ⇒ 00:05:52.540 Mustafa Raja: Yeah, so we have a dataset over here.
57 00:05:54.600 ⇒ 00:06:00.929 Mustafa Raja: We have a dataset over here, it’s only 3 rows because, this is, this is what…
58 00:06:00.930 ⇒ 00:06:16.889 Mustafa Raja: we received from Interdude. But yeah. So, as we iterate on our AI architecture, as we improve it, we keep running it on this input to see how close are we to the actual thing.
59 00:06:17.790 ⇒ 00:06:18.710 Hannah Wang: I see.
60 00:06:18.710 ⇒ 00:06:30.370 Mustafa Raja: This would be something that, interlude people would be expecting, and this is a metric that, can help us decide, okay, we are good to go.
61 00:06:31.630 ⇒ 00:06:35.590 Hannah Wang: Is there, like, a number, or, like, a… Okay.
62 00:06:37.360 ⇒ 00:06:45.420 Mustafa Raja: This, to be honest, it’s not going to be super realistic, because it never got to, production, but I can…
63 00:06:45.930 ⇒ 00:06:48.310 Mustafa Raja: I can show you what it would look like.
64 00:06:48.790 ⇒ 00:06:49.510 Hannah Wang: Okay.
65 00:06:52.010 ⇒ 00:07:01.660 Mustafa Raja: Let’s go to some other one… Okay, this is back to… Yes.
66 00:07:02.750 ⇒ 00:07:06.500 Mustafa Raja: So, so what… let’s open this up.
67 00:07:07.720 ⇒ 00:07:12.780 Mustafa Raja: Yeah, so what this, levenstein is.
68 00:07:12.890 ⇒ 00:07:21.109 Mustafa Raja: Is the… what’s it called? General, semantic similarity between two strings.
69 00:07:21.110 ⇒ 00:07:21.490 Hannah Wang: Hmm.
70 00:07:21.490 ⇒ 00:07:25.599 Mustafa Raja: So, here we have the… what’s it called? Expected thing? Expected output.
71 00:07:25.740 ⇒ 00:07:36.590 Mustafa Raja: And here, we have the input, questionnaire, so-and-so, and then here’s somewhere we will have our, output, yeah. So this is our output.
72 00:07:36.590 ⇒ 00:08:00.840 Mustafa Raja: This is the output generated by our AI agent, and what this eval is doing is it comparing, semantically comparing this output with this expected output, and this expected output is the output generated by interloop people, and seeing how close they are to each other.
73 00:08:02.240 ⇒ 00:08:08.290 Mustafa Raja: And this is the percentage, but, but, I’d say over here that,
74 00:08:08.570 ⇒ 00:08:25.529 Mustafa Raja: We needed to define some more metrics, because it’s never going to be exactly the same thing, right? Yeah. It’s never going to be exactly the same words. What needed to be done is to compare the narrative flow, the, what’s it called?
75 00:08:25.570 ⇒ 00:08:40.339 Mustafa Raja: The formatting, and the structure. So these, these would be the set of, metrics that we would want to compare our thing on, rather than just comparing the strings, because the words are never going to be the same.
76 00:08:41.549 ⇒ 00:08:42.439 Hannah Wang: Got it.
77 00:08:42.639 ⇒ 00:08:43.689 Hannah Wang: Yeah. Okay.
78 00:08:43.690 ⇒ 00:08:47.389 Mustafa Raja: This would be an overview. Let me know if you have any questions.
79 00:08:48.090 ⇒ 00:08:52.590 Hannah Wang: Yeah, wait, so is… are we still… is Interlude still a client?
80 00:08:52.860 ⇒ 00:08:54.100 Mustafa Raja: Yeah. Yeah.
81 00:08:54.260 ⇒ 00:08:56.680 Hannah Wang: But we’re just working on something else with them.
82 00:08:56.800 ⇒ 00:09:08.189 Mustafa Raja: So, so this eval thing, what happened was that we, we had, other things, and, the AI agent was evolving.
83 00:09:08.410 ⇒ 00:09:12.669 Mustafa Raja: And it came to a point where we had a human in the loop.
84 00:09:12.880 ⇒ 00:09:14.090 Mustafa Raja: steps.
85 00:09:14.090 ⇒ 00:09:37.200 Mustafa Raja: Meaning, AI agent would chat with the people, and then there would be a point where we actually generate and save the output, and that would be the point we would want to generate evals on. At the point we added the human in the loop thing, this became outdated. The pipeline we created
86 00:09:37.200 ⇒ 00:09:40.669 Mustafa Raja: For the evals became outdated and needed to be updated.
87 00:09:41.490 ⇒ 00:09:49.430 Mustafa Raja: Accordingly, because this depended on the output of the AI agent itself, but we needed the output from human in the loop.
88 00:09:50.150 ⇒ 00:09:57.300 Mustafa Raja: For this thing to work. So this needs to be updated. This is going to work. This, we’re not dropping this.
89 00:09:59.000 ⇒ 00:10:00.619 Hannah Wang: It’s just, it needs to be updated.
90 00:10:00.620 ⇒ 00:10:01.290 Mustafa Raja: Yeah.
91 00:10:02.050 ⇒ 00:10:04.380 Hannah Wang: But we haven’t gotten around to that.
92 00:10:04.380 ⇒ 00:10:07.579 Mustafa Raja: We started Interlude this week.
93 00:10:07.580 ⇒ 00:10:08.980 Hannah Wang: Oh, okay.
94 00:10:08.980 ⇒ 00:10:26.410 Mustafa Raja: Actually, mid… mid this, this week, Utam had a meeting with them on Wednesday, when we agreed on an, sorry, didn’t agree, but renewed the, contract. So yeah, we’ll be, we’ll be starting, and, and yeah, this, this pipeline will be…
95 00:10:26.580 ⇒ 00:10:28.190 Mustafa Raja: Will be renewed.
96 00:10:28.870 ⇒ 00:10:35.500 Hannah Wang: I see. Okay, but that’ll probably take some time, right? Like, for everything to be built out and tested and everything?
97 00:10:35.500 ⇒ 00:10:40.799 Mustafa Raja: Yeah, maybe, I need to meet with the team, to discuss.
98 00:10:40.800 ⇒ 00:10:41.130 Hannah Wang: Okay.
99 00:10:41.130 ⇒ 00:10:43.940 Mustafa Raja: And see how it goes.
100 00:10:44.650 ⇒ 00:10:45.479 Hannah Wang: Okay, no worries.
101 00:10:45.480 ⇒ 00:10:48.180 Mustafa Raja: Do you need this to work? If so, I can prioritize.
102 00:10:48.180 ⇒ 00:10:48.629 Hannah Wang: What is this?
103 00:10:48.630 ⇒ 00:10:49.350 Mustafa Raja: I guess.
104 00:10:50.570 ⇒ 00:10:54.150 Hannah Wang: I think it’s okay, like, I think…
105 00:10:55.730 ⇒ 00:11:00.069 Hannah Wang: I mean, it’s ideal to have, like, results and impact and stuff, but…
106 00:11:00.070 ⇒ 00:11:00.540 Mustafa Raja: Hmm.
107 00:11:00.540 ⇒ 00:11:05.590 Hannah Wang: I guess for now, it’ll be okay, and then once…
108 00:11:05.800 ⇒ 00:11:17.680 Hannah Wang: you start work with Interlude and then kind of finish everything, then I can meet with you again, and then we can revise our case study. But I think just having something about evals
109 00:11:17.850 ⇒ 00:11:24.149 Hannah Wang: for now is good, so that we can send it out to leads, and then later on, we can kind of…
110 00:11:24.400 ⇒ 00:11:32.439 Hannah Wang: Make it more, like… concrete. So I think this is… this will be fine for now.
111 00:11:32.440 ⇒ 00:11:46.219 Mustafa Raja: What I think we need to do is we need to, adjust some of this dataset that we had to include the narrative flow, and those sort of things. Yeah.
112 00:11:46.890 ⇒ 00:11:48.709 Hannah Wang: Yeah, that’s okay. They can be up and running.
113 00:11:49.290 ⇒ 00:11:53.259 Mustafa Raja: This should be up and running soon.
114 00:11:53.860 ⇒ 00:11:55.870 Hannah Wang: Oh, okay. Yeah.
115 00:11:57.130 ⇒ 00:12:01.359 Hannah Wang: Okay, that’s okay. Let me… I’ll just still, like, ask you the questions.
116 00:12:01.360 ⇒ 00:12:01.909 Mustafa Raja: Yeah, yeah, yeah.
117 00:12:01.910 ⇒ 00:12:21.180 Hannah Wang: And then I can make the case study, and then if Lutam is like, okay, let’s wait, then we could wait. But, okay, so I guess first question, how long did this project take? It can be, like, very general, like, Q2 or Q3, or, like, two months, or September, or something like that.
118 00:12:21.180 ⇒ 00:12:24.360 Mustafa Raja: Okay, so this would be only for evals, right?
119 00:12:25.480 ⇒ 00:12:29.800 Hannah Wang: Yeah, the Interlude project With evals. Yeah, how long did that take?
120 00:12:29.800 ⇒ 00:12:32.529 Mustafa Raja: Okay, so it’s been, it’s been,
121 00:12:32.740 ⇒ 00:12:38.869 Mustafa Raja: It’s been actually, I guess, a month. It’s been a month since we’re working on… Interlude.
122 00:12:39.370 ⇒ 00:12:42.499 Hannah Wang: Okay. And then I guess you…
123 00:12:42.700 ⇒ 00:12:47.389 Hannah Wang: We’re the AI engineer. Who’s the PM for Interlude?
124 00:12:47.390 ⇒ 00:12:49.809 Mustafa Raja: Rico.
125 00:12:50.180 ⇒ 00:12:54.970 Hannah Wang: Rico? Okay. And then was there any other AI engineer working on this?
126 00:12:54.970 ⇒ 00:12:57.209 Mustafa Raja: Sam joined us.
127 00:12:57.680 ⇒ 00:12:59.480 Mustafa Raja: Okay. When you join?
128 00:12:59.480 ⇒ 00:13:03.690 Hannah Wang: Okay. So…
129 00:13:03.800 ⇒ 00:13:18.190 Hannah Wang: The next set of questions will be around context. So I guess for Interlude, like, why… do you know why they wanted this project in the first place? Like, why they wanted to automate
130 00:13:18.480 ⇒ 00:13:21.279 Hannah Wang: Their deck creation and, like.
131 00:13:21.280 ⇒ 00:13:22.090 Mustafa Raja: Yeah.
132 00:13:22.090 ⇒ 00:13:24.220 Hannah Wang: Yeah, I guess, yeah, to understand the context.
133 00:13:25.020 ⇒ 00:13:36.039 Mustafa Raja: Yeah, so, so this, yeah, this takes up a lot of time and context switching, for them to focus on, really meaningful things.
134 00:13:36.630 ⇒ 00:13:48.120 Mustafa Raja: Yeah, this really was the gist of it, but it took a lot of their time, and the CEO of Interlude would do it himself.
135 00:13:48.140 ⇒ 00:14:00.669 Mustafa Raja: Matthew would do it himself, and obviously, he had, some… a lot of other things on his plate, so, this was really a bottleneck for them.
136 00:14:01.820 ⇒ 00:14:07.509 Hannah Wang: So, previously, the Matthew was the one making the slides manually.
137 00:14:07.510 ⇒ 00:14:08.510 Mustafa Raja: Yeah, yeah.
138 00:14:08.900 ⇒ 00:14:12.299 Hannah Wang: Okay, and the goal was to automate everything, right?
139 00:14:12.300 ⇒ 00:14:16.579 Mustafa Raja: Yeah, I can pull up the, Claude chats. They have…
140 00:14:18.440 ⇒ 00:14:20.890 Mustafa Raja: If you want to take a look at that, too.
141 00:14:21.050 ⇒ 00:14:21.640 Hannah Wang: Yeah.
142 00:14:21.790 ⇒ 00:14:25.050 Mustafa Raja: That will be in motion.
143 00:14:42.280 ⇒ 00:14:43.810 Hannah Wang: Oh, okay.
144 00:14:49.410 ⇒ 00:15:01.050 Mustafa Raja: Yeah, so, we can go through this example. So we see that they have a system prompt, and they then feed, Claude the files. This would be the transcript, then…
145 00:15:01.900 ⇒ 00:15:19.599 Mustafa Raja: So this is the transcript, this would be the questionnaire, and some other supporting files, and then they would, start conditioning it to, first create the narrative flow, then the deck, then improve it, multiple iterations on it.
146 00:15:19.740 ⇒ 00:15:22.929 Mustafa Raja: So yeah, this was time-consuming for them.
147 00:15:23.580 ⇒ 00:15:28.840 Mustafa Raja: So that’s where we come in, where we automate all of this stuff for them, so they can work on
148 00:15:29.070 ⇒ 00:15:30.250 Mustafa Raja: Delivery.
149 00:15:31.400 ⇒ 00:15:34.090 Hannah Wang: So, are we automating the…
150 00:15:35.000 ⇒ 00:15:35.330 Mustafa Raja: Yeah.
151 00:15:35.330 ⇒ 00:15:43.949 Hannah Wang: Slide creation, or are we automating, like, the whole process of, like, talking with the client, to getting the transcript, to…
152 00:15:44.220 ⇒ 00:15:45.040 Hannah Wang: Like.
153 00:15:45.040 ⇒ 00:15:50.620 Mustafa Raja: We… Yeah, what exactly are we doing? We’re doing the, what’s it called? The deck.
154 00:15:52.330 ⇒ 00:15:54.189 Mustafa Raja: Only the deck, they still have to.
155 00:15:54.190 ⇒ 00:15:54.630 Hannah Wang: Oh, the.
156 00:15:54.630 ⇒ 00:16:06.340 Mustafa Raja: with their clients. Also, they’ll have to deliver the thing themselves. What it does is, what our AI agent does is it goes into the Notion,
157 00:16:06.750 ⇒ 00:16:08.879 Mustafa Raja: Database and stores it there for them.
158 00:16:10.110 ⇒ 00:16:10.830 Hannah Wang: I see.
159 00:16:10.830 ⇒ 00:16:15.539 Mustafa Raja: I can go to their, platform.
160 00:16:16.330 ⇒ 00:16:20.530 Mustafa Raja: And let’s… See, it’s been a few…
161 00:16:21.410 ⇒ 00:16:24.590 Mustafa Raja: This, since I’ve gone through this… Yeah, this one.
162 00:16:24.930 ⇒ 00:16:29.909 Mustafa Raja: Yeah, so you see that we have multiple examples? Let’s go through one of them.
163 00:16:30.100 ⇒ 00:16:31.130 Mustafa Raja: Yeah, see?
164 00:16:31.920 ⇒ 00:16:34.289 Mustafa Raja: We have the deck over here, so…
165 00:16:36.260 ⇒ 00:16:53.809 Mustafa Raja: key questions would be, what the investors might ask their client based on this, what, what’s it called, the deck, and the narrative flow, would be, why did the AI agent decided this flow of the deck?
166 00:16:54.550 ⇒ 00:16:57.530 Hannah Wang: Okay. Okay, got it. So we’re automating the deck.
167 00:16:57.530 ⇒ 00:16:59.910 Mustafa Raja: We have added… Sorry?
168 00:17:00.820 ⇒ 00:17:01.719 Hannah Wang: Sorry, go ahead.
169 00:17:01.880 ⇒ 00:17:07.369 Mustafa Raja: Yeah, we really have added much more. It needs better formatting,
170 00:17:08.190 ⇒ 00:17:21.279 Mustafa Raja: But we have added, rationale to each of the slides, and what’s positive, what’s negative, and then an overall feedback.
171 00:17:21.280 ⇒ 00:17:28.979 Mustafa Raja: better questions and narrative flow. So we have added a few things. What we need to do is, tweak the…
172 00:17:29.270 ⇒ 00:17:34.869 Mustafa Raja: formatting better. And this is what we are going to do in the first week.
173 00:17:35.540 ⇒ 00:17:40.350 Hannah Wang: I see. And then who designs the deck? We just give them the outline and content.
174 00:17:40.350 ⇒ 00:17:40.870 Mustafa Raja: Yeah, yeah, yeah.
175 00:17:40.870 ⇒ 00:17:41.430 Hannah Wang: Yeah.
176 00:17:41.430 ⇒ 00:17:45.840 Mustafa Raja: In terms of visuals, right?
177 00:17:46.200 ⇒ 00:17:46.840 Hannah Wang: Yeah.
178 00:17:47.210 ⇒ 00:17:49.850 Mustafa Raja: It’s their own team that designs.
179 00:17:49.850 ⇒ 00:17:51.519 Hannah Wang: Third designer? Okay.
180 00:17:52.080 ⇒ 00:17:58.400 Mustafa Raja: Yeah, but what happens is, they give this thing to their client.
181 00:17:59.140 ⇒ 00:18:01.049 Mustafa Raja: For approval.
182 00:18:01.590 ⇒ 00:18:03.140 Mustafa Raja: Oh, yup. I can go back.
183 00:18:05.200 ⇒ 00:18:23.590 Mustafa Raja: So what they would do is give this deck to their client, their client would proofread it, ask for any, changes if they want. If not, then they would create a visualized form of it. I see. Maybe DPT or something.
184 00:18:24.210 ⇒ 00:18:26.090 Hannah Wang: Got it. Okay, cool.
185 00:18:27.400 ⇒ 00:18:42.879 Hannah Wang: So, the context is, yeah, Matthew, the head of the company, was manually creating the outline for the decks. It’s time-consuming, and they just needed a way to automate it. And so, I guess that kind of goes into the challenge as well. So.
186 00:18:42.880 ⇒ 00:18:58.719 Hannah Wang: like, I’m sure you’ve seen our case studies, there’s, like, a context portion, and then there’s, like, a challenge portion. The challenge portion is, like, the pain points that our client was having. So, yeah, you mentioned it was time-consuming, they needed to… Matthew needed to focus on other things.
187 00:18:58.730 ⇒ 00:19:05.079 Hannah Wang: And… yeah, it’s just… takes a long time. Any other challenges that they express?
188 00:19:06.090 ⇒ 00:19:14.899 Mustafa Raja: On their side, these would be the challenges. If we need to discuss the challenges we faced while developing this, I can.
189 00:19:15.370 ⇒ 00:19:20.699 Mustafa Raja: walk you through that. Let me know if this would be something we would want to discuss.
190 00:19:21.610 ⇒ 00:19:31.810 Hannah Wang: I think it’s okay. I think focusing on the client challenges is good. Wow, that workflow is crazy. It’s massive. Okay.
191 00:19:32.060 ⇒ 00:19:43.979 Hannah Wang: So I guess this is the solution, right? So I’m going into the solution portion, so I guess you can walk me through what the solution entailed, including, like, tools and what the workflow does, yeah.
192 00:19:44.140 ⇒ 00:19:55.799 Mustafa Raja: So for, tools, we can say that we are using, the Cloud API, that’s the LLM that we are using for the AI agents, and then we have this
193 00:19:56.150 ⇒ 00:20:05.860 Mustafa Raja: Orchestrator agent. This is actually… this is actually the main agent, and this main agent has some children.
194 00:20:06.340 ⇒ 00:20:06.760 Hannah Wang: Yes.
195 00:20:06.760 ⇒ 00:20:10.320 Mustafa Raja: And each children has a specific job.
196 00:20:10.790 ⇒ 00:20:24.970 Mustafa Raja: This is some sort of new architecture that NATN introduced, and we used it over here, and the timing was really great, because a single agent isn’t able to handle a lot of this workload.
197 00:20:26.430 ⇒ 00:20:45.269 Mustafa Raja: So, each agent over here has the, has its own task, so, so firstly, we’d summarize whatever the files were given to us, and then, based on those files, we… what we would do, we would, extract all of the… what’s it called?
198 00:20:45.490 ⇒ 00:21:05.710 Mustafa Raja: insights from them, because we don’t want to use, any facts, we don’t want to mention any facts outside of those files, outside of those sources. So the AI agent doesn’t really have ability to cite anything on its own. What it’s going to cite has to be within the files that they provide us.
199 00:21:07.030 ⇒ 00:21:22.290 Mustafa Raja: Once these two are done, the narrative architect comes in, based on, the insights that we have in the summarizer, well, the summarizer really just summarizes the questionnaire and the,
200 00:21:22.560 ⇒ 00:21:33.189 Mustafa Raja: transcript, based on, output of these two things, the narrative architect would architect a narrative, flow, which is going to be…
201 00:21:33.380 ⇒ 00:21:58.179 Mustafa Raja: how we are going to start, and how we are going to end, and what’s going to be the flow of the deck. And then the copywriter would add formatting and proper headlines and bullets to that. And what this QA compliance agent does is it proofreads it, sees that each slide does not have any fact mentioned that is not available.
202 00:21:58.180 ⇒ 00:22:10.040 Mustafa Raja: within the, files, and that wasn’t mentioned by, our insight analysts. This also does a SWOT analysis and some other analysis. I forgot.
203 00:22:10.040 ⇒ 00:22:15.500 Mustafa Raja: Yeah, so once this is done, this… that would become a V1…
204 00:22:15.690 ⇒ 00:22:21.269 Mustafa Raja: V1 of our deck. After that, to each slide, we’ll add Steelman.
205 00:22:21.710 ⇒ 00:22:27.950 Mustafa Raja: overview. What that is, is… let me open up the Notion thing again.
206 00:22:33.940 ⇒ 00:22:37.819 Mustafa Raja: Let me show you what Steelman is…
207 00:22:41.450 ⇒ 00:22:42.150 Mustafa Raja: Boom.
208 00:22:45.970 ⇒ 00:22:46.700 Mustafa Raja: Huh.
209 00:22:50.410 ⇒ 00:22:52.320 Mustafa Raja: Yeah, it’s given upon me.
210 00:22:54.560 ⇒ 00:22:55.959 Mustafa Raja: Let’s open it up again.
211 00:22:56.610 ⇒ 00:22:57.270 Mustafa Raja: Nope.
212 00:23:12.390 ⇒ 00:23:14.980 Hannah Wang: Oh… oh no, Notion.
213 00:23:16.680 ⇒ 00:23:19.160 Mustafa Raja: Yeah, it’s given up on me.
214 00:23:19.320 ⇒ 00:23:34.669 Mustafa Raja: So, what, so, so you saw that, with each slide, we have something positive and something negative, right? Yes. That would be the steelman remarks, and then once that’s done, we, we do an overall feedback.
215 00:23:36.180 ⇒ 00:23:53.299 Mustafa Raja: So we add all of these things, of version 1 of the deck only, then add the steelman, then add an overall feedback, and then we just format it and throw it in the Slack. Let’s actually… Okay. I can show you the Slack things, too.
216 00:23:54.170 ⇒ 00:23:56.230 Mustafa Raja: Dona Cocillos.
217 00:23:56.990 ⇒ 00:24:01.130 Mustafa Raja: Yeah, this would be an example.
218 00:24:01.760 ⇒ 00:24:05.859 Mustafa Raja: So yeah, once that’s done, it comes over here.
219 00:24:08.100 ⇒ 00:24:12.890 Mustafa Raja: So this would be the whole, whole deck.
220 00:24:13.720 ⇒ 00:24:14.330 Hannah Wang: Yeah.
221 00:24:14.460 ⇒ 00:24:19.720 Mustafa Raja: And then I can ask it to, you know,
222 00:24:20.210 ⇒ 00:24:29.670 Mustafa Raja: if I don’t like anything over here, what I can do is redo that, or add my review on that, so… for it to work on.
223 00:24:29.870 ⇒ 00:24:35.669 Mustafa Raja: If that is the case, if I give it a review, what happens is we do not come over to this loop.
224 00:24:36.490 ⇒ 00:24:38.419 Mustafa Raja: Well, where we go is this loop.
225 00:24:40.460 ⇒ 00:24:50.319 Mustafa Raja: And what this does is it checks whether the deck is approved or not. If it’s approved, we go over here, format it properly, and store it in Notion.
226 00:24:51.690 ⇒ 00:24:53.919 Mustafa Raja: If it’s a feedback, we go over here.
227 00:24:54.600 ⇒ 00:25:08.010 Mustafa Raja: We format it, and see which part of the deck is being asked, a revamped. Is it a slide? Is it the narrative? Is it the questions? Is it the feedback?
228 00:25:08.490 ⇒ 00:25:24.790 Mustafa Raja: If it’s a slide, we then break it down further to the slide, and then pass that, and then adjust whatever we want, and send in the response. Similarly, if it’s the narrative, questions, or feedback.
229 00:25:24.790 ⇒ 00:25:25.200 Hannah Wang: Hmm.
230 00:25:25.200 ⇒ 00:25:33.980 Mustafa Raja: whatever it is, we have dedicated agents for that to take care of. And then we reply back
231 00:25:34.270 ⇒ 00:25:39.599 Mustafa Raja: With the… with the updated version in the Slack.
232 00:25:40.580 ⇒ 00:25:42.550 Mustafa Raja: Oh, so…
233 00:25:42.550 ⇒ 00:25:43.180 Hannah Wang: Nice.
234 00:25:43.660 ⇒ 00:25:55.279 Mustafa Raja: Hmm, so the reason, we had to separate, these, or, what we can say is, we had to break down the whole thing is,
235 00:25:55.280 ⇒ 00:26:03.960 Mustafa Raja: AI agents cannot really take in all of that text and return back exactly the same thing and change only one specific thing.
236 00:26:03.960 ⇒ 00:26:04.430 Hannah Wang: Hmm.
237 00:26:04.430 ⇒ 00:26:15.690 Mustafa Raja: We have to break it down… we have to break this string down based on this review, to pinpoint the exact thing that’s, that wants… that needs to be changed.
238 00:26:16.820 ⇒ 00:26:17.330 Mustafa Raja: So then…
239 00:26:17.330 ⇒ 00:26:17.720 Hannah Wang: Got it.
240 00:26:17.720 ⇒ 00:26:22.159 Mustafa Raja: We keep the structure, rest of the structure exactly the same.
241 00:26:22.540 ⇒ 00:26:23.400 Hannah Wang: I see.
242 00:26:24.200 ⇒ 00:26:24.750 Mustafa Raja: Yep.
243 00:26:25.590 ⇒ 00:26:34.189 Mustafa Raja: Yeah, I guess this would be the flow. And then, obviously, once we say, yeah, it’s approved for Notion, it’s going to take this path.
244 00:26:34.330 ⇒ 00:26:35.800 Mustafa Raja: And save it in Notion.
245 00:26:36.090 ⇒ 00:26:37.090 Hannah Wang: Oh, cool.
246 00:26:38.320 ⇒ 00:26:46.669 Hannah Wang: Cool. Wow. So, the tools that are used, I saw a bunch of logos. So, it’s Slack, Notion, I saw Azure, OpenAI.
247 00:26:46.670 ⇒ 00:26:47.340 Mustafa Raja: Yay.
248 00:26:47.340 ⇒ 00:26:48.480 Hannah Wang: Quiet.
249 00:26:48.480 ⇒ 00:26:55.899 Mustafa Raja: We might remove, Azure, though. Okay. Because client wants to work only with Cloud.
250 00:26:56.240 ⇒ 00:27:01.160 Mustafa Raja: So I guess it’s… Good to mention that we only add clots.
251 00:27:01.930 ⇒ 00:27:07.749 Hannah Wang: Okay, Claude, and then, is there any other tool, like, OpenAI? I guess Claude is…
252 00:27:08.660 ⇒ 00:27:11.959 Mustafa Raja: Yeah, I guess we won’t be using OpenAI, to be honest.
253 00:27:11.960 ⇒ 00:27:12.810 Hannah Wang: Okay.
254 00:27:13.250 ⇒ 00:27:18.320 Hannah Wang: Obviously, N-A-N, we use N-A-N.
255 00:27:18.610 ⇒ 00:27:19.710 Hannah Wang: Okay, cool.
256 00:27:21.430 ⇒ 00:27:22.210 Hannah Wang: Awesome.
257 00:27:22.350 ⇒ 00:27:25.219 Hannah Wang: That’s the solution.
258 00:27:25.220 ⇒ 00:27:25.630 Mustafa Raja: No.
259 00:27:26.370 ⇒ 00:27:29.019 Mustafa Raja: we can add BrainTrust, too.
260 00:27:29.020 ⇒ 00:27:29.820 Hannah Wang: Brain Trust? Okay.
261 00:27:30.790 ⇒ 00:27:32.069 Mustafa Raja: or emails.
262 00:27:33.060 ⇒ 00:27:37.240 Hannah Wang: Okay. Oh, yeah, obviously, yeah, okay.
263 00:27:38.120 ⇒ 00:27:43.330 Hannah Wang: Cool, and so I know we have to work on, like, a revamp of this
264 00:27:43.460 ⇒ 00:27:56.359 Hannah Wang: But I guess with the previous version, as we were testing it and sending it to the client, is there anything that they said in terms of feedback? Like, oh, this is helpful, this is not helpful, like, what was the overall?
265 00:27:56.360 ⇒ 00:28:14.490 Mustafa Raja: Yeah, so they were, they were happy with, how far it’s, come along. Obviously, they had some suggestions. What it does is it, for now, it only takes in the .txt files or RTF files, so, they would either want,
266 00:28:15.050 ⇒ 00:28:30.080 Mustafa Raja: that we either don’t have to include the files at all. What they can do is just paste the text in the textbooks only and throw it. And what they would want is also support for PDFs.
267 00:28:32.210 ⇒ 00:28:32.820 Mustafa Raja: Yeah.
268 00:28:33.270 ⇒ 00:28:45.030 Mustafa Raja: So that was one thing, and the other thing was that, the other thing that I mentioned about the formatting issue that we have in Notion right now,
269 00:28:45.140 ⇒ 00:28:48.980 Mustafa Raja: So that is something that we need to fix. So these two were the…
270 00:28:49.650 ⇒ 00:28:51.979 Mustafa Raja: Things that we wanted to improve last week.
271 00:28:52.960 ⇒ 00:28:53.650 Hannah Wang: Okay.
272 00:28:54.520 ⇒ 00:28:58.760 Hannah Wang: Okay, so overall it was helpful, but obviously it’s, like, a V1, so we’re gonna.
273 00:28:58.760 ⇒ 00:28:59.430 Mustafa Raja: technique.
274 00:28:59.690 ⇒ 00:29:01.130 Hannah Wang: V2s and stuff.
275 00:29:01.130 ⇒ 00:29:02.150 Mustafa Raja: Thank you.
276 00:29:02.480 ⇒ 00:29:05.180 Hannah Wang: Is there any… like…
277 00:29:06.350 ⇒ 00:29:13.349 Hannah Wang: do you have any of the feedback from them? Like, the ones where it said, oh, this is helpful? Like, do you have a text?
278 00:29:14.510 ⇒ 00:29:16.040 Hannah Wang: Red, or something.
279 00:29:17.160 ⇒ 00:29:20.010 Mustafa Raja: I guess we can go to the meeting.
280 00:29:23.110 ⇒ 00:29:27.569 Hannah Wang: Yeah, if you can point me to the meeting, like, I don’t know which one it was.
281 00:29:27.570 ⇒ 00:29:30.879 Mustafa Raja: Yeah, I’ll, I’ll send you the link. That’s…
282 00:29:30.880 ⇒ 00:29:31.370 Hannah Wang: Okay, cool.
283 00:29:31.370 ⇒ 00:29:37.729 Mustafa Raja: Let’s see if we have interlude over here in the platform, else I’ll have to fetch it from the database.
284 00:29:38.230 ⇒ 00:29:38.820 Hannah Wang: Right.
285 00:29:42.140 ⇒ 00:29:44.329 Hannah Wang: I don’t see it.
286 00:29:44.330 ⇒ 00:29:48.660 Mustafa Raja: on its… Quickly get it from here.
287 00:30:33.890 ⇒ 00:30:39.760 Mustafa Raja: Hmm… Let’s see if Matthew…
288 00:31:09.410 ⇒ 00:31:13.480 Mustafa Raja: Hmm, it’s using the… But this is too old.
289 00:31:22.250 ⇒ 00:31:23.150 Mustafa Raja: Hmm.
290 00:31:31.120 ⇒ 00:31:33.390 Mustafa Raja: Let’s search for this…
291 00:31:37.030 ⇒ 00:31:38.020 Mustafa Raja: pounds…
292 00:31:45.620 ⇒ 00:31:46.960 Mustafa Raja: Yep, this…
293 00:31:51.040 ⇒ 00:31:53.810 Mustafa Raja: Yeah, I guess this, this one, this is the one.
294 00:31:54.330 ⇒ 00:31:54.970 Hannah Wang: Okay.
295 00:31:56.010 ⇒ 00:31:59.579 Mustafa Raja: Copy, copy the row…
296 00:32:05.200 ⇒ 00:32:06.100 Mustafa Raja: Oh, no.
297 00:32:06.390 ⇒ 00:32:07.005 Hannah Wang: Oh.
298 00:32:10.860 ⇒ 00:32:12.519 Mustafa Raja: Solid cell now.
299 00:32:12.520 ⇒ 00:32:14.120 Hannah Wang: So, yeah.
300 00:32:33.550 ⇒ 00:32:34.010 Mustafa Raja: Yep.
301 00:32:34.010 ⇒ 00:32:37.099 Hannah Wang: Oh, cool, okay. Yeah, you can just send me the link.
302 00:32:37.360 ⇒ 00:32:40.729 Hannah Wang: And then I’ll get the transcript from that.
303 00:32:42.770 ⇒ 00:32:44.750 Hannah Wang: Cool. Alright.
304 00:32:45.290 ⇒ 00:32:50.650 Hannah Wang: Okay, I think this is good for now.
305 00:32:51.000 ⇒ 00:33:00.779 Hannah Wang: after I create the case study, I might ask you to look at it just to fact check everything, but I think this is a good start. And then, like I mentioned.
306 00:33:01.070 ⇒ 00:33:08.419 Hannah Wang: Once we build, like, the V2 of the workflow and everything, and… Build the evals, and… or…
307 00:33:08.640 ⇒ 00:33:14.439 Hannah Wang: yeah, just, you know, human-in-the-loop stuff, like you mentioned. I think we can talk again.
308 00:33:14.970 ⇒ 00:33:17.600 Hannah Wang: I guess?
309 00:33:18.070 ⇒ 00:33:26.150 Hannah Wang: Yeah, you already… I guess this is a eval case study. I’m just trying to think if we talked enough about evals. I think we did, like, in the beginning.
310 00:33:26.190 ⇒ 00:33:28.380 Mustafa Raja: Before you went through the whole workflow.
311 00:33:28.560 ⇒ 00:33:31.289 Mustafa Raja: Yeah, let me know if you have any of the other questions.
312 00:33:31.980 ⇒ 00:33:32.720 Hannah Wang: Okay.
313 00:33:33.120 ⇒ 00:33:34.210 Hannah Wang: Okay, cool.
314 00:33:34.560 ⇒ 00:33:36.730 Hannah Wang: I think this is good. Thanks, Ms. Aka.
315 00:33:36.730 ⇒ 00:33:37.890 Mustafa Raja: Okay, thank you.
316 00:33:38.230 ⇒ 00:33:39.440 Hannah Wang: Thank you, bye.
317 00:33:39.440 ⇒ 00:33:40.190 Mustafa Raja: Bye, have a good.