Meeting Title: Central Doc Optimization Sync Date: 2026-02-02 Meeting participants: Samuel Roberts, Amber Lin, Pranav Narahari, Uttam Kumaran
WEBVTT
1 00:00:13.610 ⇒ 00:00:14.620 Amber Lin: Hello!
2 00:00:15.320 ⇒ 00:00:16.100 Samuel Roberts: Ayy.
3 00:00:20.910 ⇒ 00:00:23.780 Samuel Roberts: Yeah, you were right on top of that as soon as I asked, so I figured you were.
4 00:00:23.780 ⇒ 00:00:26.320 Amber Lin: Yeah, Pranav told me, and I was like, oof, sorry.
5 00:00:26.320 ⇒ 00:00:30.320 Samuel Roberts: Okay, yeah, he and I were just on a huddle right before this, so we both hopped off to hop on here.
6 00:00:30.320 ⇒ 00:00:31.660 Amber Lin: I see.
7 00:00:32.150 ⇒ 00:00:34.129 Samuel Roberts: We’re, yeah, we’re on sync here.
8 00:00:35.160 ⇒ 00:00:51.840 Amber Lin: Cool. I think Utam will be here, but he should have a call right before this, so he might be a little late. I think, Sam, since we figured out the zip code stuff, I was thinking if we should just focus on the central dock for this call.
9 00:00:52.180 ⇒ 00:00:55.180 Samuel Roberts: That’s fine. I think that makes the most sense, because that’s going to be the biggest, like.
10 00:00:55.430 ⇒ 00:00:57.169 Amber Lin: Cool. Thing for them. Cool.
11 00:00:57.390 ⇒ 00:00:58.330 Samuel Roberts: Today alert.
12 00:01:00.440 ⇒ 00:01:01.950 Samuel Roberts: And that’s, yeah.
13 00:01:05.850 ⇒ 00:01:14.340 Amber Lin: Alright, let me… I just started a Notion doc, I’m gonna share it in the chat, and then…
14 00:01:14.470 ⇒ 00:01:16.249 Amber Lin: Can go from there.
15 00:01:31.920 ⇒ 00:01:32.715 Amber Lin: Mmm…
16 00:02:06.840 ⇒ 00:02:08.829 Samuel Roberts: I’m gonna stick with one quick second, I’ll be right back, though.
17 00:02:09.020 ⇒ 00:02:09.530 Amber Lin: Okay.
18 00:02:09.530 ⇒ 00:02:10.319 Samuel Roberts: Grab some water.
19 00:02:10.460 ⇒ 00:02:11.539 Samuel Roberts: My bottle’s empty?
20 00:02:29.140 ⇒ 00:02:31.989 Pranav Narahari: I think I might be just missing a little bit of context on…
21 00:02:32.440 ⇒ 00:02:37.469 Pranav Narahari: Did we have, like, a specific thing that we wanted to talk about for this meeting, or is it kind of just
22 00:02:37.850 ⇒ 00:02:39.080 Pranav Narahari: Okay.
23 00:02:39.800 ⇒ 00:02:41.329 Amber Lin: I can…
24 00:02:41.470 ⇒ 00:02:50.870 Amber Lin: I think once Sam gets back, I’ll run through it. So, essentially, the problem we’re facing right now is that, the departments have
25 00:02:51.200 ⇒ 00:02:52.110 Amber Lin: Hello.
26 00:02:53.050 ⇒ 00:02:54.329 Samuel Roberts: Oh, perfect timing, but…
27 00:02:54.640 ⇒ 00:03:10.910 Amber Lin: Okay, awesome, so we have everybody. So I think the… to give us all context, the problem we have right now is that for the new departments, so the non-pest department, mechanical home improvement and Lawn, their central doc gives
28 00:03:10.950 ⇒ 00:03:19.020 Amber Lin: Sometimes conflicting responses, they give, inaccurate responses, and,
29 00:03:19.020 ⇒ 00:03:33.559 Amber Lin: Some is because it’s not in the central doc, which is not our issue, but some is because central doc has duplicates, it’s not organized correctly, and then… or that they don’t know how to word it when they add it to the central doc, so that it gives
30 00:03:33.560 ⇒ 00:03:40.889 Amber Lin: Wrong responses, although the client has added it into the central docket one way or another.
31 00:03:41.210 ⇒ 00:03:49.609 Amber Lin: So, I think the goal, we have here is, to figure out, one, like, what is the best
32 00:03:49.640 ⇒ 00:03:59.670 Amber Lin: organization or a structure, for the responses, and also, like, formatting guidelines and wording
33 00:03:59.700 ⇒ 00:04:13.639 Amber Lin: guidelines, so that in the future, when the client updates the central doc, we don’t have to go in and say, hey, this is the best way, so that they can follow the guidelines, so that, like, in the long run, they don’t mess it up again.
34 00:04:13.650 ⇒ 00:04:22.950 Amber Lin: I think it’s just the accumulation of chaos throughout time, and I want to give them a template or a guide that they can use.
35 00:04:23.560 ⇒ 00:04:39.739 Amber Lin: So that… I think that’s the context there. Previously, when we did the test department, so to work on their central dock, I spent a lot of time, say, one-on-one with the trainers to refine each and every
36 00:04:39.840 ⇒ 00:04:59.230 Amber Lin: document and walk them through a lot of the wording changes. I don’t… I don’t think we want to do that again, so I was trying to see, do we have any AI tools we can help with that? Or if we can split the work and work with the trainers,
37 00:04:59.890 ⇒ 00:05:10.709 Amber Lin: So that, like, we can have more capacity on this project, but that’s the context here. Any questions that I can help answer before we start discussing?
38 00:05:10.710 ⇒ 00:05:16.899 Uttam Kumaran: Can you list down the types of structural changes that are common? Like, I just want to see every type of, like.
39 00:05:17.070 ⇒ 00:05:23.229 Uttam Kumaran: Well, there’s a couple things. Even before structural changes, there’s net new information gathering, right? Isn’t that a thing?
40 00:05:23.900 ⇒ 00:05:29.269 Amber Lin: What do you mean? Like, when they produce new stuff, do they add it to the central dock?
41 00:05:29.270 ⇒ 00:05:34.749 Uttam Kumaran: Well, like, for mechanical, right, you’re… you’re gonna… you have to go get their shit from somewhere.
42 00:05:35.350 ⇒ 00:05:40.929 Amber Lin: We already have these stuff, and hypothetically, I think we have all of them.
43 00:05:41.130 ⇒ 00:05:47.269 Uttam Kumaran: But I guess, what’s the path of moving that into the, like, basically importing that in? You just copy-paste.
44 00:05:47.270 ⇒ 00:05:53.849 Amber Lin: They already made it already, so I should have all the stuff since when they sent it to me. But did you just.
45 00:05:53.850 ⇒ 00:05:55.090 Uttam Kumaran: Copy and paste it in.
46 00:05:55.270 ⇒ 00:05:55.990 Amber Lin: Yes.
47 00:05:56.510 ⇒ 00:06:02.339 Uttam Kumaran: Okay, can you… so can we put that… I just want to list down all the ways that this central doc gets created.
48 00:06:03.200 ⇒ 00:06:06.730 Uttam Kumaran: Then gets, like, basically updated and optimized.
49 00:06:07.000 ⇒ 00:06:11.839 Amber Lin: So net new information, so if you just put a thing at the bottom, you can just say one is, like.
50 00:06:11.910 ⇒ 00:06:17.390 Uttam Kumaran: you’re… you have to gather all this into a Google Drive, right? So let’s put that one. So gather into Google Drive.
51 00:06:17.390 ⇒ 00:06:19.550 Samuel Roberts: Where was this information before the Central Docs, then?
52 00:06:20.030 ⇒ 00:06:33.910 Amber Lin: It’s just, you know, random Google Drive stuff. They have multiple folders, they have different… some are PDFs, most are Words, some are Excels, and we changed them and tried to add them.
53 00:06:34.080 ⇒ 00:06:42.689 Samuel Roberts: That makes sense. Okay, yeah, I didn’t… I didn’t know all that from, like, before I joined. I wasn’t sure, like, what was their setup and what was our setup on top of what they… yeah. Okay.
54 00:06:42.690 ⇒ 00:06:44.140 Amber Lin: That’s good, that’s good to know.
55 00:06:45.170 ⇒ 00:06:48.810 Uttam Kumaran: Okay, so gather all sources in Google Doc, copy, paste it into…
56 00:06:49.570 ⇒ 00:06:53.120 Uttam Kumaran: Oh, well, I guess gather all… well, I guess gather all sources into Google Drive.
57 00:06:53.240 ⇒ 00:06:56.760 Uttam Kumaran: And then there’s a copy-paste step. Sorry, I just want to be very kind of pedantic.
58 00:06:56.760 ⇒ 00:07:07.260 Amber Lin: Yeah, so I did that, and then I made sure to list all sources in the Google Sheet, which is the spreadsheet hub that we have. Let’s just make this…
59 00:07:07.260 ⇒ 00:07:09.020 Uttam Kumaran: Three, three items, yeah.
60 00:07:09.020 ⇒ 00:07:18.500 Amber Lin: Okay, I got the items and checked with them what’s needed, And confirm duplicates.
61 00:07:18.780 ⇒ 00:07:22.819 Amber Lin: And then lastly is copy and paste.
62 00:07:23.220 ⇒ 00:07:30.539 Amber Lin: Or format, copy and paste into Google Doc.
63 00:07:32.530 ⇒ 00:07:39.240 Amber Lin: And then that’s… that gets linked, that gets, chunked and put into Superbase.
64 00:07:39.770 ⇒ 00:07:40.600 Uttam Kumaran: Okay.
65 00:07:40.600 ⇒ 00:07:45.129 Samuel Roberts: And we’ve done some work on that related to, like, the structural changes and stuff that is coming. So, like.
66 00:07:45.410 ⇒ 00:07:51.269 Samuel Roberts: headings, Passing all that information along to the LLM so it knows, you know, this is…
67 00:07:51.780 ⇒ 00:07:54.159 Samuel Roberts: Lawn, this is mowing, this is…
68 00:07:54.160 ⇒ 00:07:58.940 Uttam Kumaran: That happens via, like, an embedding? Like, that’s like a manual…
69 00:07:59.810 ⇒ 00:08:01.690 Samuel Roberts: No, no, so, like, the… we…
70 00:08:02.380 ⇒ 00:08:10.500 Samuel Roberts: The way it happens… so, like, once the dock is, you know, whatever, once we get, like, the dock together, because there were some problems where
71 00:08:10.670 ⇒ 00:08:16.219 Samuel Roberts: It was pulling from certain parts of the doc that weren’t related to… like, it might catch a word or something in the embedding.
72 00:08:16.900 ⇒ 00:08:19.009 Samuel Roberts: And realize… and it might be the wrong thing.
73 00:08:19.130 ⇒ 00:08:23.230 Samuel Roberts: We added a kind of a metadata field to all the…
74 00:08:23.890 ⇒ 00:08:26.680 Samuel Roberts: The chunks that says, like, what…
75 00:08:26.870 ⇒ 00:08:30.049 Samuel Roberts: Like, what part of the hierarchy is it in, if that makes sense.
76 00:08:30.560 ⇒ 00:08:37.079 Samuel Roberts: So, like, if the H1 reads this, the H2 at that level reads this, the H3 reads this, and that’s the section here, so that it kind of can…
77 00:08:37.640 ⇒ 00:08:39.270 Samuel Roberts: Know a little bit better.
78 00:08:39.870 ⇒ 00:08:43.589 Samuel Roberts: The greater context, because it’s only the chunks and the embedding range.
79 00:08:43.590 ⇒ 00:08:50.090 Uttam Kumaran: Is that, like… Oh, and that happens every time. Like, we’re not, like, saving that, or it gets synced.
80 00:08:50.090 ⇒ 00:08:53.089 Samuel Roberts: No, it’s getting started in Superbase along with the embedding.
81 00:08:53.710 ⇒ 00:08:55.680 Uttam Kumaran: And how often do we do that?
82 00:08:56.970 ⇒ 00:08:59.429 Samuel Roberts: Rerun it?
83 00:08:59.430 ⇒ 00:09:00.130 Uttam Kumaran: Yeah.
84 00:09:02.170 ⇒ 00:09:06.840 Samuel Roberts: I don’t know if the whole thing gets rerun, but if you look in the, it’s run on updates, I think.
85 00:09:07.100 ⇒ 00:09:07.910 Samuel Roberts: Changes to the…
86 00:09:07.910 ⇒ 00:09:14.380 Uttam Kumaran: Okay, Amber, can we put that… let’s just do on update, so we put as B under structural changes.
87 00:09:14.680 ⇒ 00:09:23.530 Uttam Kumaran: Oh, actually… Yeah, under… put it under structural changes as B. You can say on document update, rerun embeddings.
88 00:09:25.630 ⇒ 00:09:33.200 Samuel Roberts: Yeah. I think there had been a problem that they discovered recently, that, like, not everything was running for every dock. That was an issue, but…
89 00:09:34.290 ⇒ 00:09:40.400 Uttam Kumaran: Alright, I just want to list out everything, and then I’m gonna… we’re gonna go… we’re gonna go verify each thing. So, optimizing formatting.
90 00:09:40.520 ⇒ 00:09:45.429 Uttam Kumaran: wording. So, yeah, let’s keep going. So, consolidating, what.
91 00:09:45.430 ⇒ 00:09:54.119 Amber Lin: They have multiple SOPs with minimal differences, but they don’t really understand how to consolidate them.
92 00:09:54.240 ⇒ 00:10:02.409 Amber Lin: So, like, consolidating redundant SOPs and information.
93 00:10:02.490 ⇒ 00:10:05.319 Amber Lin: There are specific…
94 00:10:05.330 ⇒ 00:10:24.730 Amber Lin: infos, like, how much time does this take, how long is that, is also scattered. So when I went through that, that’s also a process of putting, the information in one place, removing all the access, so, like, similar thing. So this is another step.
95 00:10:24.780 ⇒ 00:10:30.669 Amber Lin: That makes it have less, duplicate information, and thus less errors.
96 00:10:30.870 ⇒ 00:10:36.200 Samuel Roberts: I think that’s a big part of it, because, like I said, when you ask about something, it’ll find it in a few different places, and…
97 00:10:36.690 ⇒ 00:10:40.699 Samuel Roberts: It can be consolidated, and then the specific information there would be…
98 00:10:41.810 ⇒ 00:10:59.769 Amber Lin: And then I think the last step is improving wording. This is kind of like prompting. They have to change the context words in different ways. Sometimes it comes up, like, if you… so, like, that’s the line I don’t know how to draw for them, because I… it’s also trial and error for me.
99 00:11:02.670 ⇒ 00:11:03.469 Samuel Roberts: Yeah, that one…
100 00:11:03.470 ⇒ 00:11:04.010 Uttam Kumaran: Okay.
101 00:11:04.520 ⇒ 00:11:08.510 Samuel Roberts: We can kind of attack two different ways, too, where once we see more of those errors, we can.
102 00:11:08.780 ⇒ 00:11:10.360 Amber Lin: Treat the prompt to know.
103 00:11:10.540 ⇒ 00:11:12.730 Samuel Roberts: Different aliases they use and things, but…
104 00:11:13.220 ⇒ 00:11:16.700 Amber Lin: That’s, again, you’re right, it’s a trial and error thing for that, even on that side of it.
105 00:11:17.380 ⇒ 00:11:20.310 Uttam Kumaran: So let’s move number… let’s remove number 4.
106 00:11:20.620 ⇒ 00:11:31.299 Uttam Kumaran: So, improve wording, consolidate redundant SOPs, create high-level sections, rerun embeddings.
107 00:11:31.600 ⇒ 00:11:35.729 Uttam Kumaran: Gather all sources in Drive, list all sources in Google Sheet,
108 00:11:43.320 ⇒ 00:11:53.490 Uttam Kumaran: Okay, and then for net new… for net new information, I guess for net new information, then, like, actually, let’s just do, like, updates. So they’re… they’re just gonna paste it at the bottom, basically, or how to…
109 00:11:54.030 ⇒ 00:11:55.150 Uttam Kumaran: Add it somewhere.
110 00:11:55.150 ⇒ 00:12:07.749 Amber Lin: Yeah, like, I… they should know the structure. I have an outline, it’s the table of content at the top, so they’ll find the relevant part and put it in. So the client’s responsible for adding new stuff.
111 00:12:08.220 ⇒ 00:12:08.920 Uttam Kumaran: Okay.
112 00:12:32.470 ⇒ 00:12:41.239 Amber Lin: So, what I’m working with the client right now is essentially this part, that’s step one, and then these two.
113 00:12:41.240 ⇒ 00:12:44.490 Uttam Kumaran: So how do you use AI for this right now, if at all?
114 00:12:44.670 ⇒ 00:12:45.930 Amber Lin: I…
115 00:12:46.380 ⇒ 00:13:01.069 Amber Lin: back then, I took the main sections and asked it for optimal organization. Remember the stuff we’re talking about with Scott of, like, the knowledge network and whatever? Like, that’s the organization I could use AI for.
116 00:13:01.250 ⇒ 00:13:07.190 Amber Lin: this step… has a little bit of AI, like… AI.
117 00:13:07.190 ⇒ 00:13:11.490 Uttam Kumaran: So let’s put… let’s put a… let’s put a… let’s go down and put a section, how AI is being used right now.
118 00:13:11.490 ⇒ 00:13:12.859 Amber Lin: Okay, cool.
119 00:13:13.130 ⇒ 00:13:15.659 Amber Lin: Let’s do this…
120 00:13:17.750 ⇒ 00:13:19.300 Uttam Kumaran: So, yeah.
121 00:13:19.790 ⇒ 00:13:20.540 Amber Lin: AI.
122 00:13:20.540 ⇒ 00:13:26.279 Uttam Kumaran: Well, you can just, like, you can just remove this and just put… you can just put, like, literally AIs being used to do this, yeah.
123 00:13:28.510 ⇒ 00:13:31.350 Amber Lin: for reformatting…
124 00:13:31.940 ⇒ 00:13:35.200 Uttam Kumaran: And then Pranav, are you… you’re familiar with what the central dock is?
125 00:13:36.270 ⇒ 00:13:39.730 Pranav Narahari: Yeah, I think I’ve gathered it now. It’s basically…
126 00:13:39.730 ⇒ 00:13:41.630 Uttam Kumaran: It’s just this bad document.
127 00:13:42.080 ⇒ 00:13:42.720 Pranav Narahari: Yeah.
128 00:13:43.170 ⇒ 00:13:57.209 Amber Lin: Let me paste in the link here. I sent the Notion link in our… like, this is… this is, like, the best, this is the best organized document they have so far. Just… I organize it into general SOPs, and then…
129 00:13:57.560 ⇒ 00:14:15.710 Amber Lin: specific SOPs, and then the last is, like, this is their common info, and then this is, like, their service-specific information. So this is as lean, and I can get it to be for now, but then, like, as they add things, if they don’t know the formatting, they… it still gets…
130 00:14:15.850 ⇒ 00:14:19.280 Amber Lin: The chaos accumulates still.
131 00:14:20.080 ⇒ 00:14:23.089 Pranav Narahari: And is there a reason why it needs to be just one document?
132 00:14:23.710 ⇒ 00:14:26.889 Amber Lin: I think it’s because it was very hard to…
133 00:14:27.420 ⇒ 00:14:42.159 Amber Lin: link all the documents in NAN, then you have to have a node for each doc, and you never know, like, when they change one or they don’t. So it would have been a lot more manual if it was still in their individual docs.
134 00:14:43.050 ⇒ 00:14:44.089 Pranav Narahari: Gotcha, okay.
135 00:14:44.640 ⇒ 00:14:53.319 Uttam Kumaran: Well, like, what’s the… I guess I’m interested in, like, what would an alternative be, because we still need to… we still need 2A sync, that a human being can, like, go update something, you know?
136 00:14:54.600 ⇒ 00:15:00.170 Uttam Kumaran: Because they… they need… they need some type of Google Doc environment to continue to add information to.
137 00:15:00.320 ⇒ 00:15:05.279 Samuel Roberts: Right, and one doc also simplifies just, like, the single source of truth and not, you know…
138 00:15:05.520 ⇒ 00:15:06.699 Uttam Kumaran: Yeah, consider, like, a wiki.
139 00:15:06.700 ⇒ 00:15:08.059 Samuel Roberts: Different docs, yeah.
140 00:15:08.720 ⇒ 00:15:13.439 Samuel Roberts: I feel like it gets… You could get contradicting information in different places, and we wouldn’t, you know…
141 00:15:15.460 ⇒ 00:15:18.630 Samuel Roberts: Know how to resolve that without… intervention.
142 00:15:21.590 ⇒ 00:15:27.770 Samuel Roberts: I mean, that can still happen in this document, too, and that has been happening, I think, but at least here it’s one document we can find that in.
143 00:15:31.140 ⇒ 00:15:38.100 Uttam Kumaran: So, I guess my question, Amber, AI for listing, recommended structure and categorization, how are you doing this right now? This is just in cursor?
144 00:15:38.630 ⇒ 00:15:48.500 Amber Lin: This is way back then. Back then, I just used, ChatGPT. I haven’t helped them organize it since I’ve, like, finished the work for the pest doc.
145 00:15:48.770 ⇒ 00:15:51.059 Amber Lin: Okay. Because it just takes too much time, but…
146 00:15:51.060 ⇒ 00:15:57.049 Uttam Kumaran: And then for optimizing formatting, how are you doing that now, too?
147 00:15:58.250 ⇒ 00:16:08.039 Amber Lin: So, one is FOP optimization, two, like, small wording adjustments.
148 00:16:08.230 ⇒ 00:16:13.120 Amber Lin: To make it more, like, formalized…
149 00:16:13.540 ⇒ 00:16:19.670 Amber Lin: Cover all details, lists, examples, and scripts.
150 00:16:20.280 ⇒ 00:16:23.679 Amber Lin: So, essentially, you’re just telling her, hey, can you improve this?
151 00:16:23.920 ⇒ 00:16:31.840 Amber Lin: can you cover all the details? Because their wording was not great, and AI didn’t really understand.
152 00:16:31.980 ⇒ 00:16:33.719 Amber Lin: Before we rewarded it.
153 00:16:39.040 ⇒ 00:16:39.730 Uttam Kumaran: Okay.
154 00:16:40.480 ⇒ 00:16:41.759 Uttam Kumaran: So, it seems like…
155 00:16:42.280 ⇒ 00:16:47.929 Uttam Kumaran: There’s no situation right now in where we’re taking the entire document, handing it to AI, and say.
156 00:16:48.290 ⇒ 00:16:53.450 Uttam Kumaran: go through and find redundancies, for example. We’re not doing that today.
157 00:16:53.450 ⇒ 00:17:02.119 Amber Lin: I think it was… the context wasn’t big enough in ChatGPT, but, like, you bring a really good point. I think we might be able to do that in Kersher.
158 00:17:02.120 ⇒ 00:17:07.590 Uttam Kumaran: Okay, so now that we’ve set the stage, we have 12 more minutes, I guess Sam Pranav, can we…
159 00:17:07.940 ⇒ 00:17:12.739 Uttam Kumaran: Brainstorm, like, 10 ideas on, like, how we could speed this up.
160 00:17:12.740 ⇒ 00:17:14.139 Samuel Roberts: And then I can…
161 00:17:14.140 ⇒ 00:17:16.629 Uttam Kumaran: hand it to Amber to go try these out.
162 00:17:19.040 ⇒ 00:17:25.869 Uttam Kumaran: I wrote some at the bottom here. I wrote, once a week, have opus or a reasoning agent, suggest changes based on rules.
163 00:17:26.040 ⇒ 00:17:27.109 Samuel Roberts: So, like.
164 00:17:27.109 ⇒ 00:17:40.209 Uttam Kumaran: I don’t know what the rules for the document are, but we can come up with some document rules. Yeah. The second piece is, like, take the top running queries, like, the longest running queries, and ask the agent what improvements to the doc can we use to make it faster.
165 00:17:41.710 ⇒ 00:17:45.709 Uttam Kumaran: Like, that could be, what improvements to the dock or Supabase?
166 00:17:45.910 ⇒ 00:17:46.850 Samuel Roberts: Right, right.
167 00:17:48.950 ⇒ 00:17:56.489 Pranav Narahari: Yeah, one thing that I thought of is for assessing just, like, how helpful the information is in the central doc is
168 00:17:57.690 ⇒ 00:17:59.690 Pranav Narahari: We should maybe have guidelines
169 00:17:59.880 ⇒ 00:18:07.120 Pranav Narahari: To say, like, what is useful information, what is information that we can really make accurate judgments on.
170 00:18:07.250 ⇒ 00:18:13.380 Pranav Narahari: And so when they’re writing something in there, and maybe you guys have already done this, when they’re writing something in there, they have a guideline of, like.
171 00:18:13.680 ⇒ 00:18:17.239 Pranav Narahari: Is it… does it fall… satisfy, like, X, Y, and Z?
172 00:18:17.410 ⇒ 00:18:24.790 Pranav Narahari: In terms of, like, formatting, I guess, but also… I guess,
173 00:18:25.810 ⇒ 00:18:31.500 Pranav Narahari: I guess tone… like, I don’t know if tone… tone doesn’t matter, but… Yeah, just…
174 00:18:31.770 ⇒ 00:18:38.059 Samuel Roberts: But things that they say shouldn’t be, like, ambiguous, right? Oh, I see, yes, I see, yes.
175 00:18:38.060 ⇒ 00:18:38.670 Pranav Narahari: Yeah.
176 00:18:38.670 ⇒ 00:18:55.480 Uttam Kumaran: Yeah, so I think that would be helpful for, like, guidelines, which is, like, lack of am… like, basically, like, ambiguity scale. We need to come up with a couple of these, which is, like, that’s the guidelines, and then the AI then goes through and makes sure that everything fits the guidelines.
177 00:18:55.550 ⇒ 00:18:59.710 Pranav Narahari: Yeah. I also said, like, we should pick off questions that have no answers.
178 00:18:59.960 ⇒ 00:19:05.719 Uttam Kumaran: So… every week, we should get a list of the responses where it says, I don’t know.
179 00:19:06.050 ⇒ 00:19:10.810 Uttam Kumaran: And… the AI should either identify if it’s missing information.
180 00:19:10.980 ⇒ 00:19:13.670 Uttam Kumaran: Or if it’s, like, information’s there, but, like.
181 00:19:13.930 ⇒ 00:19:16.670 Uttam Kumaran: the AI missed it because it was written wrong, you know?
182 00:19:16.670 ⇒ 00:19:17.360 Samuel Roberts: Right.
183 00:19:19.290 ⇒ 00:19:22.569 Samuel Roberts: Yeah, I’d also add, like, checking for contradictory information.
184 00:19:23.150 ⇒ 00:19:25.410 Uttam Kumaran: Yeah, great, great.
185 00:19:25.660 ⇒ 00:19:26.330 Pranav Narahari: Yeah.
186 00:19:26.330 ⇒ 00:19:32.690 Samuel Roberts: repeats and slight differences, and if they’re… that’s probably related to ambiguity more than anything, but…
187 00:19:33.400 ⇒ 00:19:39.379 Samuel Roberts: I don’t think they’re just straight up writing contradictory things for, like, the point of it, but I bet it’s… it’s misinterpreted for that reason.
188 00:19:39.380 ⇒ 00:19:43.330 Uttam Kumaran: So, finding contradictory information,
189 00:19:49.080 ⇒ 00:19:54.810 Uttam Kumaran: I guess the other thing is, like, how do we prioritize the highest access information?
190 00:19:54.980 ⇒ 00:19:58.619 Uttam Kumaran: And Sam, my question to you is, is that happening in Supabase?
191 00:19:58.990 ⇒ 00:20:06.880 Uttam Kumaran: Or, like, does that need to happen in the doc? For example, some questions are asked more often than others, and I think it’s… there’s, like, an 80-20 here.
192 00:20:07.130 ⇒ 00:20:08.410 Samuel Roberts: So…
193 00:20:08.590 ⇒ 00:20:11.240 Uttam Kumaran: Is there anything we can do to take advantage of that?
194 00:20:11.670 ⇒ 00:20:12.310 Samuel Roberts: Hmm.
195 00:20:14.840 ⇒ 00:20:19.900 Samuel Roberts: Yeah, I mean, I think at this point, it’s just straight up a RAG search without any kind of…
196 00:20:20.950 ⇒ 00:20:24.520 Samuel Roberts: like… Prioritization that way.
197 00:20:24.520 ⇒ 00:20:25.350 Uttam Kumaran: There’s no renting.
198 00:20:25.930 ⇒ 00:20:31.909 Samuel Roberts: No, I mean, that’s… I mean, that’s kind of why the ZipsDB came to be, because that was a source where this was a problem, you know, like…
199 00:20:32.870 ⇒ 00:20:35.410 Samuel Roberts: Just the rag over the documents wasn’t good enough.
200 00:20:37.960 ⇒ 00:20:44.860 Samuel Roberts: But I’m sure there’s other information that’s probably, you’re right, like, high… high query… Easy.
201 00:20:45.040 ⇒ 00:20:50.810 Samuel Roberts: We’ve done some of that with the cancellations and things. We did… Do a little… yeah.
202 00:20:52.000 ⇒ 00:20:53.190 Uttam Kumaran: Because…
203 00:20:54.580 ⇒ 00:20:59.210 Samuel Roberts: The prompt got huge, and part of that was because certain things weren’t getting responded properly.
204 00:21:00.280 ⇒ 00:21:07.730 Samuel Roberts: And one of those things was, like, cancellation flows. So one thing we’ve done with the new one is make sub-agents that handle some of these things differently, which is…
205 00:21:07.860 ⇒ 00:21:09.579 Samuel Roberts: A little bit of the prioritization.
206 00:21:09.890 ⇒ 00:21:10.570 Uttam Kumaran: Okay.
207 00:21:10.840 ⇒ 00:21:12.300 Samuel Roberts: But I bet we could do more.
208 00:21:15.490 ⇒ 00:21:17.640 Samuel Roberts: Even based on the periods over time.
209 00:21:20.600 ⇒ 00:21:22.299 Uttam Kumaran: So there’s some type of ranking.
210 00:21:22.300 ⇒ 00:21:22.990 Samuel Roberts: Yeah.
211 00:21:26.220 ⇒ 00:21:29.329 Samuel Roberts: Yeah, there were also, like, there were specific things that they needed,
212 00:21:29.900 ⇒ 00:21:33.410 Samuel Roberts: Like, verbatim responses that we were working on recently.
213 00:21:33.640 ⇒ 00:21:34.460 Samuel Roberts: I was like.
214 00:21:35.010 ⇒ 00:21:42.269 Samuel Roberts: The RAG is good for, like, what does the doc say about this, and it can tell you, but sometimes it’s, like, you have to follow this script for something.
215 00:21:42.690 ⇒ 00:21:46.340 Samuel Roberts: And so some of that has been kind of pulled out of those docs.
216 00:21:46.460 ⇒ 00:21:49.020 Samuel Roberts: Or moved around a little bit into the agents.
217 00:21:49.630 ⇒ 00:21:51.020 Uttam Kumaran: Oh, okay.
218 00:21:51.880 ⇒ 00:21:56.950 Samuel Roberts: Which, I guess, is a little bit of prioritization, because we basically have a different branch… a branching flow, almost.
219 00:21:59.770 ⇒ 00:22:05.549 Pranav Narahari: One thing that I was just thinking of is, like, with this rag system, since you’re saying, Tom, it’s, like, kind of like an 80-20,
220 00:22:05.710 ⇒ 00:22:15.120 Pranav Narahari: The content should probably also mirror that in some ways, where if it’s an 80-20 of, like, we’re asking specifically 80%, like, mechanical stuff.
221 00:22:16.550 ⇒ 00:22:23.249 Pranav Narahari: we should have probably 80% of that central dock, more or less, to have mechanical information. Is that, like, an accurate…
222 00:22:23.660 ⇒ 00:22:33.070 Uttam Kumaran: I guess it’s, like, it may not need to happen there, it can actually probably happen in Supabase, where you’re like, hey, these chunks are accessed more often than others.
223 00:22:34.430 ⇒ 00:22:35.700 Uttam Kumaran: consider them.
224 00:22:36.600 ⇒ 00:22:39.290 Uttam Kumaran: Or, like, cash them or something. I don’t know.
225 00:22:39.290 ⇒ 00:22:39.710 Samuel Roberts: Yeah.
226 00:22:39.710 ⇒ 00:22:43.970 Uttam Kumaran: Like, there’s this concept in RAG called re-ranking. Contextual does a good job with this.
227 00:22:44.240 ⇒ 00:22:45.020 Pranav Narahari: Oh, okay.
228 00:22:49.950 ⇒ 00:22:56.030 Uttam Kumaran: I’ll send it to you, but, yeah, basically…
229 00:22:56.800 ⇒ 00:23:03.810 Uttam Kumaran: You want to, like… yeah, you basically want to, like… Rank, depending on…
230 00:23:05.260 ⇒ 00:23:11.360 Uttam Kumaran: You could rank on any sort of number, but, like, you can rank based on the frequency of access.
231 00:23:12.540 ⇒ 00:23:12.880 Pranav Narahari: Okay.
232 00:23:12.880 ⇒ 00:23:13.630 Samuel Roberts: That’s good.
233 00:23:13.630 ⇒ 00:23:17.430 Uttam Kumaran: And basically, we would update, like, the rank as we go.
234 00:23:20.420 ⇒ 00:23:22.059 Uttam Kumaran: So there’s something there.
235 00:23:23.310 ⇒ 00:23:29.890 Pranav Narahari: Yeah, so, like, when we’re also… Getting the most, like…
236 00:23:30.020 ⇒ 00:23:33.279 Pranav Narahari: Relevant embeddings for the specific prompt.
237 00:23:33.420 ⇒ 00:23:39.489 Pranav Narahari: there’s probably a score… there’s a score associated with that, too. We can make sure that that score is, like.
238 00:23:39.690 ⇒ 00:23:41.270 Pranav Narahari: Higher than a certain…
239 00:23:41.890 ⇒ 00:23:42.450 Samuel Roberts: Yeah.
240 00:23:42.450 ⇒ 00:23:46.590 Pranav Narahari: certain percentage. So, like, that similarity search is essentially, like.
241 00:23:47.410 ⇒ 00:23:52.169 Pranav Narahari: higher than, like, 90%, or if, like, for this…
242 00:23:52.310 ⇒ 00:23:59.319 Pranav Narahari: There’s probably maybe gonna be certain cases where it’s lower, but if we’re thinking about it, like, from an 80-20 point of view, like…
243 00:23:59.800 ⇒ 00:24:06.790 Pranav Narahari: We need to make sure that the bulk of what they’re asking, like, we have a lot of relevant information about that in the central doc.
244 00:24:07.130 ⇒ 00:24:11.930 Pranav Narahari: And so if we’re consistently, like, not bringing in accurate,
245 00:24:12.180 ⇒ 00:24:16.799 Pranav Narahari: accurate data from the source of truth document, then that’s gonna be an issue.
246 00:24:18.410 ⇒ 00:24:23.249 Samuel Roberts: Yeah, I’m pretty sure that’s being… like, the threshold is there in some form. I think what.
247 00:24:23.250 ⇒ 00:24:23.800 Pranav Narahari: Okay.
248 00:24:24.460 ⇒ 00:24:29.389 Samuel Roberts: interesting, and I don’t know exactly how this worked on N8N, but, like, re…
249 00:24:29.680 ⇒ 00:24:36.219 Samuel Roberts: Like, responding with a question was a little bit difficult, like, clarifying things when there is ambiguity, for example.
250 00:24:36.550 ⇒ 00:24:38.980 Samuel Roberts: is something we can do better now with the Moss Room.
251 00:24:39.860 ⇒ 00:24:44.979 Samuel Roberts: that NAM wasn’t good at, so we might be able to make use of that threshold and make it even tighter now, maybe?
252 00:24:46.180 ⇒ 00:24:46.840 Pranav Narahari: Okay.
253 00:24:47.130 ⇒ 00:24:49.919 Samuel Roberts: Which is a good thought. That’s a good point.
254 00:24:57.060 ⇒ 00:25:05.640 Uttam Kumaran: Okay, so I think a couple of next steps. One is, like, I… can we, like, Sam, are we saving all the questions being asked?
255 00:25:06.020 ⇒ 00:25:06.929 Uttam Kumaran: in supervision.
256 00:25:06.930 ⇒ 00:25:11.150 Samuel Roberts: It’s all… it’s all logged to… Snowflake, I believe.
257 00:25:11.430 ⇒ 00:25:12.270 Samuel Roberts: Okay.
258 00:25:12.730 ⇒ 00:25:13.360 Uttam Kumaran: So…
259 00:25:13.360 ⇒ 00:25:23.969 Samuel Roberts: So that’s where we’ve pulled some of the thumbs up, thumbs-down stuff, so I’m sure we can pull just, like, questions, you know, regardless of how the response was. We’ve been looking a lot more at how the agent responds, but…
260 00:25:23.970 ⇒ 00:25:28.079 Uttam Kumaran: Are you just logging the question, or you’re not… you’re not categorizing it on log?
261 00:25:28.590 ⇒ 00:25:45.369 Samuel Roberts: I don’t actually know what’s happening in Snowflake with that. I think… I think we were categorizing it after, so I don’t think it was. There’s a few things we’ve been talking about adding now that we’re gonna be logging from Austria. I mean, that… I don’t know how that was set up at the time, that was before me, but there were things missing from that Snowflake logging, like, even just, like.
262 00:25:47.420 ⇒ 00:25:50.249 Samuel Roberts: Execution time, like, the actual…
263 00:25:50.250 ⇒ 00:25:50.670 Uttam Kumaran: Okay.
264 00:25:50.670 ⇒ 00:25:56.660 Samuel Roberts: date time. We had to, like, parse that out of the name of the run and everything, so there’s a lot more that can be cleaned up there.
265 00:25:57.250 ⇒ 00:26:00.110 Samuel Roberts: One has been cleaned up a little bit, there’s a little bit more in there now.
266 00:26:00.950 ⇒ 00:26:04.309 Samuel Roberts: But yeah, I don’t think categorization’s happening. We’ve talked about doing that on, like, a…
267 00:26:04.470 ⇒ 00:26:07.289 Samuel Roberts: a cadence, as well, for thumbs up, thumbs down stuff.
268 00:26:08.380 ⇒ 00:26:09.110 Uttam Kumaran: Okay.
269 00:26:09.110 ⇒ 00:26:12.340 Samuel Roberts: We could probably do that for just sort of categorization of questions.
270 00:26:13.010 ⇒ 00:26:23.150 Uttam Kumaran: And then my other suggestion, Amber, after this meeting, is use Opus and GPT 5.2, basically within,
271 00:26:23.320 ⇒ 00:26:28.980 Uttam Kumaran: within cursor, with the Andy repo open.
272 00:26:29.400 ⇒ 00:26:34.110 Uttam Kumaran: Send this transcript plus these notes.
273 00:26:34.550 ⇒ 00:26:39.829 Uttam Kumaran: And ask those agents to help you brainstorm further.
274 00:26:40.370 ⇒ 00:26:40.990 Samuel Roberts: Yeah.
275 00:26:44.020 ⇒ 00:26:47.199 Amber Lin: I was on mute. Cool. That sounds good.
276 00:26:47.800 ⇒ 00:26:55.169 Uttam Kumaran: Yeah, I would say if you can do that, and just paste in anything, any ideas that it gives you, we’ll have a full idea list, and then…
277 00:26:55.480 ⇒ 00:26:55.890 Amber Lin: Okay.
278 00:26:55.890 ⇒ 00:26:57.449 Uttam Kumaran: like, we can ticket it out. Yeah, some of the.
279 00:26:57.450 ⇒ 00:26:58.030 Amber Lin: Sounds good.
280 00:26:58.030 ⇒ 00:27:01.830 Uttam Kumaran: Some of these, it should… I just want to ask it if it has any other ideas.
281 00:27:01.830 ⇒ 00:27:07.310 Samuel Roberts: Yeah, there’s probably maybe something that’s just, like, even… Okay. I…
282 00:27:07.400 ⇒ 00:27:14.399 Amber Lin: I told the client that I’m gonna send them, like, a guideline on what they can help in this process.
283 00:27:15.110 ⇒ 00:27:24.879 Amber Lin: Do you think we can… if I do this today, do you think we can have a meeting, like, tomorrow, or do it async, so I can send them, like, any guidelines for them to do?
284 00:27:26.360 ⇒ 00:27:33.280 Uttam Kumaran: Oh, yeah, yeah, yeah, yeah. Let’s see the… let’s see the range of responses first. I mean, from my sense, I feel like…
285 00:27:33.390 ⇒ 00:27:36.060 Uttam Kumaran: I don’t want them to do anything, like…
286 00:27:36.170 ⇒ 00:27:44.810 Uttam Kumaran: If we can just get AI to help us reformat, then all they have to do is get the shit in there, and maybe once a week, we build something that helps us do this.
287 00:27:45.400 ⇒ 00:27:47.560 Amber Lin: Focus a lot less on…
288 00:27:47.560 ⇒ 00:27:49.449 Uttam Kumaran: Focus a lot less on the how.
289 00:27:49.690 ⇒ 00:27:53.619 Uttam Kumaran: Right now, I’ll… we’ll figure that out for you. I think…
290 00:27:53.620 ⇒ 00:27:53.990 Amber Lin: Sounds.
291 00:27:53.990 ⇒ 00:28:00.979 Uttam Kumaran: Let’s just get as many ideas here, and then me, Sam, and Pranav will rank them in terms of impact and speed, and then…
292 00:28:01.350 ⇒ 00:28:03.460 Uttam Kumaran: We’ll, we’ll, we’ll, we’ll execute.
293 00:28:03.740 ⇒ 00:28:17.240 Amber Lin: Cool, okay. Like, the only thing I think we need their involvement is, like, some things that were very confu- like, in a very ambiguous way, and we need to ask them to clarify, or else, like, we’re not gonna get good
294 00:28:17.460 ⇒ 00:28:24.249 Amber Lin: good, optimized formatting or words out of it, because, like, we need their input there. Like, other than that…
295 00:28:24.250 ⇒ 00:28:31.089 Uttam Kumaran: That’s idea one, which is, like, find out of guideline Like, content, right?
296 00:28:34.930 ⇒ 00:28:35.920 Samuel Roberts: Battling? What?
297 00:28:36.120 ⇒ 00:28:37.140 Samuel Roberts: Don’t follow.
298 00:28:37.140 ⇒ 00:28:39.849 Uttam Kumaran: This is the guidelines, like, formatting.
299 00:28:39.850 ⇒ 00:28:43.799 Samuel Roberts: Oh, out of guideline content. Sorry, I originally just find out…
300 00:28:43.940 ⇒ 00:28:47.300 Samuel Roberts: Oh. And I was like, what does that mean? No, I… yeah, no, no.
301 00:28:47.300 ⇒ 00:28:51.249 Uttam Kumaran: Basically, like, find, like, it’s not adhere… things that are out of policy.
302 00:28:51.630 ⇒ 00:28:52.849 Samuel Roberts: Yeah, exactly, exactly.
303 00:28:52.850 ⇒ 00:28:58.059 Uttam Kumaran: So we’ll… so basically, Amber, we’ll end up creating something that’s, like, a policy adherence sort of flow.
304 00:28:58.060 ⇒ 00:29:00.639 Amber Lin: That identifies things that are out of policy.
305 00:29:00.640 ⇒ 00:29:04.389 Uttam Kumaran: And then ideally, either can fix it, or it needs to flag it up.
306 00:29:04.810 ⇒ 00:29:12.690 Uttam Kumaran: And then my last question is, where is that at? Sam, can we just continue to build more tools on the admin UI? Is that hosted… where is it hosted?
307 00:29:12.940 ⇒ 00:29:15.770 Samuel Roberts: Yeah, I mean, right now that’s on Heroku.
308 00:29:16.230 ⇒ 00:29:27.590 Samuel Roberts: This was something I talked about a while ago, like, just, like, another UI for some of this stuff, because, you know, not that Google is bad, but, like, were there other things I was thinking, and then I was thinking contextual, and we gave them that demo at one point.
309 00:29:28.850 ⇒ 00:29:33.899 Samuel Roberts: So there were… yeah, we could… we could put more there. It just was sort of… we weren’t… like, that was…
310 00:29:34.290 ⇒ 00:29:41.600 Samuel Roberts: out of necessity, because the forms weren’t working on NDN well. But if there’s more we want to add to that admin UI and host, we could certainly do it.
311 00:29:42.990 ⇒ 00:29:45.670 Uttam Kumaran: Can we host it on Google… on their Google, or no?
312 00:29:45.670 ⇒ 00:29:50.469 Samuel Roberts: Yeah, I mean, we just haven’t moved yet. You know, that was set up before we started moving to GCP stuff, but now that we’ve got access.
313 00:29:50.470 ⇒ 00:29:51.250 Uttam Kumaran: Okay.
314 00:29:51.250 ⇒ 00:29:53.970 Samuel Roberts: Everything’s over there. It’s all in the same repo, I’m pretty sure, so…
315 00:29:54.310 ⇒ 00:29:54.750 Uttam Kumaran: Okay.
316 00:29:54.750 ⇒ 00:29:59.739 Amber Lin: Cool. Who’s doing step one? Like, Sam, can you do the export, or can you add.
317 00:29:59.740 ⇒ 00:30:04.149 Uttam Kumaran: I can… I wanna… let’s do this first, Amber.
318 00:30:04.150 ⇒ 00:30:04.740 Amber Lin: Excellent.
319 00:30:05.110 ⇒ 00:30:09.649 Uttam Kumaran: Yeah, so if you could do this, then I just want to get all the ideas down.
320 00:30:10.060 ⇒ 00:30:10.600 Samuel Roberts: Yeah.
321 00:30:11.420 ⇒ 00:30:14.109 Uttam Kumaran: So this is the complete blocking step right now.
322 00:30:14.110 ⇒ 00:30:15.149 Amber Lin: Okay, cool, sounds good.
323 00:30:15.150 ⇒ 00:30:18.080 Uttam Kumaran: And then… and then we’ll… let’s… let’s, like, nail this async today, yeah.
324 00:30:18.080 ⇒ 00:30:18.850 Amber Lin: Cool, okay.
325 00:30:18.850 ⇒ 00:30:19.390 Samuel Roberts: Cool.
326 00:30:20.160 ⇒ 00:30:20.830 Uttam Kumaran: Okay.
327 00:30:21.040 ⇒ 00:30:21.930 Uttam Kumaran: Alright. Thank you, everyone.
328 00:30:21.930 ⇒ 00:30:23.239 Amber Lin: I’ll ping you later. Thanks.
329 00:30:23.240 ⇒ 00:30:24.169 Uttam Kumaran: Okay, okay. Bye.
330 00:30:24.170 ⇒ 00:30:24.560 Samuel Roberts: Sounds good.
331 00:30:24.910 ⇒ 00:30:25.470 Amber Lin: I…
332 00:30:25.600 ⇒ 00:30:26.180 Uttam Kumaran: Aye.