Meeting Title: ABC Project Daily Stand-up Date: 2025-12-15 Meeting participants: Amber Lin, Mustafa Raja, Samuel Roberts, Uttam Kumaran
WEBVTT
1 00:00:15.710 ⇒ 00:00:16.490 Amber Lin: Okay.
2 00:00:19.080 ⇒ 00:00:19.940 Uttam Kumaran: Okay.
3 00:00:20.230 ⇒ 00:00:21.850 Uttam Kumaran: Is everyone here?
4 00:00:22.240 ⇒ 00:00:22.720 Samuel Roberts: Hmm, yeah.
5 00:00:22.720 ⇒ 00:00:23.270 Mustafa Raja: Yeah.
6 00:00:24.010 ⇒ 00:00:25.540 Uttam Kumaran: Okay,
7 00:00:25.690 ⇒ 00:00:32.660 Uttam Kumaran: So yeah, I mean, I’m sort of, like, I think I’m just gonna try to lead, like, well, I’m just gonna have to lead a daily thing on this until it’s fixed.
8 00:00:32.970 ⇒ 00:00:38.229 Uttam Kumaran: So what… can someone walk me through, first, like.
9 00:00:38.720 ⇒ 00:00:44.880 Uttam Kumaran: how this process works end-to-end. I have a good idea, but…
10 00:00:45.050 ⇒ 00:00:57.140 Uttam Kumaran: I just want to confirm, like, what… where the data is getting inputted from this, and how it’s flowing. So that’s, like, one thing. I think the second item here is we have…
11 00:00:57.330 ⇒ 00:01:00.980 Uttam Kumaran: We’ve not given them any confidence, that…
12 00:01:01.130 ⇒ 00:01:07.960 Uttam Kumaran: like, this gets fixed, and that they have a clear understanding that it is fixed. So I…
13 00:01:08.130 ⇒ 00:01:12.849 Uttam Kumaran: I’m not sure why we haven’t just, like, ran the thing every morning.
14 00:01:12.970 ⇒ 00:01:16.149 Uttam Kumaran: To show that… This is still working, but…
15 00:01:16.390 ⇒ 00:01:19.569 Uttam Kumaran: Again, like, customers are finding out that there’s issues, so…
16 00:01:19.680 ⇒ 00:01:24.029 Uttam Kumaran: Do we even, like, do we have a controlled data set at all, like, where we can test
17 00:01:24.450 ⇒ 00:01:27.420 Uttam Kumaran: You submit a question, you get an answer out, and is it…
18 00:01:28.430 ⇒ 00:01:31.420 Uttam Kumaran: Correct? Like, are we still running evals, and…
19 00:01:31.610 ⇒ 00:01:35.519 Uttam Kumaran: Is there anything that’s coming out of those evals that’s showing that this is inaccurate?
20 00:01:36.210 ⇒ 00:01:39.820 Amber Lin: Our eval says not… we’ve not been paying attention to evals.
21 00:01:41.100 ⇒ 00:01:42.150 Uttam Kumaran: Oh, why not?
22 00:01:42.710 ⇒ 00:01:48.790 Amber Lin: Oh, oh, I just… that was the current state. Not saying that it’s correct, we’ve just not been doing it.
23 00:01:49.660 ⇒ 00:01:50.200 Uttam Kumaran: Okay.
24 00:01:50.200 ⇒ 00:01:53.739 Samuel Roberts: What do you guys are even set up on this N8N is my question right now, too.
25 00:01:55.580 ⇒ 00:01:57.790 Uttam Kumaran: Okay, I mean, but if Sam doesn’t even know, then it’s.
26 00:01:57.790 ⇒ 00:02:04.400 Samuel Roberts: Yeah, no, I… that’s why I’m not even sure what was here. Especially if they’re changing the database and stuff, I’m not sure…
27 00:02:04.690 ⇒ 00:02:12.400 Samuel Roberts: what that data said… and then my real question is, why is… so, two different users are getting two different responses? Is that what we just saw?
28 00:02:14.440 ⇒ 00:02:17.360 Uttam Kumaran: Yeah, that seems like… it seems like two different people are getting…
29 00:02:17.860 ⇒ 00:02:22.710 Uttam Kumaran: two different responses, but also, I don’t think it’s, like…
30 00:02:23.190 ⇒ 00:02:30.359 Uttam Kumaran: So I guess, like, maybe let’s start by just walking through… the system, like, Is that something…
31 00:02:30.940 ⇒ 00:02:36.259 Uttam Kumaran: Mustafa, you can do, or Sam, you can do, to walk through this part of how this works.
32 00:02:44.220 ⇒ 00:02:44.830 Mustafa Raja: Okay.
33 00:02:45.060 ⇒ 00:03:00.259 Mustafa Raja: Okay, so the issues that they’ve been having are, are with the… with this QueryDB tool, and, how it works is, whenever a user, asks a question over here in Andy,
34 00:03:01.110 ⇒ 00:03:12.579 Mustafa Raja: This webhook gets triggered, and this is the router, that, that just tells if, if the question is for a department. If so, then which department.
35 00:03:12.610 ⇒ 00:03:24.479 Mustafa Raja: And then it comes to Andy, and then if it’s a question for zip codes, we just go into, this, DB tool that lives in…
36 00:03:24.790 ⇒ 00:03:25.900 Mustafa Raja: Windmill.
37 00:03:27.190 ⇒ 00:03:29.590 Mustafa Raja: And that sort of gives the answer.
38 00:03:30.470 ⇒ 00:03:34.650 Uttam Kumaran: Do you have traces on the question that Janiece asked? Can you pull up that.
39 00:03:34.650 ⇒ 00:03:42.180 Mustafa Raja: Yes, let me actually… Open up Slack… And then here…
40 00:03:44.160 ⇒ 00:03:45.549 Uttam Kumaran: NQs, or where is that?
41 00:03:46.540 ⇒ 00:03:48.190 Mustafa Raja: Sorry, what?
42 00:03:48.980 ⇒ 00:03:51.320 Uttam Kumaran: Is that in length use, or is that still an N8M?
43 00:03:52.870 ⇒ 00:04:02.040 Mustafa Raja: So tracing, yeah, tracing, I believe, is still in NHN, because we cannot, you know, connect LangFuse.
44 00:04:02.040 ⇒ 00:04:02.390 Uttam Kumaran: So.
45 00:04:02.390 ⇒ 00:04:06.820 Mustafa Raja: Let’s pull up… let’s pull up the trace for this issue.
46 00:04:07.560 ⇒ 00:04:08.550 Uttam Kumaran: And…
47 00:04:08.550 ⇒ 00:04:10.920 Mustafa Raja: So… 2028…
48 00:04:25.760 ⇒ 00:04:35.570 Amber Lin: I put the issue Janiece was referring to in the chat. It’s from Bo Jenkins, if you can find that user. Yes, yeah, I’m…
49 00:04:35.570 ⇒ 00:04:37.830 Mustafa Raja: On December 16th.
50 00:04:39.040 ⇒ 00:04:39.970 Amber Lin: Huh, it does.
51 00:04:39.970 ⇒ 00:04:42.650 Mustafa Raja: Yeah, December 16th, around… around 28th.
52 00:04:42.650 ⇒ 00:04:43.480 Amber Lin: Oh, okay.
53 00:04:43.650 ⇒ 00:04:50.830 Mustafa Raja: I’m just seeing… yeah, this one. So… hmm… Okay.
54 00:04:53.080 ⇒ 00:04:55.639 Mustafa Raja: Okay, this should be prior to this…
55 00:05:02.640 ⇒ 00:05:04.349 Mustafa Raja: This is me…
56 00:05:10.730 ⇒ 00:05:18.169 Mustafa Raja: Okay, I need to schedule… is it this one? Yeah, so…
57 00:05:18.420 ⇒ 00:05:26.050 Mustafa Raja: Here, Bo, here Bo said, Bo’s feedback was… This is wrong. So…
58 00:05:26.050 ⇒ 00:05:28.349 Uttam Kumaran: Yeah, I don’t care about the feedback much, we already know.
59 00:05:28.350 ⇒ 00:05:30.340 Samuel Roberts: Yeah, we gotta see that, we gotta see that actual…
60 00:05:30.340 ⇒ 00:05:38.349 Uttam Kumaran: Look at the… look at the QBD… look at the QB… yeah, the database tool, and let’s check, like, how it pulled and, like, what it actually pulled.
61 00:05:42.340 ⇒ 00:05:49.309 Mustafa Raja: Yeah, so… Let me actually… Logs in here, too.
62 00:06:15.940 ⇒ 00:06:16.700 Samuel Roberts: ecosystem.
63 00:06:19.970 ⇒ 00:06:20.780 Samuel Roberts: Oops.
64 00:06:24.820 ⇒ 00:06:25.610 Samuel Roberts: Hmm.
65 00:06:54.680 ⇒ 00:06:57.160 Mustafa Raja: Yeah, this one simply goes on.
66 00:06:58.160 ⇒ 00:06:59.130 Mustafa Raja: Longs.
67 00:07:03.350 ⇒ 00:07:05.850 Mustafa Raja: Okay, so we don’t have much in logs.
68 00:07:12.380 ⇒ 00:07:14.890 Mustafa Raja: Thanks, yeah, so for this.
69 00:07:14.890 ⇒ 00:07:17.330 Uttam Kumaran: But I also don’t get, like, why is this in…
70 00:07:17.500 ⇒ 00:07:19.340 Uttam Kumaran: Like, why is this in windmill?
71 00:07:24.860 ⇒ 00:07:29.240 Samuel Roberts: This was because, Casey was using Langchains to do the text-to-SQL.
72 00:07:30.580 ⇒ 00:07:31.070 Mustafa Raja: Yeah.
73 00:07:31.070 ⇒ 00:07:33.549 Samuel Roberts: NAN doesn’t let us really do that.
74 00:07:34.580 ⇒ 00:07:35.230 Uttam Kumaran: Okay.
75 00:07:35.380 ⇒ 00:07:37.589 Samuel Roberts: So this was, yeah, this was hitting this tool.
76 00:07:37.720 ⇒ 00:07:42.139 Samuel Roberts: Which again, like, It’s probably… I don’t… I’m wondering, is that…
77 00:07:43.720 ⇒ 00:07:48.730 Uttam Kumaran: So, let’s… can you show the exact query that was run, Mustafa?
78 00:07:48.730 ⇒ 00:07:51.130 Mustafa Raja: Yeah, this, it’s this right now.
79 00:07:51.130 ⇒ 00:07:53.569 Uttam Kumaran: And what was the question he asked?
80 00:07:54.530 ⇒ 00:07:59.739 Mustafa Raja: Mosquito misting system estimate in Corpus.
81 00:08:02.080 ⇒ 00:08:02.920 Uttam Kumaran: And so…
82 00:08:02.920 ⇒ 00:08:04.420 Mustafa Raja: No, that’s what was…
83 00:08:04.420 ⇒ 00:08:08.270 Samuel Roberts: That was what it… that was what it figured out it needed to ask, what was he actually asking?
84 00:08:09.740 ⇒ 00:08:12.070 Samuel Roberts: I need to schedule a mosquito Missing System Estimate Incorporated.
85 00:08:12.070 ⇒ 00:08:13.210 Mustafa Raja: Oh, okay, okay.
86 00:08:13.210 ⇒ 00:08:17.499 Samuel Roberts: No, I’m saying, I think I’m just making sure, that’s what… that’s what the input was here. What’s the…
87 00:08:18.350 ⇒ 00:08:19.929 Uttam Kumaran: Yo, what’s the user input?
88 00:08:20.650 ⇒ 00:08:23.110 Mustafa Raja: It says, I need to schedule the… this…
89 00:08:24.220 ⇒ 00:08:33.510 Uttam Kumaran: And so… I need to schedule a mosquito missing system, and so what is the expected call?
90 00:08:36.909 ⇒ 00:08:37.519 Mustafa Raja: Oh, yeah.
91 00:08:37.520 ⇒ 00:08:38.329 Uttam Kumaran: How is that, how is that.
92 00:08:40.069 ⇒ 00:08:47.609 Uttam Kumaran: No, no, no, like, you tell me, like, what is that supposed… like, how is that supposed… like, let’s say you were doing it manually, like, how is that supposed to get answered?
93 00:08:52.159 ⇒ 00:08:53.619 Uttam Kumaran: No, no, no, Mustafa.
94 00:08:53.909 ⇒ 00:08:59.039 Uttam Kumaran: What is the actual answer to the question? Like, what is the… what is the control answer?
95 00:08:59.569 ⇒ 00:09:03.219 Uttam Kumaran: I don’t care what the query is, like, what… how do you… how would you find it manually?
96 00:09:04.009 ⇒ 00:09:05.950 Amber Lin: Like, do we have an expected correction?
97 00:09:05.950 ⇒ 00:09:07.130 Uttam Kumaran: Dancer for that?
98 00:09:08.530 ⇒ 00:09:13.440 Amber Lin: We’ll have to manually find it in the inspector spreadsheet as well.
99 00:09:16.050 ⇒ 00:09:18.269 Uttam Kumaran: Okay, I mean, I just want to know, like, what…
100 00:09:18.520 ⇒ 00:09:24.809 Uttam Kumaran: Given that question, like, what is the answer we should have given, and what is the answer we did give? Very simple.
101 00:09:31.340 ⇒ 00:09:33.280 Mustafa Raja: Yeah, I haven’t been working on,
102 00:09:33.280 ⇒ 00:09:34.000 Samuel Roberts: Yeah.
103 00:09:34.000 ⇒ 00:09:34.850 Mustafa Raja: So…
104 00:09:34.850 ⇒ 00:09:40.229 Samuel Roberts: I’m running that query now, trying to see what we can get back, and try to actually find the right answer here. Hold on.
105 00:09:40.230 ⇒ 00:09:40.820 Uttam Kumaran: Okay.
106 00:09:41.050 ⇒ 00:09:42.289 Samuel Roberts: I’m just actually…
107 00:09:42.290 ⇒ 00:09:48.430 Uttam Kumaran: I’m just… I’m just actually walking into ABC to go get some files, so I’ll just be… I’ll be back on, like, 5 minutes.
108 00:09:48.560 ⇒ 00:09:56.049 Uttam Kumaran: But yeah, let’s just see if we can find… let’s see if we can hold these two examples, and then, basically, I want to find out…
109 00:09:56.260 ⇒ 00:10:05.180 Uttam Kumaran: for these two examples, what is… like, what is the question actually saying? Like, was it an input issue? And then second, I want to find out whether
110 00:10:05.340 ⇒ 00:10:10.629 Uttam Kumaran: It wrote the query correctly, and then if the query was written correctly.
111 00:10:10.760 ⇒ 00:10:13.339 Uttam Kumaran: Then it’s the… whatever the result is like, so…
112 00:10:13.340 ⇒ 00:10:13.720 Samuel Roberts: It’s been…
113 00:10:13.720 ⇒ 00:10:15.199 Uttam Kumaran: Okay, I’ll be right, I’ll be right back.
114 00:10:15.200 ⇒ 00:10:15.609 Samuel Roberts: Okay, cool.
115 00:10:15.930 ⇒ 00:10:17.140 Mustafa Raja: Okay.
116 00:10:17.480 ⇒ 00:10:21.879 Samuel Roberts: Okay, so I’ve got it here, so I was able to… let me share this, too.
117 00:10:23.910 ⇒ 00:10:28.729 Samuel Roberts: So, what I just did was I took that query, someone else is sharing…
118 00:10:29.510 ⇒ 00:10:38.939 Samuel Roberts: Just gonna try to share my screen, okay? Okay. So this was the… the actual SQL query that got generated.
119 00:10:39.630 ⇒ 00:10:48.779 Samuel Roberts: When I run that, I get nothing, and I’m… what I did was I just started removing these AND, Corpus, and inspector, and it’s… I’m noticing here it’s all technicians.
120 00:10:49.590 ⇒ 00:10:51.300 Samuel Roberts: On mosquito nesting here.
121 00:10:51.790 ⇒ 00:10:52.780 Amber Lin: Yeah.
122 00:10:53.560 ⇒ 00:10:54.860 Amber Lin: So…
123 00:10:55.390 ⇒ 00:10:58.669 Samuel Roberts: Is that a thing that is an inspector or a technician?
124 00:10:58.670 ⇒ 00:11:00.900 Amber Lin: That should be an inspector.
125 00:11:01.260 ⇒ 00:11:03.159 Samuel Roberts: Who should? Like, all of these?
126 00:11:03.590 ⇒ 00:11:10.359 Amber Lin: They’re asking specifically for mosquito misting inspector.
127 00:11:10.610 ⇒ 00:11:12.910 Samuel Roberts: Yeah, let me get rid of this limit here and see.
128 00:11:13.820 ⇒ 00:11:18.150 Samuel Roberts: How many we got? Oh, okay, that’s inspected, but yeah, so there are no inspectors, is what I’m finding.
129 00:11:20.790 ⇒ 00:11:21.220 Amber Lin: Aww.
130 00:11:21.220 ⇒ 00:11:36.939 Mustafa Raja: So we have… we have the wrong answer for two of them. One is… the other question is, who should we schedule a holiday light estimate? And this is what Janice mentioned, in her meeting also.
131 00:11:40.030 ⇒ 00:11:46.440 Mustafa Raja: Is the missing one really, really coming… should be coming from, DB, though?
132 00:11:47.040 ⇒ 00:11:52.479 Samuel Roberts: It is coming from the DB, but there are no inspectors that do mosquito misting, it’s only technicians, is what I’m.
133 00:11:52.480 ⇒ 00:11:59.430 Mustafa Raja: I mean, the… the question here was… If I go back…
134 00:11:59.850 ⇒ 00:12:03.220 Amber Lin: Right, sorry, let me show…
135 00:12:03.220 ⇒ 00:12:10.029 Mustafa Raja: I need to schedule… I need to… so the question here was, I need to schedule a misting system estimate, right? So…
136 00:12:10.280 ⇒ 00:12:16.390 Mustafa Raja: It shouldn’t… Should this be pulling up people, or should this be, going into… what’s it called?
137 00:12:16.390 ⇒ 00:12:28.819 Amber Lin: it should be pulling up people. I think, one, he didn’t ask it in a very standard way. He didn’t say, who is the mosquito system inspector in X zip code, or in this area.
138 00:12:28.820 ⇒ 00:12:29.810 Samuel Roberts: Yeah, what was the plan?
139 00:12:29.810 ⇒ 00:12:43.750 Amber Lin: Like, this is the answer, essentially. I think it essentially is that everybody can do mosquito system estimates, and I think when we did it, we already put… we did put it in, and people can do mosquito system.
140 00:12:43.810 ⇒ 00:12:50.080 Amber Lin: Sam, would you try to change the query into mosquito systems to see what… what changes that?
141 00:12:50.280 ⇒ 00:12:53.250 Samuel Roberts: Mosquito system? Is that what you said?
142 00:12:53.250 ⇒ 00:12:53.830 Amber Lin: Yeah.
143 00:12:55.920 ⇒ 00:13:01.839 Samuel Roberts: Oops, hold on, actually, let me search mosquito and see what shows up here.
144 00:13:02.740 ⇒ 00:13:06.509 Samuel Roberts: Okay, then I see inspectors for residential mosquito system estimates.
145 00:13:06.510 ⇒ 00:13:08.680 Amber Lin: Yeah, could you share your screen?
146 00:13:08.680 ⇒ 00:13:09.360 Samuel Roberts: Yeah.
147 00:13:12.030 ⇒ 00:13:16.099 Samuel Roberts: Yeah, let me jump back over. Okay, so it is here.
148 00:13:16.950 ⇒ 00:13:19.990 Samuel Roberts: So I just got rid of, misting here.
149 00:13:20.870 ⇒ 00:13:27.340 Samuel Roberts: And went back and did… ran this, and this is what I got. So there are inspectors that do residential mosquito system estimates.
150 00:13:28.490 ⇒ 00:13:29.240 Amber Lin: Yeah.
151 00:13:29.240 ⇒ 00:13:30.280 Samuel Roberts: That’s the service name.
152 00:13:30.280 ⇒ 00:13:33.479 Amber Lin: That’s the right one, that’s the one we want… we wanna get.
153 00:13:33.860 ⇒ 00:13:37.510 Samuel Roberts: Okay. So, I guess the problem was that when we did.
154 00:13:37.600 ⇒ 00:13:45.780 Amber Lin: Mosquito and systems… mosquito estimates together, like, mosquito and misting, there was nothing that matched it.
155 00:13:45.910 ⇒ 00:13:49.500 Samuel Roberts: Exactly. Misting is not here, and then there are no inspectors that do misting.
156 00:13:49.500 ⇒ 00:13:49.920 Amber Lin: Yeah.
157 00:13:50.500 ⇒ 00:13:52.479 Samuel Roberts: Technicians that do misting.
158 00:13:52.940 ⇒ 00:13:53.620 Amber Lin: Yeah.
159 00:13:55.060 ⇒ 00:14:09.860 Amber Lin: So… Let me try his question again. So, I would say… Mosquito… Mosquito System Estimate in Corpus.
160 00:14:10.160 ⇒ 00:14:10.970 Samuel Roberts: Yeah.
161 00:14:10.970 ⇒ 00:14:12.010 Amber Lin: Let me try.
162 00:14:12.350 ⇒ 00:14:25.650 Amber Lin: And similarly, for the other question, let me put something in the chat, this is what I found. So, his other question, the holiday lights inspectors, the way he asked it, it went into the central dock.
163 00:14:25.850 ⇒ 00:14:29.860 Amber Lin: It went to the Home Improvement Central doc and found a.
164 00:14:29.860 ⇒ 00:14:30.350 Samuel Roberts: what?
165 00:14:30.350 ⇒ 00:14:38.110 Amber Lin: technician, but I asked specifically then for the word inspector, and then it went to the right location.
166 00:14:38.110 ⇒ 00:14:44.160 Samuel Roberts: So he needs to know what he’s asking for, or I guess to be more specific what he’s asking for, or we need to figure out how to clarify that, right?
167 00:14:44.370 ⇒ 00:14:50.869 Amber Lin: I think as a… as a verbal interpretation, I think what he asks is pretty clear.
168 00:14:51.070 ⇒ 00:14:55.590 Amber Lin: I think… I feel like asking the user to be so specific on their.
169 00:14:55.590 ⇒ 00:15:00.620 Samuel Roberts: Well, that’s what I’m asking. So, like, is there… is there… Why… why is it finding…
170 00:15:02.440 ⇒ 00:15:10.930 Samuel Roberts: Okay, so this, the, what you shared, schedule a holiday light estimate, These two different answers are…
171 00:15:13.210 ⇒ 00:15:18.949 Samuel Roberts: the Central… it went to the… so is this from Central Doc, this Jose Melinderez?
172 00:15:19.530 ⇒ 00:15:28.859 Amber Lin: Yeah, the first answer, yeah, Jose, is from, like, you know the Department Home Improvement that it lists there? That’s from the Home Improvement Central doc.
173 00:15:29.490 ⇒ 00:15:33.059 Samuel Roberts: So there’s a different piece of information in the central doc than in the ZIPS database?
174 00:15:33.060 ⇒ 00:15:37.180 Amber Lin: Yes. Let me check what it says in the doc, actually.
175 00:15:37.180 ⇒ 00:15:40.690 Samuel Roberts: Well, the dock might be old, too, because it wasn’t syncing, because of our permission issues now, so…
176 00:15:40.690 ⇒ 00:15:41.280 Amber Lin: Yeah, but…
177 00:15:41.280 ⇒ 00:15:43.450 Mustafa Raja: Yeah, but, it should be synced now.
178 00:15:43.450 ⇒ 00:15:45.819 Samuel Roberts: It should be now, yeah. Good point, good point.
179 00:15:46.700 ⇒ 00:15:48.350 Amber Lin: That’s Jose…
180 00:15:52.140 ⇒ 00:15:59.209 Samuel Roberts: Yeah, Larry Lark… okay, so it looks like you’re… I’m seeing your thing come through. So, yeah, the system… it doesn’t know that mosquito misting and mosquito…
181 00:15:59.590 ⇒ 00:16:01.589 Samuel Roberts: System are the same thing.
182 00:16:02.750 ⇒ 00:16:08.550 Amber Lin: Yeah, and I don’t even think there’s a Jose Melendrez…
183 00:16:08.550 ⇒ 00:16:12.640 Samuel Roberts: That’s definitely an old system doc thing, or a central dock thing, rather.
184 00:16:12.760 ⇒ 00:16:13.260 Samuel Roberts: Right?
185 00:16:13.260 ⇒ 00:16:13.820 Mustafa Raja: Yeah.
186 00:16:13.820 ⇒ 00:16:16.600 Amber Lin: Yeah… I couldn’t.
187 00:16:16.600 ⇒ 00:16:20.829 Mustafa Raja: I mean, I just, I just, cleaned it up, so.
188 00:16:21.130 ⇒ 00:16:24.129 Samuel Roberts: Yeah, is this… what is this screenshot from that you…
189 00:16:26.040 ⇒ 00:16:27.130 Amber Lin: My screenshot?
190 00:16:27.410 ⇒ 00:16:30.189 Samuel Roberts: The… yeah, hold on, let me… the one…
191 00:16:30.350 ⇒ 00:16:33.229 Samuel Roberts: Was that just now? Because it says 2 minutes ago and now, is that just now?
192 00:16:33.780 ⇒ 00:16:36.460 Amber Lin: Yeah, I just asked… no.
193 00:16:36.650 ⇒ 00:16:38.840 Samuel Roberts: Okay. So it’s still points on Denver.
194 00:16:40.790 ⇒ 00:16:41.790 Samuel Roberts: Interesting.
195 00:16:42.290 ⇒ 00:16:44.010 Mustafa Raja: It’s still giving hosing.
196 00:16:46.400 ⇒ 00:16:46.920 Amber Lin: What do you mean?
197 00:16:47.110 ⇒ 00:16:50.130 Samuel Roberts: I’m trying to check the run that gave you Jose.
198 00:16:50.470 ⇒ 00:16:51.210 Amber Lin: Yeah.
199 00:16:51.530 ⇒ 00:16:56.619 Amber Lin: Yeah, I don’t know where the hotel came from, because I also cannot find it in our central dock.
200 00:16:56.990 ⇒ 00:17:00.569 Samuel Roberts: Caleb Hall, is that the other one? Okay, so that’s the one before that.
201 00:17:00.950 ⇒ 00:17:03.839 Samuel Roberts: I’m looking now to see that execution.
202 00:17:04.220 ⇒ 00:17:10.280 Samuel Roberts: Is this the one that responded with… Who is this?
203 00:17:11.369 ⇒ 00:17:15.709 Samuel Roberts: What… oh, get out of here! Cat is kind of chin in my lap.
204 00:17:18.800 ⇒ 00:17:25.390 Samuel Roberts: This first one… Schedule a holiday light estimate, there it is, okay.
205 00:17:25.640 ⇒ 00:17:32.079 Samuel Roberts: And then the response is… okay. So, transform… what did we get? The context agent pulled…
206 00:17:37.170 ⇒ 00:17:37.779 Samuel Roberts: So the.
207 00:17:37.780 ⇒ 00:17:39.990 Mustafa Raja: Which execution is it?
208 00:17:40.280 ⇒ 00:17:42.159 Samuel Roberts: Here, I’ll send it.
209 00:17:42.160 ⇒ 00:17:45.719 Mustafa Raja: Is it the second from the top, or third, or something?
210 00:17:46.680 ⇒ 00:17:49.080 Samuel Roberts: It’s a… well, yeah, I just… I just sent it, because I don’t…
211 00:17:50.000 ⇒ 00:17:52.200 Samuel Roberts: I don’t know how best to communicate that right now, so…
212 00:17:52.410 ⇒ 00:17:55.090 Amber Lin: Yeah, do you want to share screens so we can just…
213 00:17:55.090 ⇒ 00:18:00.610 Samuel Roberts: Oh, sure, sorry, I forgot I wasn’t right now, my bad, I forgot I stopped so I could see the screenshot, but I can’t. Okay.
214 00:18:01.280 ⇒ 00:18:04.679 Samuel Roberts: So I… this… I think it’s this one. So I saw here…
215 00:18:04.840 ⇒ 00:18:06.880 Samuel Roberts: I wanna look like this.
216 00:18:07.090 ⇒ 00:18:10.959 Samuel Roberts: Schedule Holiday Light, and then the response it gives is…
217 00:18:11.830 ⇒ 00:18:13.299 Samuel Roberts: Yeah, okay, so that’s the wrong…
218 00:18:13.300 ⇒ 00:18:13.740 Amber Lin: Yeah.
219 00:18:13.740 ⇒ 00:18:15.930 Mustafa Raja: Jose is actually coming from database.
220 00:18:16.140 ⇒ 00:18:17.100 Samuel Roberts: Jose’s in the database.
221 00:18:17.100 ⇒ 00:18:18.060 Amber Lin: Yes.
222 00:18:18.760 ⇒ 00:18:20.510 Amber Lin: But… Okay.
223 00:18:21.270 ⇒ 00:18:22.220 Amber Lin: confused.
224 00:18:23.680 ⇒ 00:18:24.600 Samuel Roberts: Select Cru- yeah, hold on.
225 00:18:24.600 ⇒ 00:18:25.060 Mustafa Raja: Yeah.
226 00:18:25.060 ⇒ 00:18:25.600 Samuel Roberts: There it is.
227 00:18:25.600 ⇒ 00:18:27.290 Mustafa Raja: Jose came from database.
228 00:18:27.420 ⇒ 00:18:27.910 Samuel Roberts: Yep.
229 00:18:27.910 ⇒ 00:18:33.550 Amber Lin: Then, Josepha, why did it say that it came from… home improvement…
230 00:18:33.550 ⇒ 00:18:34.730 Mustafa Raja: Hmm.
231 00:18:35.460 ⇒ 00:18:36.330 Samuel Roberts: This one we just added.
232 00:18:38.370 ⇒ 00:18:40.830 Amber Lin: Yeah, it says Department Home Improvement.
233 00:18:40.830 ⇒ 00:18:41.360 Mustafa Raja: soon.
234 00:18:42.000 ⇒ 00:18:48.000 Amber Lin: Like, technically, I guess, technically, in the technicians, it is under home improvement?
235 00:18:48.140 ⇒ 00:18:55.529 Mustafa Raja: No, no, no, it should not do this. It did this because the system prompt is really bloated.
236 00:18:56.090 ⇒ 00:18:56.880 Samuel Roberts: Oh…
237 00:18:56.880 ⇒ 00:19:06.790 Mustafa Raja: One of the restrictions that I have given it is, if… it should only include departments when, the… what’s it called? The vector… vector tool is used.
238 00:19:07.180 ⇒ 00:19:14.530 Mustafa Raja: And I believe memory is also messing it up a little.
239 00:19:15.180 ⇒ 00:19:16.080 Mustafa Raja: We need to…
240 00:19:16.080 ⇒ 00:19:16.770 Samuel Roberts: Oh, yeah.
241 00:19:16.770 ⇒ 00:19:27.819 Mustafa Raja: Because… because what’s it… about 80% or 75% of the… of the system prompt is really just bloat. We are hard-coding information into the system prompt.
242 00:19:28.190 ⇒ 00:19:29.730 Samuel Roberts: Yeah, we gotta… ugh, okay.
243 00:19:32.430 ⇒ 00:19:39.760 Mustafa Raja: This is… I added a very small instruction, there, and
244 00:19:40.150 ⇒ 00:19:46.789 Mustafa Raja: Yeah, I believe if the system prompt was optimized, this should not have happened.
245 00:19:46.790 ⇒ 00:19:49.740 Samuel Roberts: Right, but we’re not gonna print that off right now.
246 00:19:51.770 ⇒ 00:19:53.650 Amber Lin: Let’s see…
247 00:19:55.790 ⇒ 00:20:08.370 Amber Lin: I think that’s our broader problem. Let’s say for our immediate problem, so for these two questions that Bo have asked, what is our conclusion of what went wrong?
248 00:20:10.430 ⇒ 00:20:13.229 Mustafa Raja: So, Mystic was just a naming convention, right?
249 00:20:13.230 ⇒ 00:20:14.200 Amber Lin: Okay.
250 00:20:14.930 ⇒ 00:20:15.490 Amber Lin: to one…
251 00:20:15.490 ⇒ 00:20:16.419 Mustafa Raja: So do we…
252 00:20:16.640 ⇒ 00:20:21.770 Amber Lin: Mary is too inflexible that it gives mosquito and misting together.
253 00:20:21.930 ⇒ 00:20:22.470 Amber Lin: Is that…
254 00:20:22.470 ⇒ 00:20:27.580 Samuel Roberts: Well, it’s sort of that. It’s that… there’s two different service names.
255 00:20:28.070 ⇒ 00:20:30.340 Samuel Roberts: And misting is only in one of them.
256 00:20:30.570 ⇒ 00:20:31.190 Samuel Roberts: So…
257 00:20:31.820 ⇒ 00:20:39.579 Samuel Roberts: it doesn’t know that… like, because like I said, it had mosquito space misting, so that’s never going to match mosquito system.
258 00:20:39.580 ⇒ 00:20:40.760 Amber Lin: Yes, that’s correct.
259 00:20:40.760 ⇒ 00:20:43.360 Samuel Roberts: So… We need to…
260 00:20:46.120 ⇒ 00:20:51.649 Samuel Roberts: Somehow, like, that the service names are the same, or that misting is always a mosquito thing?
261 00:20:52.590 ⇒ 00:20:58.390 Samuel Roberts: Like… If I can get rid of inspectors, we’ll see…
262 00:20:59.500 ⇒ 00:21:02.319 Mustafa Raja: I mean, if they are going to query mosquitoes…
263 00:21:02.320 ⇒ 00:21:06.069 Samuel Roberts: There’s mosquito suppression as well. Mosquito misting.
264 00:21:07.500 ⇒ 00:21:09.190 Amber Lin: But they’re technicians, right?
265 00:21:09.810 ⇒ 00:21:12.570 Samuel Roberts: Those are all techniques, yeah, the ins… I was just trying to see, like, what other phrases were.
266 00:21:12.570 ⇒ 00:21:13.190 Amber Lin: I see.
267 00:21:13.190 ⇒ 00:21:14.000 Samuel Roberts: keto.
268 00:21:14.150 ⇒ 00:21:22.540 Samuel Roberts: Because yeah, all the technicians do suppression, but the inspector does mosquito systems estimates, so how best do we tell it to keep
269 00:21:23.490 ⇒ 00:21:27.210 Samuel Roberts: Like, whenever Mosquito is asked, just… And we search mosquito?
270 00:21:27.760 ⇒ 00:21:30.200 Samuel Roberts: But that might mess it up other whales.
271 00:21:31.520 ⇒ 00:21:37.260 Samuel Roberts: Like, there’s no way for it to know that these… mosquito system estimates.
272 00:21:37.550 ⇒ 00:21:39.770 Samuel Roberts: are for misting.
273 00:21:42.750 ⇒ 00:21:46.710 Samuel Roberts: So it either needs to be less specific, or we need to… add…
274 00:21:46.910 ⇒ 00:21:50.260 Samuel Roberts: That information somewhere where it can know that.
275 00:21:54.180 ⇒ 00:21:55.239 Uttam Kumaran: Hey guys, I’m back.
276 00:21:55.430 ⇒ 00:21:55.770 Samuel Roberts: Hey.
277 00:21:55.770 ⇒ 00:22:02.189 Amber Lin: Yeah. We figure out the correct answer, and we have the initial
278 00:22:02.790 ⇒ 00:22:10.520 Amber Lin: guess of what is wrong. The first question is with our queries. Our queries are…
279 00:22:11.800 ⇒ 00:22:14.680 Amber Lin: Not matching up the exact words.
280 00:22:16.710 ⇒ 00:22:17.460 Uttam Kumaran: A…
281 00:22:17.670 ⇒ 00:22:25.480 Samuel Roberts: Yeah, so what we… so there are technicians that do mosquito suppression, and technicians that do… Mosquito misting.
282 00:22:25.890 ⇒ 00:22:40.729 Samuel Roberts: There are no inspectors that do either one of those. All that inspector… the only inspector that does anything does residential mosquito system estimates. And so, what the query it built included mosquito misting, because that’s what was specifically asked.
283 00:22:41.070 ⇒ 00:22:41.660 Uttam Kumaran: Okay.
284 00:22:42.250 ⇒ 00:22:48.069 Samuel Roberts: So we need to somehow either tell it to, you know, know that Mosquito Systems is both
285 00:22:48.750 ⇒ 00:22:50.810 Samuel Roberts: Misting and, I guess, suppression?
286 00:22:51.020 ⇒ 00:22:54.579 Samuel Roberts: Or somehow convey that that’s in here.
287 00:22:55.130 ⇒ 00:22:56.259 Samuel Roberts: Some other way.
288 00:22:57.150 ⇒ 00:23:00.660 Uttam Kumaran: So, like, I guess, in that sense, should we have…
289 00:23:01.680 ⇒ 00:23:04.540 Uttam Kumaran: qualified the input more? Like, what is the.
290 00:23:04.540 ⇒ 00:23:05.010 Samuel Roberts: as well.
291 00:23:05.010 ⇒ 00:23:05.600 Uttam Kumaran: Shannon.
292 00:23:06.260 ⇒ 00:23:08.450 Samuel Roberts: That’s what I’m trying… I’m trying to understand, like.
293 00:23:09.500 ⇒ 00:23:12.840 Samuel Roberts: I mean, part of it is the naming convention, and I just don’t know…
294 00:23:13.450 ⇒ 00:23:18.749 Samuel Roberts: If it’s, like, the model doesn’t understand misting versus system.
295 00:23:19.040 ⇒ 00:23:21.430 Samuel Roberts: I don’t know if we can…
296 00:23:23.460 ⇒ 00:23:27.510 Samuel Roberts: Well, I don’t know, because this was very specific, mosquito misting, so that’s not like we can…
297 00:23:27.740 ⇒ 00:23:31.020 Samuel Roberts: loosen that, necessarily. That’s gonna be what’s generated.
298 00:23:33.350 ⇒ 00:23:35.700 Uttam Kumaran: Can you explain it one more time? Sorry.
299 00:23:35.700 ⇒ 00:23:41.420 Samuel Roberts: Yeah, so okay, so the question… I don’t have it up anymore, but the question he asked was…
300 00:23:41.540 ⇒ 00:23:42.400 Samuel Roberts: That’s it.
301 00:23:43.110 ⇒ 00:23:45.309 Samuel Roberts: I can’t view anything in the…
302 00:23:45.310 ⇒ 00:23:51.009 Amber Lin: I need to schedule a mosquito misting system estimate in Corpus.
303 00:23:51.010 ⇒ 00:23:53.890 Uttam Kumaran: You said mosquito Misting System?
304 00:23:54.750 ⇒ 00:24:06.140 Samuel Roberts: Okay, yeah. So we… it needs to know that mosquitoes… well, I don’t… that’s getting very ad hoc, I just… something about misting is what carried through, and of course, that’s not in the system name.
305 00:24:07.500 ⇒ 00:24:11.880 Samuel Roberts: So maybe we need a… well, no, that notes was just gonna add another thing that we have to search over.
306 00:24:12.830 ⇒ 00:24:15.489 Uttam Kumaran: Wait, wait, I guess I’m not falling.
307 00:24:16.240 ⇒ 00:24:21.499 Uttam Kumaran: I guess, can you… wait, but can we just walk through one more time, like, why that didn’t work?
308 00:24:21.570 ⇒ 00:24:26.170 Samuel Roberts: Yeah, so there… there are no… so when it writes this query.
309 00:24:26.320 ⇒ 00:24:34.670 Samuel Roberts: Which is mosquito misting in corpus, and looking for an inspector. The only inspectors Do mosquito system estimates.
310 00:24:35.200 ⇒ 00:24:35.790 Samuel Roberts: But it was so.
311 00:24:35.790 ⇒ 00:24:37.990 Uttam Kumaran: So there’s no one that does a misting system.
312 00:24:38.370 ⇒ 00:24:38.739 Samuel Roberts: There’s no…
313 00:24:38.740 ⇒ 00:24:41.440 Amber Lin: Misting is a subset of systems.
314 00:24:44.220 ⇒ 00:24:50.720 Amber Lin: So, they don’t divide as far, and he has a specific question that Andy registered.
315 00:24:51.860 ⇒ 00:24:52.769 Uttam Kumaran: But I guess, in this case.
316 00:24:52.770 ⇒ 00:24:54.210 Amber Lin: that word.
317 00:24:54.450 ⇒ 00:24:57.530 Uttam Kumaran: In this situation, what should the answer have…
318 00:24:58.430 ⇒ 00:25:00.280 Uttam Kumaran: It should have said, there is nobody.
319 00:25:01.520 ⇒ 00:25:02.559 Uttam Kumaran: Should’ve had said.
320 00:25:02.560 ⇒ 00:25:14.620 Amber Lin: It should have concluded it under just general mosquito… mosquito system estimates, and once I did that, I asked mosquito system estimates in Corpus. It gave me the right person.
321 00:25:16.410 ⇒ 00:25:20.849 Uttam Kumaran: But then the misting system, when it asked that, what did it actually end up wearing?
322 00:25:21.380 ⇒ 00:25:24.929 Samuel Roberts: Yeah, what was the result of that? That was this, that was this, which I believe…
323 00:25:24.930 ⇒ 00:25:27.740 Amber Lin: there was no person. There was nobody.
324 00:25:28.000 ⇒ 00:25:30.029 Uttam Kumaran: Oh, okay.
325 00:25:30.030 ⇒ 00:25:31.680 Samuel Roberts: So when I remove the misting…
326 00:25:31.970 ⇒ 00:25:32.540 Uttam Kumaran: Yeah.
327 00:25:32.980 ⇒ 00:25:36.269 Samuel Roberts: we get that one inspector that does that for the zips.
328 00:25:37.540 ⇒ 00:25:40.860 Samuel Roberts: But there’s no anywhere in here. So there’s…
329 00:25:40.970 ⇒ 00:25:47.950 Samuel Roberts: It doesn’t know that mosquito system estimates includes a misting element, like sub…
330 00:25:48.540 ⇒ 00:25:51.599 Samuel Roberts: Like, I guess I don’t know how these services are organized.
331 00:25:51.710 ⇒ 00:25:54.520 Samuel Roberts: Cause these are just the names, like, there’s no other…
332 00:25:54.700 ⇒ 00:25:57.669 Samuel Roberts: I don’t know, I can dig in a little bit more here.
333 00:25:57.710 ⇒ 00:26:02.589 Uttam Kumaran: But I guess, like, in that situation, like, maybe let’s walk through, like.
334 00:26:02.810 ⇒ 00:26:09.079 Uttam Kumaran: So the AI should have seen Mosquito misting system, it should have first… and it should have just found…
335 00:26:09.760 ⇒ 00:26:11.550 Uttam Kumaran: mosquito system.
336 00:26:12.040 ⇒ 00:26:16.420 Samuel Roberts: Okay, that’s… that’s something we could do. We could have it check the service areas first.
337 00:26:17.210 ⇒ 00:26:21.540 Samuel Roberts: Like… Make sure that it’s looking up the right.
338 00:26:21.540 ⇒ 00:26:26.030 Uttam Kumaran: I guess, additionally, it could… if it produced no results.
339 00:26:26.290 ⇒ 00:26:32.380 Uttam Kumaran: It should ask for a clarification, because will there ever be a situation in where it doesn’t give results?
340 00:26:33.090 ⇒ 00:26:34.110 Uttam Kumaran: You know?
341 00:26:35.300 ⇒ 00:26:45.630 Amber Lin: That situation would be they don’t service that area. In that case, we would refer to the service area sheet and tell them, hey, you don’t service this area at all. So there would still be an answer.
342 00:26:47.360 ⇒ 00:26:48.330 Uttam Kumaran: Okay.
343 00:26:48.330 ⇒ 00:26:49.790 Samuel Roberts: Is that the answer he got, though?
344 00:26:50.210 ⇒ 00:26:57.650 Amber Lin: No, because we never checked the service area sheet. We just… Like, we didn’t take.
345 00:26:57.650 ⇒ 00:26:59.609 Samuel Roberts: It just, it just returned no results. Okay.
346 00:26:59.610 ⇒ 00:27:00.200 Amber Lin: Yeah.
347 00:27:00.200 ⇒ 00:27:03.730 Samuel Roberts: So there is or should be another step, is that the point here?
348 00:27:04.620 ⇒ 00:27:09.490 Uttam Kumaran: Okay. I guess, like, well, I guess that’s what we should debate. I’m just trying to think through, like.
349 00:27:11.040 ⇒ 00:27:11.980 Uttam Kumaran: I mean…
350 00:27:12.270 ⇒ 00:27:18.480 Uttam Kumaran: I guess they’re just… right now, like, I’m wondering, what are we providing the AI to form the query?
351 00:27:18.950 ⇒ 00:27:23.280 Uttam Kumaran: Like, do we have… is there context that we’re providing the tool, agent?
352 00:27:23.570 ⇒ 00:27:25.150 Uttam Kumaran: And, like, maybe we can…
353 00:27:25.670 ⇒ 00:27:32.860 Uttam Kumaran: play with that more, because there’s several steps here, right? Like, okay, first of all, if the AI isn’t confident.
354 00:27:33.850 ⇒ 00:27:42.869 Uttam Kumaran: like, I would rather the AI ask a clarifying question than give a wrong answer. I think I’ve expressed that a couple of times, but I still don’t know if we’ve…
355 00:27:43.970 ⇒ 00:27:47.180 Uttam Kumaran: Done anything on the input validation side?
356 00:27:48.050 ⇒ 00:27:54.629 Uttam Kumaran: Like, is there anything that sort of gives a judgment on, like, okay, this input is good enough, or no?
357 00:27:58.290 ⇒ 00:28:04.080 Samuel Roberts: I don’t believe so. I mean, Mustafa, is there anything in here? I don’t think so. I’m not as familiar with this.
358 00:28:04.080 ⇒ 00:28:08.539 Amber Lin: We sometimes qualify for, is this the right department?
359 00:28:09.130 ⇒ 00:28:09.450 Samuel Roberts: And.
360 00:28:09.450 ⇒ 00:28:11.209 Amber Lin: We only qualify for that, but that’s.
361 00:28:11.210 ⇒ 00:28:14.140 Samuel Roberts: Let’s qualify the department, is that the routing agent here?
362 00:28:14.660 ⇒ 00:28:16.699 Amber Lin: That’s not related to zip codes.
363 00:28:17.090 ⇒ 00:28:21.569 Samuel Roberts: Right, so we wanna… so we… what we should probably do… well, no, that’s not probably.
364 00:28:21.700 ⇒ 00:28:28.409 Samuel Roberts: But basically, the fact that it was searching for mosquito misting… well, no, it would have found a valid service area for that, so that’s not necessarily…
365 00:28:28.710 ⇒ 00:28:38.370 Uttam Kumaran: Well, can we just build… maybe what we can do is let’s move to… can we just open a Google Sheet, or we can open the ABC Platform docs? Let’s just create a quick scenario table.
366 00:28:38.610 ⇒ 00:28:44.039 Uttam Kumaran: like… And let’s just start listing out, like, types of query that we could get.
367 00:28:44.360 ⇒ 00:28:48.930 Uttam Kumaran: And what should… how Andy should basically handle it.
368 00:28:49.220 ⇒ 00:28:52.300 Uttam Kumaran: I’d like to just start there.
369 00:28:52.750 ⇒ 00:29:00.370 Uttam Kumaran: Because, for example, like, an easy way to do input validation here is, like, if there… if there is no,
370 00:29:00.900 ⇒ 00:29:10.590 Uttam Kumaran: If there is no… we’re not… if you need 4 different pieces to run the query, and you’re missing one of them, or one’s… one needs clarification, you can ask.
371 00:29:10.710 ⇒ 00:29:11.450 Uttam Kumaran: Right?
372 00:29:11.570 ⇒ 00:29:19.530 Uttam Kumaran: So, for example, if it knows that… if it… if it looks his misting system, and then is like, I don’t know what that is, it should ask.
373 00:29:19.790 ⇒ 00:29:23.949 Uttam Kumaran: Right? Like, that’s the level of input validation I’m talking about that we haven’t done here.
374 00:29:23.950 ⇒ 00:29:28.980 Samuel Roberts: So you’re saying… you’re saying it should… it should match against that service name? Because I’m… I mean, it has misting…
375 00:29:29.150 ⇒ 00:29:32.040 Samuel Roberts: Like, if I… if it wasn’t looking for inspectors.
376 00:29:32.180 ⇒ 00:29:37.150 Samuel Roberts: Mosquito misting was a valid service name. It’s just not missing system.
377 00:29:39.150 ⇒ 00:29:43.550 Uttam Kumaran: Yeah, I mean, I have to go look through… I mean, I don’t… this is where I just…
378 00:29:43.720 ⇒ 00:29:48.100 Uttam Kumaran: I have to see all the scenarios for us to think about, like, how do we fix this?
379 00:29:48.100 ⇒ 00:29:50.169 Samuel Roberts: Yeah. I don’t even have we done it.
380 00:29:50.170 ⇒ 00:29:51.439 Uttam Kumaran: Have we looked through, like.
381 00:29:51.770 ⇒ 00:29:56.179 Uttam Kumaran: Yeah, I just, like, need to get all the information in one place here, so we can…
382 00:29:56.570 ⇒ 00:29:59.630 Uttam Kumaran: like, just fix this, because I feel like we’ve just been patching this.
383 00:29:59.630 ⇒ 00:30:02.850 Samuel Roberts: Yeah, well, that’s why we gotta move to… that’s why, you know…
384 00:30:02.850 ⇒ 00:30:14.060 Uttam Kumaran: But, like, again, like, that’s not done, and it’s not… it’s week 6 of issues. So I’m kind of, like, I’m kind of a… I don’t have a choice but to sort of intervene here, and so…
385 00:30:14.340 ⇒ 00:30:20.179 Uttam Kumaran: Like, I don’t know what to do. We have to keep working. Unless we can do that migration safely.
386 00:30:20.520 ⇒ 00:30:23.029 Uttam Kumaran: Today, we have to patch this.
387 00:30:23.130 ⇒ 00:30:33.639 Uttam Kumaran: And so, I think, like, I have to… I’m just running to an interview, I’ll be… I’m back on my laptop in, like, 20 minutes.
388 00:30:33.860 ⇒ 00:30:38.580 Uttam Kumaran: I think let’s just start a… let’s start… if we can start a Google Sheet where we can start listing
389 00:30:38.720 ⇒ 00:30:42.000 Uttam Kumaran: The scenario of the issue, the input.
390 00:30:42.170 ⇒ 00:30:53.769 Uttam Kumaran: and, like, the current and expect… current output and expected output, if we can just start there. I’d first just like to get an index of all the issues that we’re facing, and we faced in the last, like, 6 weeks.
391 00:30:54.140 ⇒ 00:30:58.710 Uttam Kumaran: Even if it’s been solved, I just want to, like, have that.
392 00:30:58.880 ⇒ 00:31:02.779 Uttam Kumaran: I don’t know whether that’s… Already exists somewhere, or whatever, but…
393 00:31:03.070 ⇒ 00:31:07.949 Uttam Kumaran: for me to feel confident that we patch this, like, I have to kind of see that, so that’s something…
394 00:31:08.120 ⇒ 00:31:09.900 Uttam Kumaran: Mustafa, you can do…
395 00:31:10.060 ⇒ 00:31:16.589 Uttam Kumaran: or whatever, if we can put something together, like a quick spreadsheet like that, I can start there. And then…
396 00:31:17.150 ⇒ 00:31:25.400 Uttam Kumaran: For this scenario in particular, if we can list all the helpful, like, links, like, the link to…
397 00:31:25.840 ⇒ 00:31:34.310 Uttam Kumaran: Like, the link to… Supabase, like this view, the link to the N8N run,
398 00:31:35.050 ⇒ 00:31:44.660 Uttam Kumaran: Because I… basically, the way I’m gonna solve this is I’m just gonna… I wanna give them a clear explanation of, like, all the issues we’ve seen, and how we’re patching them.
399 00:31:44.790 ⇒ 00:31:47.599 Uttam Kumaran: Because if we can’t… first thing is, like.
400 00:31:48.030 ⇒ 00:31:58.939 Uttam Kumaran: if we’re not confident the system is going to work all the time, then we need to give them confidence that we can… that we’re catching issues, and that we’re validating it, and we’re fixing it. It’s clear now that, like, we’re not…
401 00:31:59.580 ⇒ 00:32:01.830 Uttam Kumaran: We’re not responding to feedback, like…
402 00:32:02.050 ⇒ 00:32:05.449 Uttam Kumaran: fast enough, like, Bo Jenkins is the son of the CEO.
403 00:32:05.650 ⇒ 00:32:08.090 Uttam Kumaran: So if he finds an issue, then there’s, like…
404 00:32:08.200 ⇒ 00:32:13.700 Uttam Kumaran: it’s kind of like red alert. I’m not seeing any sort of, like, alerting still on the fact that, like.
405 00:32:13.960 ⇒ 00:32:19.289 Uttam Kumaran: this… this is… this happened. So, there’s a couple things I just want to work on, and we can work on tonight.
406 00:32:19.420 ⇒ 00:32:24.029 Uttam Kumaran: But, like, if we can just start by putting those issues in a spreadsheet, then I can start
407 00:32:24.230 ⇒ 00:32:27.269 Uttam Kumaran: To plan out, like, how do we tackle this this week?
408 00:32:27.380 ⇒ 00:32:28.380 Uttam Kumaran: That’s okay.
409 00:32:30.840 ⇒ 00:32:36.950 Samuel Roberts: Okay, yeah, so that would be taking the, what, the linear tickets, Mustafa, and putting them, like, starting to pull all the…
410 00:32:37.570 ⇒ 00:32:38.660 Amber Lin: Ins and outs.
411 00:32:39.230 ⇒ 00:32:40.040 Samuel Roberts: That’s virtually.
412 00:32:40.040 ⇒ 00:32:46.330 Amber Lin: We… yeah, I mean, feel free to hop, and then we can whilst you’re on to make that spreadsheet.
413 00:32:46.330 ⇒ 00:32:46.920 Samuel Roberts: Yeah.
414 00:32:47.670 ⇒ 00:32:48.689 Uttam Kumaran: Okay, okay, perfect, thank you.
415 00:32:48.690 ⇒ 00:32:49.270 Amber Lin: God.
416 00:32:49.610 ⇒ 00:32:50.650 Uttam Kumaran: Talk soon.
417 00:32:51.580 ⇒ 00:32:59.230 Amber Lin: I guess, Samuel, shouldn’t there be a log in NAN or somewhere as well that has
418 00:32:59.390 ⇒ 00:33:07.760 Amber Lin: So, so that we can also give UTAM the link, the NAN run link, to that specific issue, because we’ll have to click back.
419 00:33:08.440 ⇒ 00:33:09.400 Mustafa Raja: Yeah, so…
420 00:33:09.400 ⇒ 00:33:10.700 Amber Lin: We’re logged somewhere.
421 00:33:10.700 ⇒ 00:33:12.860 Samuel Roberts: Logging is in Snowflake, right?
422 00:33:13.550 ⇒ 00:33:21.010 Mustafa Raja: Logging is in Snowflake, and then this, client ABC Logs channel, let me see if you’re in the… you’re not in there, let me audio.
423 00:33:22.560 ⇒ 00:33:23.640 Mustafa Raja: Somewhere…
424 00:33:25.280 ⇒ 00:33:27.920 Samuel Roberts: Me calling it ABC Logs.
425 00:33:29.470 ⇒ 00:33:31.200 Mustafa Raja: Yeah, I’ve added you in the…
426 00:33:31.720 ⇒ 00:33:44.250 Mustafa Raja: Now, the way I find the exact execution is I just see, what time they gave feedback to, and around that time is going to be the execution, too.
427 00:33:47.770 ⇒ 00:33:48.320 Samuel Roberts: Also, we don’t even.
428 00:33:50.100 ⇒ 00:33:54.659 Mustafa Raja: Yeah, I wanted to, but, we just never got, got there.
429 00:33:54.830 ⇒ 00:33:57.190 Samuel Roberts: No, that’s fine, I just wasn’t sure what was here.
430 00:34:00.930 ⇒ 00:34:03.439 Mustafa Raja: Let me see if the triages have the links.
431 00:34:05.350 ⇒ 00:34:11.400 Amber Lin: Not to any NAN friends, there’s the Slack link, that’s all.
432 00:34:11.889 ⇒ 00:34:16.459 Samuel Roberts: Where’s the… Snowflake logging is…
433 00:34:16.460 ⇒ 00:34:17.370 Amber Lin: Excuse me.
434 00:34:18.730 ⇒ 00:34:19.980 Samuel Roberts: Okay…
435 00:34:25.170 ⇒ 00:34:32.509 Samuel Roberts: a cleaner… Where does this get logged? Oh… oh, that’s a windmill query? Okay, hold on.
436 00:34:34.610 ⇒ 00:34:35.510 Samuel Roberts: Okay.
437 00:34:43.130 ⇒ 00:34:43.900 Samuel Roberts: this one.
438 00:34:47.560 ⇒ 00:34:49.799 Samuel Roberts: Matt gets along to…
439 00:34:55.500 ⇒ 00:35:03.239 Samuel Roberts: Record ID, Snowflake logger… That goes to… wait… It’s nothing longer, aren’t we?
440 00:35:04.340 ⇒ 00:35:06.100 Samuel Roberts: Notified blogger…
441 00:35:30.710 ⇒ 00:35:32.969 Mustafa Raja: Sorry, what are you trying to do?
442 00:35:32.970 ⇒ 00:35:34.440 Samuel Roberts: I’m just trying to give him a snowflake now.
443 00:35:35.710 ⇒ 00:35:36.470 Mustafa Raja: Oh, dear.
444 00:35:37.070 ⇒ 00:35:40.610 Samuel Roberts: I’m trying to see what… We can pull from there for these logs.
445 00:35:41.080 ⇒ 00:35:43.190 Samuel Roberts: And, of course, my ratings.
446 00:35:44.190 ⇒ 00:35:46.680 Samuel Roberts: Logged in properly… again…
447 00:35:50.330 ⇒ 00:35:51.300 Samuel Roberts: Okay.
448 00:36:06.390 ⇒ 00:36:09.270 Samuel Roberts: ABC Logs, is that what we’re looking for in here?
449 00:36:17.370 ⇒ 00:36:18.760 Samuel Roberts: A few moments…
450 00:36:30.010 ⇒ 00:36:31.199 Samuel Roberts: Where is that being?
451 00:36:32.250 ⇒ 00:36:33.360 Mustafa Raja: Yeah, did you get that?
452 00:36:34.380 ⇒ 00:36:35.039 Samuel Roberts: Did I what?
453 00:36:35.770 ⇒ 00:36:37.400 Mustafa Raja: The… the logs.
454 00:36:37.850 ⇒ 00:36:44.159 Samuel Roberts: Yeah, I’m just finding ABC logs, ABC schema, I’m just finding the right… table now, I think.
455 00:36:48.220 ⇒ 00:36:54.890 Samuel Roberts: And then the table… User account…
456 00:36:55.170 ⇒ 00:36:57.980 Mustafa Raja: Yeah, it’s… I guess it’s ABC Log Z2, right?
457 00:36:58.650 ⇒ 00:37:00.170 Samuel Roberts: warehouse.
458 00:37:00.400 ⇒ 00:37:01.350 Samuel Roberts: I’m sorry?
459 00:37:02.070 ⇒ 00:37:05.060 Mustafa Raja: The table name would be ABC Logs V2, right?
460 00:37:05.060 ⇒ 00:37:08.729 Samuel Roberts: Oh, V2, okay, that’s what I was trying to figure out which one it was, but I couldn’t see in the windmill.
461 00:37:09.270 ⇒ 00:37:10.580 Samuel Roberts: V2…
462 00:37:12.990 ⇒ 00:37:16.399 Mustafa Raja: Yeah, I see the query it has… it’s storing in that.
463 00:37:19.760 ⇒ 00:37:21.080 Mustafa Raja: In… in Eden.
464 00:37:22.330 ⇒ 00:37:24.390 Samuel Roberts: Oh, okay, I was looking at the windmill.
465 00:37:25.370 ⇒ 00:37:27.719 Samuel Roberts: Does it say the table here? Okay, I see.
466 00:37:28.590 ⇒ 00:37:35.579 Samuel Roberts: Oh, I see there, yeah, okay, insert into, cool, okay, sorry. Alright, yeah, so we’re there, and then V2…
467 00:37:39.120 ⇒ 00:37:46.429 Mustafa Raja: I’m just trying to see what’s even… Okay, so, so the, so the goal here would be, to sort of create,
468 00:37:47.270 ⇒ 00:37:48.539 Mustafa Raja: What’s it called?
469 00:37:51.960 ⇒ 00:37:54.099 Mustafa Raja: A CSV or something, right?
470 00:37:56.300 ⇒ 00:37:57.339 Samuel Roberts: Yeah, but on a small sheet…
471 00:37:57.340 ⇒ 00:37:57.980 Mustafa Raja: Max.
472 00:37:58.740 ⇒ 00:38:03.670 Samuel Roberts: With, yeah, we gotta figure out, like, ins and outs, like, the proper ins and outs, basically.
473 00:38:04.980 ⇒ 00:38:05.770 Samuel Roberts: To compare.
474 00:38:06.640 ⇒ 00:38:11.040 Mustafa Raja: Yeah, this sort of… Stores everything.
475 00:38:13.560 ⇒ 00:38:14.610 Samuel Roberts: Hmm…
476 00:38:15.370 ⇒ 00:38:16.310 Mustafa Raja: Oh my god.
477 00:38:20.480 ⇒ 00:38:24.380 Amber Lin: Oh, can someone share screen so I also know what you guys are talking about?
478 00:38:26.660 ⇒ 00:38:27.710 Mustafa Raja: Yeah.
479 00:38:28.690 ⇒ 00:38:33.479 Amber Lin: So, essentially, I think we were talking once, input, output.
480 00:38:34.280 ⇒ 00:38:40.459 Amber Lin: link to the NAN run log or SQL query.
481 00:38:41.260 ⇒ 00:38:46.609 Amber Lin: And then, like, we can manually figure out what should be right.
482 00:38:48.340 ⇒ 00:38:50.040 Amber Lin: Which will take some effort.
483 00:38:50.800 ⇒ 00:38:51.390 Samuel Roberts: Yeah.
484 00:38:51.730 ⇒ 00:39:04.720 Mustafa Raja: Okay, so if I’m understanding it correctly, we can, for the past week, maybe, fetch or create a CSV for all of the feedback with thumbs down.
485 00:39:04.720 ⇒ 00:39:05.440 Amber Lin: Yeah.
486 00:39:06.820 ⇒ 00:39:10.559 Mustafa Raja: Create a column with the ideal output, right?
487 00:39:10.840 ⇒ 00:39:15.209 Amber Lin: Yeah, great. Can you do past…
488 00:39:15.690 ⇒ 00:39:17.980 Amber Lin: 6 weeks? We might not be able.
489 00:39:17.980 ⇒ 00:39:18.410 Mustafa Raja: to cover.
490 00:39:18.410 ⇒ 00:39:20.860 Amber Lin: For all of that, but… Like…
491 00:39:21.290 ⇒ 00:39:25.810 Mustafa Raja: Oh, yeah, because 6 weeks since we have been having issues, right?
492 00:39:25.810 ⇒ 00:39:26.520 Amber Lin: Yeah.
493 00:39:26.840 ⇒ 00:39:29.130 Amber Lin: But just the thumbs down is okay.
494 00:39:30.100 ⇒ 00:39:33.670 Mustafa Raja: Okay. Sam, would you… would you query?
495 00:39:33.670 ⇒ 00:39:35.640 Samuel Roberts: Yeah, I’m taking a look right now, hold on.
496 00:39:35.640 ⇒ 00:39:37.989 Mustafa Raja: Okay, so, alright, we’re looking for…
497 00:39:38.240 ⇒ 00:39:39.280 Samuel Roberts: Hold on.
498 00:39:39.280 ⇒ 00:39:39.890 Mustafa Raja: I used to it.
499 00:39:39.890 ⇒ 00:39:44.350 Samuel Roberts: Opera… what do you… Oh, you created a new file? Okay, I wasn’t sure. I must… alright.
500 00:39:45.020 ⇒ 00:39:47.210 Mustafa Raja: Oh, yeah, yeah.
501 00:39:47.500 ⇒ 00:39:53.720 Samuel Roberts: Yeah, I just wasn’t sure how to actually execute the sequel.
502 00:39:54.090 ⇒ 00:39:54.780 Mustafa Raja: I see the…
503 00:39:54.780 ⇒ 00:39:57.100 Samuel Roberts: table here and everything, I wasn’t sure.
504 00:39:57.810 ⇒ 00:40:02.909 Samuel Roberts: Alright, so we’re looking for… That’s the conversation ID, the record ID…
505 00:40:05.360 ⇒ 00:40:08.340 Samuel Roberts: Alright, what is this? Okay, so hold on. Select…
506 00:40:13.160 ⇒ 00:40:18.210 Samuel Roberts: All the logs that have… This is down by the last.
507 00:40:18.890 ⇒ 00:40:19.940 Samuel Roberts: Sweet.
508 00:40:20.100 ⇒ 00:40:22.659 Samuel Roberts: It’s not gonna do it quite randomly.
509 00:40:23.770 ⇒ 00:40:27.020 Samuel Roberts: Not authorized sauna. Okay, good, that’s great, alright.
510 00:40:27.210 ⇒ 00:40:30.229 Samuel Roberts: Sorry, I can share my screen here, but yeah. Are you getting it there?
511 00:40:31.420 ⇒ 00:40:34.439 Mustafa Raja: I’m just… I’m just trying something.
512 00:40:34.440 ⇒ 00:40:38.400 Samuel Roberts: Okay, yeah, ABC logs, ABC schema…
513 00:40:39.390 ⇒ 00:40:41.510 Samuel Roberts: And then I want to say select.
514 00:40:42.790 ⇒ 00:40:46.830 Mustafa Raja: However… Oh.
515 00:40:55.230 ⇒ 00:40:57.070 Mustafa Raja: Okay.
516 00:41:03.890 ⇒ 00:41:06.340 Mustafa Raja: Oh, we’re not recording this?
517 00:41:06.340 ⇒ 00:41:07.670 Amber Lin: time field?
518 00:41:08.830 ⇒ 00:41:10.589 Mustafa Raja: Oh, we are…
519 00:41:11.180 ⇒ 00:41:11.790 Samuel Roberts: What?
520 00:41:11.790 ⇒ 00:41:16.230 Mustafa Raja: We are not… I don’t see one.
521 00:41:16.230 ⇒ 00:41:18.200 Amber Lin: Wait, can you go to the first column?
522 00:41:20.520 ⇒ 00:41:21.720 Samuel Roberts: We’re an idea.
523 00:41:21.720 ⇒ 00:41:22.200 Amber Lin: Right there.
524 00:41:22.200 ⇒ 00:41:23.460 Samuel Roberts: Oh, boy, oh boy.
525 00:41:23.460 ⇒ 00:41:24.420 Mustafa Raja: Oh, no.
526 00:41:24.420 ⇒ 00:41:29.770 Amber Lin: I guess we can just do… Hmm.
527 00:41:32.260 ⇒ 00:41:34.319 Mustafa Raja: Oh, yeah.
528 00:41:34.320 ⇒ 00:41:39.729 Amber Lin: Like, worst case, we can just do, like, 2020… just include all the dates.
529 00:41:40.150 ⇒ 00:41:41.339 Samuel Roberts: I think I got 25.
530 00:41:41.340 ⇒ 00:41:48.339 Mustafa Raja: Oh, so… So we need, for 2025-12 and 2025-11, right?
531 00:41:50.720 ⇒ 00:41:51.200 Amber Lin: Huh?
532 00:41:51.200 ⇒ 00:41:52.150 Samuel Roberts: Yeah, yeah, do that.
533 00:41:52.510 ⇒ 00:41:52.910 Samuel Roberts: actually.
534 00:41:52.910 ⇒ 00:41:54.419 Mustafa Raja: That amount’s going to be successful.
535 00:41:54.420 ⇒ 00:41:57.340 Amber Lin: Great, awesome. Oh, we’re so smart.
536 00:41:58.910 ⇒ 00:42:00.160 Mustafa Raja: Oh, man…
537 00:42:03.550 ⇒ 00:42:05.919 Amber Lin: Record ID is like.
538 00:42:09.640 ⇒ 00:42:14.699 Amber Lin: Oh, I’m sorry, just like, not is like. I have not worked here before.
539 00:42:20.620 ⇒ 00:42:26.740 Amber Lin: You might need to add a percentage sign before and after Yeah.
540 00:42:27.820 ⇒ 00:42:29.090 Amber Lin: Can you check that?
541 00:42:35.420 ⇒ 00:42:38.740 Amber Lin: Mmm… Oh.
542 00:42:38.740 ⇒ 00:42:40.320 Mustafa Raja: It might just be this.
543 00:42:40.320 ⇒ 00:42:41.729 Amber Lin: Yeah, single quotes?
544 00:42:47.590 ⇒ 00:42:48.370 Mustafa Raja: I understand.
545 00:42:48.370 ⇒ 00:42:51.599 Samuel Roberts: What is it? We’re a record? Okay, cool. How many we got there?
546 00:42:53.190 ⇒ 00:42:53.970 Mustafa Raja: Yeah.
547 00:42:54.300 ⇒ 00:43:00.070 Amber Lin: a thousand… 300. I guess we need to also say where feedback.
548 00:43:01.040 ⇒ 00:43:02.159 Mustafa Raja: Yeah, we need to.
549 00:43:02.160 ⇒ 00:43:03.070 Samuel Roberts: Okay.
550 00:43:04.630 ⇒ 00:43:05.100 Amber Lin: And…
551 00:43:05.120 ⇒ 00:43:06.130 Mustafa Raja: And…
552 00:43:09.250 ⇒ 00:43:11.910 Samuel Roberts: What was it? I’m just trying to follow along, where record ID?
553 00:43:13.660 ⇒ 00:43:16.835 Mustafa Raja: Tom’s feedback… close…
554 00:43:21.260 ⇒ 00:43:27.899 Mustafa Raja: Down… They should do.
555 00:43:29.300 ⇒ 00:43:30.140 Mustafa Raja: Oh.
556 00:43:30.550 ⇒ 00:43:31.450 Amber Lin: We’re records.
557 00:43:31.450 ⇒ 00:43:33.260 Mustafa Raja: Where is it that, though?
558 00:43:33.260 ⇒ 00:43:36.690 Amber Lin: And just add that behind the OR.
559 00:43:37.160 ⇒ 00:43:38.130 Samuel Roberts: Or…
560 00:43:38.540 ⇒ 00:43:41.040 Amber Lin: Or where record ID?
561 00:43:41.180 ⇒ 00:43:44.480 Amber Lin: Oh, sorry, or record ID, yeah, there we go.
562 00:43:49.660 ⇒ 00:43:50.389 Mustafa Raja: 10 minutes.
563 00:43:51.120 ⇒ 00:43:53.159 Samuel Roberts: What do we got? Sorry, I’m still looking at it here.
564 00:43:53.160 ⇒ 00:43:58.570 Amber Lin: Do we have the… November ones?
565 00:43:59.900 ⇒ 00:44:02.020 Mustafa Raja: Yeah, I’m trying to see that.
566 00:44:02.780 ⇒ 00:44:04.620 Samuel Roberts: Let me… I’m gonna get rid of everything but.
567 00:44:04.620 ⇒ 00:44:06.329 Mustafa Raja: Oh, I don’t think we have those.
568 00:44:08.940 ⇒ 00:44:09.730 Samuel Roberts: Hmm.
569 00:44:13.320 ⇒ 00:44:15.940 Samuel Roberts: And I get no results. Oh, that’s because I messed that up, hold off.
570 00:44:17.130 ⇒ 00:44:22.640 Samuel Roberts: I get 100… yeah, I get 100 rows from that, 25, 11, yeah, so are you not getting those?
571 00:44:23.670 ⇒ 00:44:26.489 Amber Lin: Can you delete the like to…
572 00:44:26.670 ⇒ 00:44:29.679 Amber Lin: 20… just delete the Nov… December one.
573 00:44:29.680 ⇒ 00:44:32.440 Samuel Roberts: Yeah, I’m getting… hold on, let me see.
574 00:44:33.640 ⇒ 00:44:36.729 Amber Lin: Do we even have… we don’t even have that?
575 00:44:36.730 ⇒ 00:44:40.169 Samuel Roberts: I get 1885 when I do Just November.
576 00:44:40.580 ⇒ 00:44:42.990 Amber Lin: Okay, thumbs down.
577 00:44:43.440 ⇒ 00:44:47.139 Samuel Roberts: Nope, just total. I’m just still trying to make sure we were having… we had November.
578 00:44:48.490 ⇒ 00:44:51.560 Amber Lin: Okay. So there is…
579 00:44:51.920 ⇒ 00:44:52.800 Samuel Roberts: They’re… yeah.
580 00:44:58.350 ⇒ 00:45:00.800 Samuel Roberts: Hold on, if I do that, let me see what happens there…
581 00:45:04.320 ⇒ 00:45:08.540 Samuel Roberts: 1885… why are we getting only 1885 now?
582 00:45:22.750 ⇒ 00:45:23.420 Mustafa Raja: Hmm.
583 00:45:26.050 ⇒ 00:45:28.349 Mustafa Raja: For me, it says… oh, for me, it says…
584 00:45:28.350 ⇒ 00:45:32.860 Samuel Roberts: I just keep messing up. I get $32.81 total for November and December.
585 00:45:35.570 ⇒ 00:45:36.990 Mustafa Raja: With thumbs down?
586 00:45:36.990 ⇒ 00:45:43.970 Samuel Roberts: Nope, no, that’s just total. I was still trying to make it work where I was getting, issues with mine. And then we want where, record ID, record ID…
587 00:45:44.080 ⇒ 00:45:45.989 Samuel Roberts: And, what’s the column that’s.
588 00:45:45.990 ⇒ 00:45:49.480 Mustafa Raja: Oh, I don’t think 11 has thumbs down.
589 00:45:51.000 ⇒ 00:45:52.069 Samuel Roberts: Oh, is that the issue?
590 00:45:52.460 ⇒ 00:45:55.630 Amber Lin: That doesn’t seem right.
591 00:45:57.230 ⇒ 00:45:59.270 Amber Lin: Could you just check…
592 00:46:00.260 ⇒ 00:46:03.849 Mustafa Raja: Yeah, so if I… if I… yeah, if I remove this…
593 00:46:05.550 ⇒ 00:46:10.559 Mustafa Raja: It says no results, and then if I remove thumbs down, there will be results.
594 00:46:12.980 ⇒ 00:46:15.519 Samuel Roberts: Wait, go back to… to do the thumbs down again? Can you…
595 00:46:21.290 ⇒ 00:46:23.250 Amber Lin: Could you scroll on the right?
596 00:46:23.690 ⇒ 00:46:27.680 Amber Lin: to, like, November or sometime to check.
597 00:46:28.460 ⇒ 00:46:30.699 Samuel Roberts: So you said there’s none in November, is that what you think?
598 00:46:30.700 ⇒ 00:46:32.900 Amber Lin: That doesn’t make too much sense to me.
599 00:46:32.900 ⇒ 00:46:37.760 Samuel Roberts: I’m getting… no, thumbs down is not working, but my bad, I’m mistyping things here.
600 00:46:40.040 ⇒ 00:46:41.070 Samuel Roberts: Okay.
601 00:46:45.240 ⇒ 00:46:46.490 Mustafa Raja: Yeah, for December, this…
602 00:46:48.470 ⇒ 00:46:49.200 Amber Lin: Hmm?
603 00:46:57.130 ⇒ 00:46:57.840 Amber Lin: Hmm?
604 00:47:10.850 ⇒ 00:47:12.960 Amber Lin: Okay, something’s wrong.
605 00:47:14.120 ⇒ 00:47:18.210 Samuel Roberts: Definitely not the right query, but… Beautiful.
606 00:47:23.220 ⇒ 00:47:27.880 Amber Lin: Could we just… Could you just download all the thumbs-down ones?
607 00:47:29.760 ⇒ 00:47:30.900 Amber Lin: And then we’ll just manually.
608 00:47:30.900 ⇒ 00:47:38.610 Mustafa Raja: I mean, what we could do is just get, 2025, 12, and 11 ones, and in the notebook, we could filter.
609 00:47:38.610 ⇒ 00:47:39.330 Samuel Roberts: Yeah.
610 00:47:39.330 ⇒ 00:47:39.889 Amber Lin: No.
611 00:47:39.890 ⇒ 00:47:41.289 Samuel Roberts: Oh, here we go, hold on, what did I get here?
612 00:47:41.290 ⇒ 00:47:47.609 Amber Lin: Just filter for, like, thumbs down once, not by date, because we don’t have too many of those anyways.
613 00:47:49.330 ⇒ 00:47:52.219 Samuel Roberts: Oh, it’s a text field, that’s the issue here, I’m seeing, okay.
614 00:47:53.190 ⇒ 00:47:53.680 Samuel Roberts: Beautiful.
615 00:47:53.680 ⇒ 00:47:54.560 Amber Lin: Who’s…
616 00:47:58.540 ⇒ 00:47:59.870 Samuel Roberts: Thumb down.
617 00:48:01.290 ⇒ 00:48:02.499 Mustafa Raja: Oh, it’s thumb down?
618 00:48:04.470 ⇒ 00:48:06.160 Samuel Roberts: It’s thumb down.
619 00:48:06.160 ⇒ 00:48:08.840 Amber Lin: Oh, my… Wow.
620 00:48:09.690 ⇒ 00:48:10.840 Amber Lin: Okay.
621 00:48:12.520 ⇒ 00:48:20.939 Amber Lin: Cool, okay, then let’s add the date field, or we can just download this 900 rows, and then copy the top ones.
622 00:48:22.790 ⇒ 00:48:27.620 Samuel Roberts: I guess I could… yeah, hold on, I can do it right, let me just… let me just do that, hold on. I’m gonna download all of our.
623 00:48:28.070 ⇒ 00:48:30.369 Amber Lin: Yeah, just… 3281 rows.
624 00:48:31.170 ⇒ 00:48:41.620 Amber Lin: Well, I don’t think we need that many. I don’t think we got 30 to 81 feedback. We just need the 900 thumbs down feedbacks that Mustafa has here.
625 00:48:44.060 ⇒ 00:48:45.290 Samuel Roberts: Are those all thumbs down?
626 00:48:45.590 ⇒ 00:48:50.460 Amber Lin: Yeah, yeah, just all the thumbs down, or just what we have here, yeah, that’s great.
627 00:48:51.070 ⇒ 00:48:54.489 Samuel Roberts: Go to the… scroll over to the thumb feedback, thumbs Feedback column.
628 00:48:57.990 ⇒ 00:48:58.880 Amber Lin: Huh?
629 00:48:59.790 ⇒ 00:49:01.529 Samuel Roberts: Yeah, that’s what I was worried about.
630 00:49:01.530 ⇒ 00:49:02.600 Amber Lin: Yeah.
631 00:49:02.600 ⇒ 00:49:05.529 Samuel Roberts: That’s why I’m trying to figure out how to actually get… how to sort it here real quick.
632 00:49:05.530 ⇒ 00:49:06.710 Mustafa Raja: Oh, hmm…
633 00:49:06.710 ⇒ 00:49:13.089 Amber Lin: Gustavo, can you just download everything that’s thumbs down? Just record… re-delete all the record IDs?
634 00:49:13.580 ⇒ 00:49:18.280 Amber Lin: And then we’ll just… we’ll filter by data after we download it, is that okay?
635 00:49:18.990 ⇒ 00:49:20.059 Mustafa Raja: Yeah, it’s good now.
636 00:49:20.060 ⇒ 00:49:21.560 Amber Lin: Awesome. Alright.
637 00:49:21.560 ⇒ 00:49:22.990 Mustafa Raja: We have 190 rows.
638 00:49:23.030 ⇒ 00:49:27.919 Amber Lin: Alright, yeah, great, let’s download that, and then here is the spreadsheet.
639 00:49:28.190 ⇒ 00:49:29.949 Amber Lin: Where we can put it in.
640 00:49:30.810 ⇒ 00:49:35.159 Amber Lin: I’ll put it in our chart.
641 00:49:35.160 ⇒ 00:49:36.200 Mustafa Raja: Hmm…
642 00:49:36.200 ⇒ 00:49:37.519 Amber Lin: In our Zoom chat.
643 00:49:37.730 ⇒ 00:49:39.560 Samuel Roberts: Wait, so how many did we end up with? I’m sorry?
644 00:49:39.560 ⇒ 00:49:39.990 Amber Lin: 19.
645 00:49:39.990 ⇒ 00:49:41.020 Mustafa Raja: 190.
646 00:49:41.150 ⇒ 00:49:48.310 Amber Lin: Which sounds about right. Might be a little bit more, but it lasts 2 months, so… Okay. About… that’s about…
647 00:49:49.180 ⇒ 00:49:54.139 Amber Lin: Yeah, that’s about, like, 23 per week.
648 00:49:55.960 ⇒ 00:49:56.810 Amber Lin: Cool.
649 00:50:04.430 ⇒ 00:50:11.790 Amber Lin: Yeah, I don’t think we will be able to go through everything, so let’s…
650 00:50:12.020 ⇒ 00:50:19.739 Amber Lin: Let’s read, and then find the ones… That is about zip codes.
651 00:50:20.000 ⇒ 00:50:24.380 Amber Lin: So, I’ll just make a field on the…
652 00:50:28.630 ⇒ 00:50:30.480 Amber Lin: Make a field here.
653 00:50:31.630 ⇒ 00:50:35.790 Amber Lin: And then… Cool.
654 00:50:50.320 ⇒ 00:50:51.380 Amber Lin: Alright.
655 00:50:53.620 ⇒ 00:50:55.230 Mustafa Raja: Why is this a checkbox?
656 00:50:55.660 ⇒ 00:50:56.290 Amber Lin: Huh?
657 00:50:56.770 ⇒ 00:50:57.490 Samuel Roberts: Hmm.
658 00:50:59.510 ⇒ 00:51:06.149 Amber Lin: Oh, I added a checkbox so we can qualify if it is zip code questions, because we can’t… we don’t.
659 00:51:06.150 ⇒ 00:51:07.310 Mustafa Raja: Oh, okay.
660 00:51:07.310 ⇒ 00:51:08.220 Amber Lin: everything.
661 00:51:08.450 ⇒ 00:51:09.150 Mustafa Raja: Okay, okay.
662 00:51:09.150 ⇒ 00:51:09.710 Amber Lin: Yeah.
663 00:51:09.840 ⇒ 00:51:11.050 Mustafa Raja: So… .
664 00:51:11.950 ⇒ 00:51:20.229 Amber Lin: K… You guys also helped me check that of… What’s… what has…
665 00:51:20.230 ⇒ 00:51:22.589 Mustafa Raja: Can Gemini help this, help us here?
666 00:51:22.590 ⇒ 00:51:24.800 Amber Lin: Sure, how would we do that?
667 00:51:29.380 ⇒ 00:51:34.549 Amber Lin: Like, anything that mentions zip codes or locations?
668 00:51:35.220 ⇒ 00:51:43.980 Amber Lin: So there’s stuff about, like, San Marcos, Corpus Christi, or 78703, like, what about those stuff?
669 00:52:05.670 ⇒ 00:52:09.159 Mustafa Raja: I’m just, trying to see if this works or not.
670 00:52:31.150 ⇒ 00:52:33.320 Mustafa Raja: And this… this formula looks good.
671 00:52:38.180 ⇒ 00:52:39.219 Samuel Roberts: What are you looking at here?
672 00:52:41.600 ⇒ 00:52:42.900 Samuel Roberts: Oh, I’m zoomed in as well.
673 00:52:55.770 ⇒ 00:52:56.330 Amber Lin: Hmm.
674 00:53:16.650 ⇒ 00:53:18.010 Amber Lin: Oh, wow.
675 00:53:18.370 ⇒ 00:53:19.970 Amber Lin: Wow, that’s crazy.
676 00:53:25.420 ⇒ 00:53:26.270 Mustafa Raja: What happened?
677 00:53:26.270 ⇒ 00:53:29.200 Amber Lin: Did it just… I think it just clicked the…
678 00:53:29.710 ⇒ 00:53:33.630 Amber Lin: It helped us sort for the bottom ones, too.
679 00:53:34.270 ⇒ 00:53:37.000 Samuel Roberts: Oh, I didn’t… wait, what happened? I’m not sure what I’m seeing here.
680 00:53:37.000 ⇒ 00:53:41.680 Amber Lin: I think Mustafa applied an AI field-based.
681 00:53:41.680 ⇒ 00:53:42.520 Samuel Roberts: Oh!
682 00:53:42.520 ⇒ 00:53:47.760 Amber Lin: field, so we’re check… we’re right now trying to select the questions that.
683 00:53:47.760 ⇒ 00:53:49.950 Samuel Roberts: Got it, okay, I see it now. Sorry, I was…
684 00:53:49.950 ⇒ 00:53:51.490 Amber Lin: Yeah.
685 00:53:51.940 ⇒ 00:53:52.360 Samuel Roberts: That’s cool.
686 00:53:52.360 ⇒ 00:53:56.829 Mustafa Raja: So now it, it only, it only, checked, I believe,
687 00:53:57.330 ⇒ 00:53:59.660 Mustafa Raja: Anything that has 5 digits.
688 00:53:59.900 ⇒ 00:54:01.190 Mustafa Raja: You know?
689 00:54:01.190 ⇒ 00:54:01.530 Amber Lin: Okay.
690 00:54:01.530 ⇒ 00:54:02.899 Mustafa Raja: In input.
691 00:54:02.900 ⇒ 00:54:10.700 Amber Lin: Can I… can you also ask it to check those dimension, like, location or town names?
692 00:54:11.930 ⇒ 00:54:16.269 Mustafa Raja: I believe we’ll have to tell it the town names or something.
693 00:54:16.600 ⇒ 00:54:17.130 Amber Lin: I see.
694 00:54:17.130 ⇒ 00:54:18.770 Samuel Roberts: Can we get a list out of it, maybe?
695 00:54:19.680 ⇒ 00:54:22.160 Mustafa Raja: Yeah, I know corporate…
696 00:54:22.160 ⇒ 00:54:26.650 Amber Lin: Not much left, can I just manually… Manually…
697 00:54:26.650 ⇒ 00:54:27.260 Mustafa Raja: Yay!
698 00:54:28.210 ⇒ 00:54:29.730 Amber Lin: Okay, cool.
699 00:54:29.730 ⇒ 00:54:30.510 Mustafa Raja: Yeah.
700 00:54:31.070 ⇒ 00:54:34.569 Samuel Roberts: 36 checkbox… check boxes, is what I’m seeing here.
701 00:54:34.570 ⇒ 00:54:35.400 Amber Lin: Okay.
702 00:54:36.350 ⇒ 00:54:37.510 Amber Lin: Yeah.
703 00:54:37.700 ⇒ 00:54:43.029 Amber Lin: That’s the… I can’t… right now, I can’t manually do that anymore.
704 00:54:43.030 ⇒ 00:54:44.930 Samuel Roberts: Oh, because it’s still… yeah.
705 00:54:44.930 ⇒ 00:54:50.669 Amber Lin: Is this still AI-fueling? Mustafa, can I do the manual ones right now? Can I try?
706 00:54:50.670 ⇒ 00:54:54.000 Mustafa Raja: I try… I closed it, but…
707 00:54:54.000 ⇒ 00:54:55.110 Amber Lin: I don’t think so.
708 00:54:55.180 ⇒ 00:55:00.469 Mustafa Raja: Oh, so what, what’s happened is, we have, what’s it called? Formula.
709 00:55:01.010 ⇒ 00:55:01.790 Amber Lin: I see.
710 00:55:01.790 ⇒ 00:55:02.140 Mustafa Raja: Come on.
711 00:55:02.800 ⇒ 00:55:03.280 Mustafa Raja: Yeah.
712 00:55:03.280 ⇒ 00:55:05.629 Amber Lin: I’ll do one for manual checks.
713 00:55:06.520 ⇒ 00:55:07.290 Mustafa Raja: Oh, yeah.
714 00:55:07.530 ⇒ 00:55:10.260 Samuel Roberts: Well, can we hide all the ones that are not checked anymore? Like, do we need them?
715 00:55:10.260 ⇒ 00:55:16.930 Amber Lin: I want to make sure that we also cover some of them, because this also only covers stuff that is…
716 00:55:16.930 ⇒ 00:55:20.160 Samuel Roberts: Oh, I see what you’re saying. Okay, yeah, yeah, I… the one.
717 00:55:20.560 ⇒ 00:55:22.960 Amber Lin: Found names and this and that.
718 00:55:23.110 ⇒ 00:55:24.120 Samuel Roberts: Okay, good call.
719 00:55:24.120 ⇒ 00:55:37.080 Amber Lin: So, checkbox… How do I do this?
720 00:55:40.320 ⇒ 00:55:48.630 Amber Lin: Could you add this, like, a third column that is essentially, like, if the previous two is checked, make sure that it’s checked?
721 00:55:49.760 ⇒ 00:55:50.630 Samuel Roberts: Well, as if…
722 00:55:50.820 ⇒ 00:55:51.420 Mustafa Raja: Yep.
723 00:55:51.680 ⇒ 00:55:52.340 Samuel Roberts: I got it.
724 00:55:52.530 ⇒ 00:55:53.160 Amber Lin: Yeah, okay.
725 00:55:53.160 ⇒ 00:55:54.250 Mustafa Raja: Oh, okay, thank you.
726 00:55:54.250 ⇒ 00:55:54.880 Samuel Roberts: on it.
727 00:56:02.870 ⇒ 00:56:03.620 Uttam Kumaran: Hey, guys.
728 00:56:04.180 ⇒ 00:56:05.450 Amber Lin: Hello! Yay.
729 00:56:06.100 ⇒ 00:56:07.480 Uttam Kumaran: Okay…
730 00:56:07.830 ⇒ 00:56:15.469 Amber Lin: Alright, we found all the questions for the last two months. We’re narrowing it down to SIPCO-related questions.
731 00:56:15.470 ⇒ 00:56:15.950 Uttam Kumaran: Okay.
732 00:56:15.950 ⇒ 00:56:20.710 Amber Lin: Right now, it’s only gonna take another, like, 3-5 minutes.
733 00:56:20.850 ⇒ 00:56:21.310 Uttam Kumaran: Okay.
734 00:56:21.310 ⇒ 00:56:24.900 Samuel Roberts: You want… so you want to see ones that are… have… are both checked? Is that what you were saying?
735 00:56:24.900 ⇒ 00:56:25.300 Amber Lin: Yay.
736 00:56:25.300 ⇒ 00:56:26.830 Samuel Roberts: Either, either one is tracked.
737 00:56:27.200 ⇒ 00:56:29.550 Amber Lin: either one, please.
738 00:56:29.550 ⇒ 00:56:30.570 Samuel Roberts: Okay, that’s what I meant.
739 00:56:32.940 ⇒ 00:56:34.139 Samuel Roberts: Yeah, I did that bad.
740 00:56:35.480 ⇒ 00:56:36.220 Samuel Roberts: Oops.
741 00:56:38.810 ⇒ 00:56:39.799 Samuel Roberts: Why is it not going back?
742 00:56:42.010 ⇒ 00:56:42.929 Samuel Roberts: There you go.
743 00:56:58.050 ⇒ 00:56:58.990 Samuel Roberts: Excuse me.
744 00:57:01.480 ⇒ 00:57:02.200 Amber Lin: Great.
745 00:59:35.910 ⇒ 00:59:37.270 Samuel Roberts: Excuse me.
746 00:59:41.620 ⇒ 00:59:46.500 Uttam Kumaran: And there’s no way to do, like, Lang views, right?
747 00:59:47.000 ⇒ 00:59:48.850 Uttam Kumaran: Right now, but is there any.
748 00:59:48.850 ⇒ 00:59:49.450 Mustafa Raja: Yeah, no.
749 00:59:49.720 ⇒ 00:59:52.109 Uttam Kumaran: Is there any way to do, like, evals?
750 00:59:54.890 ⇒ 00:59:58.640 Samuel Roberts: I think NADN does have some… Fairly basic stuff.
751 00:59:58.640 ⇒ 01:00:03.799 Uttam Kumaran: But were we using… we were using BrainTrust for, like, more than a year. What happened to the Brain Trust stuff?
752 01:00:10.750 ⇒ 01:00:11.990 Samuel Roberts: trust for…
753 01:00:13.100 ⇒ 01:00:15.820 Uttam Kumaran: like, we were using Brain Trust for Andy.
754 01:00:19.330 ⇒ 01:00:23.309 Uttam Kumaran: I guess, like, is Casey the only one working on this, like…
755 01:00:23.870 ⇒ 01:00:29.030 Uttam Kumaran: How does it… how do you guys not know if we’re using Brain Trust for Andy?
756 01:00:29.030 ⇒ 01:00:32.580 Mustafa Raja: Yeah, I didn’t really get,
757 01:00:33.200 ⇒ 01:00:36.479 Mustafa Raja: see anything on BrainTrust regarding ABC, to be honest.
758 01:00:36.730 ⇒ 01:00:38.009 Samuel Roberts: Yeah, I don’t remember that.
759 01:00:39.240 ⇒ 01:00:42.099 Uttam Kumaran: I just opened… I opened Brain Trust, and the first thing is.
760 01:00:42.100 ⇒ 01:00:45.690 Samuel Roberts: Yeah, that’s what I’m looking at. Yeah, what was this? This was back in March?
761 01:00:48.380 ⇒ 01:00:49.370 Uttam Kumaran: Yeah, but like…
762 01:00:49.370 ⇒ 01:00:55.160 Samuel Roberts: May? Miguel? Yeah, I don’t… We have not touched this since I’ve been around.
763 01:00:56.950 ⇒ 01:01:03.639 Uttam Kumaran: Okay… I guess this is just, like, kind of concerning, like, we just feel like…
764 01:01:03.980 ⇒ 01:01:06.230 Uttam Kumaran: Casey’s the only one running this.
765 01:01:09.050 ⇒ 01:01:11.340 Uttam Kumaran: Okay, I mean, I just wanna, like…
766 01:01:14.120 ⇒ 01:01:20.709 Uttam Kumaran: It’s just so tough, guys, because so many people are using this product, And it keeps failing.
767 01:01:21.080 ⇒ 01:01:22.610 Uttam Kumaran: I don’t know what to do.
768 01:01:23.130 ⇒ 01:01:25.070 Uttam Kumaran: And I feel like…
769 01:01:26.950 ⇒ 01:01:31.219 Uttam Kumaran: I don’t… I don’t know, I just… I’m just trying to, like… trying to guide us to, like…
770 01:01:31.320 ⇒ 01:01:38.200 Uttam Kumaran: Caring about it, but… It’s not clear that we have a solution towards, like.
771 01:01:38.390 ⇒ 01:01:44.100 Uttam Kumaran: Basic monitoring of if the answers are correct or not. Like, what kind of questions there can be.
772 01:01:44.310 ⇒ 01:01:46.469 Uttam Kumaran: How do we know what correct means?
773 01:01:46.790 ⇒ 01:01:54.329 Uttam Kumaran: So I’m just having trouble trying to, like, help here. I mean, I’m just gonna have to run, like, we’re just gonna have to do daily stand-ups.
774 01:01:54.490 ⇒ 01:01:55.460 Uttam Kumaran: on this.
775 01:01:56.120 ⇒ 01:01:57.530 Uttam Kumaran: Until we figure it out.
776 01:01:59.120 ⇒ 01:01:59.820 Uttam Kumaran: You know…
777 01:02:08.810 ⇒ 01:02:10.020 Samuel Roberts: Yeah, I mean, I think…
778 01:02:10.800 ⇒ 01:02:20.179 Samuel Roberts: I mean, I’ve been sort of somewhat avoiding the N8N, like, eval stuff, because I didn’t know what we had there, and brain trust at all, certainly, but I knew the N8N stuff was kind of limited, but…
779 01:02:20.630 ⇒ 01:02:21.520 Uttam Kumaran: Okay.
780 01:02:21.520 ⇒ 01:02:22.969 Samuel Roberts: We can.
781 01:02:22.970 ⇒ 01:02:27.999 Uttam Kumaran: Do we have… do we know for the migration? Like, where are we? Like, is this, like…
782 01:02:29.370 ⇒ 01:02:33.780 Uttam Kumaran: Do we have a sense of, like, how much is left? I tried to ask for…
783 01:02:34.400 ⇒ 01:02:41.450 Uttam Kumaran: like, a Gantt, or even some… something, but do we know how much is left? Like… When… yeah.
784 01:02:43.400 ⇒ 01:03:02.259 Samuel Roberts: I mean, we’re kind of experimenting with some other agent models, and that’s why I was, like, not committing too hard yet, but what we have right now is basically just, like, Andy in Mastra, but we haven’t tested it to full, like, completion yet. But what Mustaf and I had discussed was trying to… because a big…
785 01:03:02.960 ⇒ 01:03:08.619 Samuel Roberts: or an issue, I believe, is the fact that there’s a huge amount of context in the system prompt.
786 01:03:11.560 ⇒ 01:03:15.490 Samuel Roberts: And so we’re trying to move… like, get rid of some of that so that we can actually get more…
787 01:03:15.910 ⇒ 01:03:22.199 Samuel Roberts: Answers from specific agents that know which dock to look in, and not just… like, the entire…
788 01:03:22.840 ⇒ 01:03:25.869 Samuel Roberts: What is it, the cancellation flow? Is that it, Mustafa?
789 01:03:25.870 ⇒ 01:03:26.670 Mustafa Raja: Yeah.
790 01:03:26.880 ⇒ 01:03:32.459 Samuel Roberts: Takes up, like, 75% of the… Like, it’s, it’s, it’s… it’s…
791 01:03:32.690 ⇒ 01:03:40.329 Samuel Roberts: not great. But yeah, I mean, I think this is actually more helpful than not, because we gotta see, like, what other things are breaking here.
792 01:03:41.040 ⇒ 01:03:50.260 Samuel Roberts: Because I don’t have a good sense of, like… I knew there were issues with the DB stuff, and there’s some… the Zips DB, in terms of, like, what is getting queried, like we saw today.
793 01:03:50.800 ⇒ 01:03:54.749 Samuel Roberts: But I’m not sure the other, like, pieces of it.
794 01:03:57.710 ⇒ 01:03:59.440 Uttam Kumaran: Okay, yeah, just…
795 01:04:00.830 ⇒ 01:04:09.570 Uttam Kumaran: Okay, I mean, we’ll talk about it tomorrow in Casey gets here, but I just want to work on, like, a much more detailed plan, and, like, I need to know when it will be done.
796 01:04:09.570 ⇒ 01:04:10.230 Samuel Roberts: Yeah.
797 01:04:10.230 ⇒ 01:04:14.039 Uttam Kumaran: Otherwise, like, I… like, we’re completely… we’re completely jammed here.
798 01:04:14.230 ⇒ 01:04:24.890 Uttam Kumaran: Like, I can’t go to the client, and… Right now, like, Amber is taking the brunt of the, this is not working. Yeah. Like, I don’t… I’m gonna start to take that, because…
799 01:04:25.020 ⇒ 01:04:26.420 Uttam Kumaran: I have to fix this.
800 01:04:26.420 ⇒ 01:04:27.680 Samuel Roberts: So…
801 01:04:28.030 ⇒ 01:04:29.800 Uttam Kumaran: I just need to know, like.
802 01:04:30.600 ⇒ 01:04:35.809 Uttam Kumaran: where we are with the migration at all. And, like, I know we had a plan document.
803 01:04:36.660 ⇒ 01:04:39.829 Uttam Kumaran: And it’s at 5 weeks.
804 01:04:40.490 ⇒ 01:04:43.050 Uttam Kumaran: But I don’t know, it’s been, like, 3 weeks.
805 01:04:45.310 ⇒ 01:04:53.069 Uttam Kumaran: And I don’t… yeah, so we’ll… I guess we’ll talk about it tomorrow. Let’s, so can I… so let me look at… so can you walk… walk me through this?
806 01:04:53.520 ⇒ 01:04:57.339 Uttam Kumaran: Spreadsheet hub, and we’ll talk about that right now.
807 01:04:58.790 ⇒ 01:04:59.640 Uttam Kumaran: Yeah.
808 01:04:59.640 ⇒ 01:05:02.739 Amber Lin: We grabbed all the past…
809 01:05:04.370 ⇒ 01:05:10.150 Amber Lin: Errors that… and the feedback related to zip codes from the past 2 months.
810 01:05:10.540 ⇒ 01:05:10.860 Uttam Kumaran: Right.
811 01:05:10.860 ⇒ 01:05:14.119 Amber Lin: Now, going… I’m going through it to see…
812 01:05:14.610 ⇒ 01:05:28.309 Amber Lin: One, if I can put the expected correct answer down, and two, I’ll add a note to say what this… what error, is causing this.
813 01:05:29.210 ⇒ 01:05:31.069 Uttam Kumaran: So what is this quality score?
814 01:05:31.280 ⇒ 01:05:34.199 Amber Lin: That is our previous…
815 01:05:34.530 ⇒ 01:05:43.429 Amber Lin: Quality 4, which here we can see that this is not working correctly, because it’s giving high quality stores to wrong answers.
816 01:05:43.700 ⇒ 01:05:46.019 Uttam Kumaran: But where does the quality score come from?
817 01:05:46.320 ⇒ 01:05:49.660 Amber Lin: Likely Brain Trust that we edited in March.
818 01:05:50.780 ⇒ 01:05:51.920 Uttam Kumaran: Okay…
819 01:05:51.920 ⇒ 01:05:56.279 Samuel Roberts: I’m gonna score here… I’m looking at the NADN, all I see is it…
820 01:05:58.930 ⇒ 01:06:01.760 Samuel Roberts: It’s just a LM as a judge.
821 01:06:02.380 ⇒ 01:06:03.090 Uttam Kumaran: Okay.
822 01:06:04.000 ⇒ 01:06:07.200 Samuel Roberts: Just, yeah.
823 01:06:08.130 ⇒ 01:06:10.629 Samuel Roberts: It’s just a prompt and an AN.
824 01:06:12.480 ⇒ 01:06:20.160 Uttam Kumaran: So, the one that, bo just asked about… Is that the bottom here?
825 01:06:22.440 ⇒ 01:06:27.150 Amber Lin: That would be… Mmm…
826 01:06:27.870 ⇒ 01:06:29.370 Uttam Kumaran: Okay, yeah, I see it here.
827 01:06:29.520 ⇒ 01:06:30.190 Amber Lin: Cool.
828 01:06:37.160 ⇒ 01:06:38.220 Uttam Kumaran: Okay, I will…
829 01:06:43.990 ⇒ 01:06:45.919 Uttam Kumaran: Do we have, okay.
830 01:07:07.600 ⇒ 01:07:10.489 Uttam Kumaran: Will you have the execution date, or no?
831 01:07:11.740 ⇒ 01:07:13.780 Mustafa Raja: Yeah, it’s in the record ID.
832 01:07:14.000 ⇒ 01:07:15.360 Amber Lin: in here.
833 01:07:15.810 ⇒ 01:07:18.039 Mustafa Raja: We can extract that.
834 01:07:18.750 ⇒ 01:07:19.280 Samuel Roberts: Yeah.
835 01:07:19.280 ⇒ 01:07:21.480 Mustafa Raja: With some regular expressions or something.
836 01:07:27.730 ⇒ 01:07:32.390 Uttam Kumaran: Alright, I guess I’m more interested in us just making sure that there’s, like, a priority list.
837 01:07:34.200 ⇒ 01:07:39.899 Uttam Kumaran: Of, like, what, oh, my neighbors.
838 01:07:57.930 ⇒ 01:08:03.540 Amber Lin: Let’s say that everything that says, I’m sorry, I don’t know, is top priority.
839 01:08:03.680 ⇒ 01:08:07.009 Amber Lin: And then we can look at…
840 01:08:07.840 ⇒ 01:08:14.850 Amber Lin: Stuff where it’s completely raw, unless we can look at ones that’s missing some details.
841 01:08:15.530 ⇒ 01:08:16.479 Uttam Kumaran: Okay.
842 01:08:57.310 ⇒ 01:08:59.949 Uttam Kumaran: So, this is all, I don’t know.
843 01:09:04.510 ⇒ 01:09:05.940 Amber Lin: Still filtering.
844 01:09:06.779 ⇒ 01:09:10.719 Uttam Kumaran: This… but these are also all, like, 13 seconds, that’s expected.
845 01:09:13.689 ⇒ 01:09:17.079 Amber Lin: For zip code questions, it usually takes at least 10.
846 01:09:47.729 ⇒ 01:09:52.729 Uttam Kumaran: So, can we also start to put in for these P0s, like, a link to the trace?
847 01:09:56.250 ⇒ 01:09:57.710 Samuel Roberts: Yeah, I can start pulling those.
848 01:10:01.640 ⇒ 01:10:03.629 Samuel Roberts: Some of these times match up, that’ll be…
849 01:10:22.590 ⇒ 01:10:27.359 Samuel Roberts: I’m gonna strip out the… The date and time, because it’ll be easier.
850 01:12:57.880 ⇒ 01:13:03.600 Uttam Kumaran: And then, maybe, I guess, Sam, are you working on getting the traces? And Mustavo, let’s just look at, like, one of the…
851 01:13:04.720 ⇒ 01:13:08.960 Uttam Kumaran: I’d just like to look at… look through the primary trace that went wrong here.
852 01:13:09.200 ⇒ 01:13:11.050 Uttam Kumaran: Which is the Bo Jenkins one.
853 01:13:11.990 ⇒ 01:13:13.760 Mustafa Raja: Yep,
854 01:13:27.990 ⇒ 01:13:29.570 Mustafa Raja: Look at this one…
855 01:14:04.370 ⇒ 01:14:05.840 Uttam Kumaran: Can you send it to me, please?
856 01:14:07.300 ⇒ 01:14:11.519 Uttam Kumaran: Or put it in, yeah, if I… send it to me, you can put it in the spreadsheet, so…
857 01:14:13.120 ⇒ 01:14:14.550 Mustafa Raja: The link, right?
858 01:14:15.170 ⇒ 01:14:15.840 Uttam Kumaran: Yep.
859 01:14:17.220 ⇒ 01:14:18.860 Mustafa Raja: Drive repo…
860 01:14:26.840 ⇒ 01:14:28.610 Mustafa Raja: Hmm, yeah, this one.
861 01:14:29.680 ⇒ 01:14:36.130 Uttam Kumaran: So, like, how else, like, how else can we find the… the trace, like… Can you use.
862 01:14:36.130 ⇒ 01:14:39.840 Mustafa Raja: Yeah, we’ll have to… we’ll have to go… we’ll have to look at the time.
863 01:14:40.170 ⇒ 01:14:43.420 Mustafa Raja: They submitted the, review.
864 01:14:43.980 ⇒ 01:14:47.939 Mustafa Raja: So, yeah, this is a… this would be our only queue.
865 01:14:48.350 ⇒ 01:14:52.439 Mustafa Raja: To getting, to the… Do you use a conversation ID or anything?
866 01:14:53.450 ⇒ 01:15:01.830 Mustafa Raja: No, because this ID is actually, generated by, I think, the Gchat?
867 01:15:02.120 ⇒ 01:15:10.930 Mustafa Raja: And, I don’t think we’re storing execution ID, that might have helped. Let me actually check in…
868 01:15:11.500 ⇒ 01:15:12.420 Mustafa Raja: Yo.
869 01:15:20.780 ⇒ 01:15:22.050 Mustafa Raja: condition.
870 01:15:23.410 ⇒ 01:15:28.530 Mustafa Raja: Yeah, we are not storing execution ID. If we were, we would…
871 01:15:29.130 ⇒ 01:15:31.309 Mustafa Raja: We would have, been able to.
872 01:15:33.390 ⇒ 01:15:37.350 Samuel Roberts: Alright, I’m figuring out the timestamps and… Hold on.
873 01:15:37.670 ⇒ 01:15:39.259 Samuel Roberts: Making some progress.
874 01:15:39.860 ⇒ 01:15:42.330 Samuel Roberts: Is there one specific you wanted first? What was the one?
875 01:15:42.840 ⇒ 01:15:44.389 Uttam Kumaran: The one for Beau?
876 01:15:45.160 ⇒ 01:15:45.930 Mustafa Raja: Yeah, I’ve gone to.
877 01:15:46.700 ⇒ 01:15:47.550 Samuel Roberts: Oh, okay.
878 01:15:55.250 ⇒ 01:15:56.299 Samuel Roberts: Oh, yeah.
879 01:16:10.030 ⇒ 01:16:10.560 Mustafa Raja: Okay.
880 01:16:10.560 ⇒ 01:16:16.590 Uttam Kumaran: Can we, can we modify Mustafa to add the, execution idea?
881 01:16:16.590 ⇒ 01:16:18.000 Mustafa Raja: Execution? Yeah.
882 01:16:18.620 ⇒ 01:16:22.779 Mustafa Raja: Yeah, I’ll also add that to the logs, and then to the triages also.
883 01:16:23.090 ⇒ 01:16:23.780 Samuel Roberts: Yeah.
884 01:16:24.300 ⇒ 01:16:24.730 Mustafa Raja: It is…
885 01:16:24.730 ⇒ 01:16:25.520 Uttam Kumaran: Beautiful.
886 01:16:25.520 ⇒ 01:16:26.560 Mustafa Raja: This annoys a lot.
887 01:16:32.750 ⇒ 01:16:41.500 Uttam Kumaran: Do we always… do we also have a sense of what our, like… P80, P90, P99, like… .
888 01:16:42.250 ⇒ 01:16:44.049 Mustafa Raja: Yeah, so P80 would be…
889 01:16:44.050 ⇒ 01:16:44.670 Uttam Kumaran: the diamonds.
890 01:16:44.670 ⇒ 01:16:46.460 Mustafa Raja: 80% of the… yeah.
891 01:16:46.660 ⇒ 01:16:51.440 Mustafa Raja: 80% of requests, how much longer they’re taking to.
892 01:16:51.440 ⇒ 01:16:56.439 Uttam Kumaran: Like, this is way… 15 seconds and all it’s, like, way too long.
893 01:16:57.600 ⇒ 01:17:00.079 Uttam Kumaran: So, I guess, can we look at, like.
894 01:17:00.500 ⇒ 01:17:04.700 Uttam Kumaran: the P… yeah, it would be great. This is another, just, like.
895 01:17:05.100 ⇒ 01:17:10.440 Uttam Kumaran: the way I’m gonna sort of organize this is, like, we need… we should come out of this with, like, a better system.
896 01:17:10.560 ⇒ 01:17:19.019 Uttam Kumaran: So, one quick wins is, like, adding the execution ID. Second is, like, I want to do, like, a P80, P95…
897 01:17:19.320 ⇒ 01:17:21.960 Uttam Kumaran: And, like, be 99, like…
898 01:17:22.250 ⇒ 01:17:24.519 Samuel Roberts: That we can probably do from the Snowflake pretty well.
899 01:17:24.520 ⇒ 01:17:26.380 Uttam Kumaran: Yeah, execution time?
900 01:17:26.380 ⇒ 01:17:27.029 Samuel Roberts: Yeah, that’s awesome.
901 01:17:27.030 ⇒ 01:17:30.959 Uttam Kumaran: And based on that, I want to try to, like, attack that, too.
902 01:17:31.160 ⇒ 01:17:31.670 Samuel Roberts: Yeah.
903 01:17:31.670 ⇒ 01:17:35.250 Uttam Kumaran: To find out, like, what is… what is, like, taking up…
904 01:17:37.220 ⇒ 01:17:40.319 Uttam Kumaran: I don’t know what is taking the most time.
905 01:17:43.250 ⇒ 01:17:46.000 Uttam Kumaran: Okay.
906 01:17:54.960 ⇒ 01:18:00.889 Mustafa Raja: Yeah, I think we might have to, if we want to really, sort this out.
907 01:18:01.060 ⇒ 01:18:02.770 Mustafa Raja: I’ll take a look at this, because…
908 01:18:03.030 ⇒ 01:18:11.249 Mustafa Raja: We, this responded in 11 seconds and 9 seconds, or at least 10 seconds of it was just the TB tool.
909 01:18:12.580 ⇒ 01:18:13.350 Uttam Kumaran: Okay.
910 01:18:14.570 ⇒ 01:18:15.949 Samuel Roberts: Sorry, what are you, Sean?
911 01:18:17.470 ⇒ 01:18:18.940 Samuel Roberts: 10 seconds of the query.
912 01:18:19.240 ⇒ 01:18:20.279 Samuel Roberts: The QueryDB tool.
913 01:18:20.280 ⇒ 01:18:24.769 Mustafa Raja: Yeah, 10 seconds… so, here, about 10 seconds were the query.
914 01:18:24.770 ⇒ 01:18:25.440 Samuel Roberts: Oh, wow, yeah.
915 01:18:25.440 ⇒ 01:18:29.090 Mustafa Raja: And the whole agent took about, 11 seconds.
916 01:18:29.090 ⇒ 01:18:30.530 Samuel Roberts: Go back to the query tool real quick.
917 01:18:34.510 ⇒ 01:18:37.350 Samuel Roberts: Mosquito Mustang. Do we have,
918 01:18:38.160 ⇒ 01:18:43.399 Samuel Roberts: So we don’t know what time that request was specifically made to the query tool, do we? Is it gonna be around then?
919 01:18:43.400 ⇒ 01:18:43.840 Mustafa Raja: Interesting.
920 01:18:44.290 ⇒ 01:18:46.139 Samuel Roberts: Let me see if I can find that in the window.
921 01:18:47.300 ⇒ 01:18:51.370 Mustafa Raja: So this time would be, I think this is UTC time?
922 01:18:51.660 ⇒ 01:18:52.610 Samuel Roberts: Yeah, yeah.
923 01:18:53.080 ⇒ 01:18:54.010 Mustafa Raja: Yeah.
924 01:18:54.010 ⇒ 01:18:55.120 Uttam Kumaran: I can, I…
925 01:18:55.120 ⇒ 01:18:56.150 Samuel Roberts: I got it, hold on, let me…
926 01:18:56.150 ⇒ 01:19:00.330 Uttam Kumaran: execution link… Doesn’t go to the actual…
927 01:19:01.740 ⇒ 01:19:04.949 Uttam Kumaran: At least for me, it’s not going to the actual, like, execution.
928 01:19:07.630 ⇒ 01:19:09.570 Mustafa Raja: Let me, let me, let me, let me check that again.
929 01:19:09.570 ⇒ 01:19:10.580 Uttam Kumaran: It’s just taking…
930 01:19:10.580 ⇒ 01:19:11.309 Mustafa Raja: me there.
931 01:19:11.310 ⇒ 01:19:11.890 Uttam Kumaran: Maybe you shouldn’.
932 01:19:11.890 ⇒ 01:19:15.510 Samuel Roberts: Me there, too. Are you logged in with the workflow and everything open, or…
933 01:19:15.940 ⇒ 01:19:17.679 Uttam Kumaran: Wait, open on your side?
934 01:19:19.920 ⇒ 01:19:21.350 Samuel Roberts: Yeah, must have a click on them.
935 01:19:21.940 ⇒ 01:19:24.119 Samuel Roberts: Click the first one yet, I don’t think you’ve opened that one yet.
936 01:19:25.370 ⇒ 01:19:26.420 Samuel Roberts: my work studio.
937 01:19:27.460 ⇒ 01:19:28.950 Mustafa Raja: Yeah, it took me to the execution.
938 01:19:28.950 ⇒ 01:19:31.449 Uttam Kumaran: Yeah, but it’s taking me to the latest execution.
939 01:19:32.050 ⇒ 01:19:35.309 Mustafa Raja: No, no, no, so it’s always going to be on the top, yeah.
940 01:19:35.310 ⇒ 01:19:37.510 Samuel Roberts: Yeah, yeah, when you, when you… to it.
941 01:19:38.070 ⇒ 01:19:45.329 Mustafa Raja: Obviously, I’m on December 12th, but the latest one is December 16th, right? So, whatever we select comes to the top.
942 01:19:45.330 ⇒ 01:19:49.140 Uttam Kumaran: Yeah, when you link to it. Okay, great, sorry, it just looked like it was in order.
943 01:19:49.140 ⇒ 01:19:50.530 Samuel Roberts: No, no, you’re good, it’s another…
944 01:19:50.530 ⇒ 01:19:55.649 Mustafa Raja: When they did that, implemented that, I got confused also.
945 01:19:55.650 ⇒ 01:19:56.030 Samuel Roberts: hour.
946 01:19:57.230 ⇒ 01:19:59.009 Mustafa Raja: Yeah, it’s a little confusing.
947 01:20:01.330 ⇒ 01:20:07.199 Mustafa Raja: So, with that, we want the ideal output, right? Expected output.
948 01:20:12.610 ⇒ 01:20:17.440 Mustafa Raja: Well, you did, additionally.
949 01:20:24.930 ⇒ 01:20:28.310 Uttam Kumaran: Mosquito Misting System Estimate in Corpus.
950 01:20:29.230 ⇒ 01:20:36.550 Uttam Kumaran: So then I looked for the service name, like Mosquito Misting, So, can we open the…
951 01:20:36.910 ⇒ 01:20:39.820 Uttam Kumaran: query, DB, tool, and windmill.
952 01:20:40.270 ⇒ 01:20:40.620 Samuel Roberts: Yeah.
953 01:20:40.620 ⇒ 01:20:41.440 Uttam Kumaran: You go through, like…
954 01:20:41.440 ⇒ 01:20:45.780 Samuel Roberts: I was looking… I was trying to find that one right now and see what was… Specifically.
955 01:20:45.780 ⇒ 01:20:47.050 Mustafa Raja: Yeah, this is you.
956 01:20:47.830 ⇒ 01:20:52.650 Samuel Roberts: Mixes… 3…
957 01:20:53.730 ⇒ 01:20:56.400 Mustafa Raja: Yeah, it’s this, this video.
958 01:20:56.400 ⇒ 01:20:57.030 Samuel Roberts: Right.
959 01:20:58.330 ⇒ 01:20:58.990 Samuel Roberts: Can you…
960 01:20:58.990 ⇒ 01:21:01.989 Mustafa Raja: Let me… Let me answer…
961 01:21:02.410 ⇒ 01:21:03.070 Samuel Roberts: I’m the thing.
962 01:21:03.070 ⇒ 01:21:04.560 Uttam Kumaran: Go to the Python.
963 01:21:04.560 ⇒ 01:21:06.079 Samuel Roberts: Yeah, I wanted to send it to him.
964 01:21:06.520 ⇒ 01:21:07.160 Mustafa Raja: Okay.
965 01:21:07.570 ⇒ 01:21:10.279 Mustafa Raja: Let me actually… I’m just also linking the…
966 01:21:10.640 ⇒ 01:21:11.310 Samuel Roberts: Perfect.
967 01:21:11.310 ⇒ 01:21:12.760 Mustafa Raja: Windmill execution.
968 01:21:12.880 ⇒ 01:21:15.329 Mustafa Raja: And then… let me go to the Python.
969 01:21:16.290 ⇒ 01:21:20.750 Samuel Roberts: I hope that link is… That link is just to the top level.
970 01:21:22.740 ⇒ 01:21:25.850 Samuel Roberts: I think you gotta open it in the bread page, but yeah.
971 01:21:26.220 ⇒ 01:21:27.849 Mustafa Raja: Wait, let me get that.
972 01:21:29.830 ⇒ 01:21:34.769 Uttam Kumaran: And the, even the system prompt for this isn’t, like, too crazy, so…
973 01:21:35.030 ⇒ 01:21:40.309 Samuel Roberts: No, I’m not sure why this one… it might just be the… I think we talked about this, didn’t we, stuff with the cold startup time?
974 01:21:41.220 ⇒ 01:21:44.589 Mustafa Raja: Yes, we do have cold startup time in here, too.
975 01:21:44.590 ⇒ 01:21:46.410 Samuel Roberts: So that might be a chunk of this.
976 01:21:48.550 ⇒ 01:21:50.750 Mustafa Raja: It is…
977 01:21:56.550 ⇒ 01:21:57.450 Mustafa Raja: Why…
978 01:22:00.460 ⇒ 01:22:03.729 Mustafa Raja: Okay, yeah, there should be now a link to do it. Thank you.
979 01:22:07.680 ⇒ 01:22:14.200 Samuel Roberts: Mosquito missed estimate. Oh, interesting, okay. So it just totally ignored… So small.
980 01:22:16.340 ⇒ 01:22:20.710 Samuel Roberts: Misting… Hmm.
981 01:22:23.450 ⇒ 01:22:37.289 Mustafa Raja: I believe, this is an issue with how they are referring to their services. So, if we had a list on how people are going to query certain services, you know, apart from their name would be helpful, no?
982 01:22:40.490 ⇒ 01:22:45.169 Uttam Kumaran: Hmm… wait, hold on. Let me just, go… let me just find this one sec.
983 01:22:49.680 ⇒ 01:22:50.590 Samuel Roberts: Oh, excuse me.
984 01:22:51.310 ⇒ 01:22:55.380 Uttam Kumaran: Okay, so this is an ABC Home Commercial DBQuery Tool.
985 01:23:01.100 ⇒ 01:23:02.159 Samuel Roberts: And the input role.
986 01:23:05.270 ⇒ 01:23:05.990 Uttam Kumaran: Oh, God.
987 01:23:07.460 ⇒ 01:23:08.110 Mustafa Raja: So…
988 01:23:19.440 ⇒ 01:23:20.120 Samuel Roberts: Yeah.
989 01:23:20.120 ⇒ 01:23:32.840 Uttam Kumaran: Here’s another bigger system prompt. Given an input question, create a syntactically correct query to run. Once the user specifies the question. The number of examples, always limit to the top K.
990 01:23:33.600 ⇒ 01:23:35.669 Samuel Roberts: What are you… what are you reading right there?
991 01:23:36.810 ⇒ 01:23:40.390 Uttam Kumaran: I don’t… yeah, this prompt that… that’s Mustafa showing.
992 01:23:40.390 ⇒ 01:23:41.610 Samuel Roberts: Oh, okay, okay.
993 01:23:46.040 ⇒ 01:23:49.980 Uttam Kumaran: Okay, so let’s… let me start with my first question.
994 01:23:50.690 ⇒ 01:23:52.629 Uttam Kumaran: 1st of May.
995 01:23:52.800 ⇒ 01:23:55.280 Uttam Kumaran: Can we move this out of here?
996 01:23:55.700 ⇒ 01:23:59.700 Uttam Kumaran: to something that triggers something in laying fuse.
997 01:23:59.820 ⇒ 01:24:02.520 Uttam Kumaran: that hits… the DV.
998 01:24:05.200 ⇒ 01:24:06.669 Samuel Roberts: You mean off windmill completely?
999 01:24:06.960 ⇒ 01:24:07.600 Uttam Kumaran: Yeah.
1000 01:24:11.180 ⇒ 01:24:14.199 Uttam Kumaran: This is… this is just hitting a super base, dude.
1001 01:24:16.980 ⇒ 01:24:19.469 Samuel Roberts: Yeah, it’s generating the SQL query and then hitting the.
1002 01:24:19.470 ⇒ 01:24:24.580 Uttam Kumaran: But I guess, like, why can’t… why does this need to, like… can we rewrite this?
1003 01:24:25.050 ⇒ 01:24:26.670 Uttam Kumaran: in Mastra.
1004 01:24:26.670 ⇒ 01:24:28.989 Samuel Roberts: Yeah. That’s, I mean, that’s… yeah.
1005 01:24:28.990 ⇒ 01:24:32.159 Uttam Kumaran: Can we do that? Can we do that tonight?
1006 01:24:33.040 ⇒ 01:24:38.999 Uttam Kumaran: So we can debug this, because otherwise, how are we going to debug this DB tool? It’s so hard.
1007 01:24:39.190 ⇒ 01:24:39.770 Samuel Roberts: Yeah.
1008 01:24:40.480 ⇒ 01:24:43.649 Uttam Kumaran: Right? Like, how can I debug this? It’s impossible.
1009 01:24:44.670 ⇒ 01:24:48.710 Uttam Kumaran: Because what I can do, guys, is, like, I can go fuck with the props.
1010 01:24:49.420 ⇒ 01:24:54.060 Uttam Kumaran: But you have to give me it in a position for me to go test.
1011 01:24:54.060 ⇒ 01:24:54.630 Samuel Roberts: Yeah.
1012 01:24:54.630 ⇒ 01:24:56.010 Uttam Kumaran: That’s with new inputs.
1013 01:24:56.180 ⇒ 01:24:58.019 Uttam Kumaran: I can’t do that here at all.
1014 01:24:58.380 ⇒ 01:25:04.470 Samuel Roberts: Okay, I think what we could do… so, Mustafa, we’ve got the current, master code.
1015 01:25:06.170 ⇒ 01:25:07.369 Uttam Kumaran: Like, even, yeah, just hope.
1016 01:25:07.370 ⇒ 01:25:07.930 Samuel Roberts: Can we…
1017 01:25:07.930 ⇒ 01:25:09.680 Uttam Kumaran: A piecemeal you need is something.
1018 01:25:09.680 ⇒ 01:25:13.020 Samuel Roberts: Yeah, yeah, I was gonna say, Mustafa, can you take this…
1019 01:25:13.480 ⇒ 01:25:18.850 Samuel Roberts: into Mastra, and added to that current, like, Andy Mostra setup?
1020 01:25:19.640 ⇒ 01:25:25.339 Mustafa Raja: Yeah, and if… if we… if we want it hosted, I believe we could even…
1021 01:25:25.570 ⇒ 01:25:29.550 Mustafa Raja: Put it in temporarily in platform, if you want that.
1022 01:25:32.800 ⇒ 01:25:35.760 Samuel Roberts: I mean, if we’re just looking at debug, I don’t know if that’s as important. It could be pretty.
1023 01:25:35.760 ⇒ 01:25:42.640 Mustafa Raja: We could easily, easily run it locally. I’ll just… I’ll just… I’ll just… Turn this into… into my.
1024 01:25:42.640 ⇒ 01:25:48.520 Samuel Roberts: But that’s one of the things I was gonna… yeah, this whole tool is gonna get moved into Moscow as well, because we’ll already have code execution, so…
1025 01:25:48.520 ⇒ 01:25:49.240 Mustafa Raja: Yeah.
1026 01:25:49.540 ⇒ 01:25:50.590 Samuel Roberts: Which should, one.
1027 01:25:50.590 ⇒ 01:25:57.050 Uttam Kumaran: Like, what is… if we recreate this in Moscow, what is the odds that we turn this off, like, today?
1028 01:25:57.600 ⇒ 01:26:03.700 Mustafa Raja: And we… and we move that there, like… We’ll have to host that, too, then.
1029 01:26:04.390 ⇒ 01:26:07.749 Uttam Kumaran: Bo, why not? Can we hope… do we… so we haven’t…
1030 01:26:07.910 ⇒ 01:26:10.240 Uttam Kumaran: Can we just host this on our…
1031 01:26:11.350 ⇒ 01:26:15.870 Uttam Kumaran: Heroku, or should we host it on our away, or like, right now.
1032 01:26:15.870 ⇒ 01:26:16.300 Samuel Roberts: Yeah.
1033 01:26:16.300 ⇒ 01:26:19.990 Uttam Kumaran: On our… on a window, so, like, anything is sort of better than this.
1034 01:26:19.990 ⇒ 01:26:20.520 Samuel Roberts: Yeah.
1035 01:26:20.520 ⇒ 01:26:23.199 Mustafa Raja: Yeah. Yeah, Heroku would work, I believe.
1036 01:26:23.200 ⇒ 01:26:23.710 Samuel Roberts: We could spend…
1037 01:26:23.710 ⇒ 01:26:24.050 Mustafa Raja: You say…
1038 01:26:24.050 ⇒ 01:26:24.870 Samuel Roberts: pretty easily.
1039 01:26:25.320 ⇒ 01:26:25.910 Mustafa Raja: Yeah.
1040 01:26:27.510 ⇒ 01:26:29.670 Mustafa Raja: So that is going to be an express app, right?
1041 01:26:30.830 ⇒ 01:26:34.310 Uttam Kumaran: That’s fine, I would suggest, yeah, yeah, I’m…
1042 01:26:34.840 ⇒ 01:26:37.280 Samuel Roberts: Monstra has a way to host it, I think, I’m not… I’m not…
1043 01:26:37.280 ⇒ 01:26:38.500 Uttam Kumaran: Monster Cloud?
1044 01:26:38.720 ⇒ 01:26:42.390 Samuel Roberts: No, no, no, I mean, like, I’m not sure if we need an express server, or if Master.
1045 01:26:42.390 ⇒ 01:26:43.030 Mustafa Raja: Oh, yeah.
1046 01:26:43.030 ⇒ 01:26:44.330 Samuel Roberts: HTTP built in.
1047 01:26:44.510 ⇒ 01:26:50.239 Mustafa Raja: Oh, yeah, yeah, you’re right, you’re right. We don’t need to… yeah, it gives the… the UI comes out of the box.
1048 01:26:50.240 ⇒ 01:26:53.300 Samuel Roberts: Yeah, so I think, I think you should be able to…
1049 01:26:53.300 ⇒ 01:26:53.759 Mustafa Raja: Is it that way?
1050 01:26:53.760 ⇒ 01:26:57.940 Samuel Roberts: you’re using it within Next.js, but with the master… yeah, you should be able to get that.
1051 01:26:58.600 ⇒ 01:26:59.780 Mustafa Raja: Yeah, I understand that.
1052 01:26:59.780 ⇒ 01:27:01.030 Samuel Roberts: And expose those.
1053 01:27:01.030 ⇒ 01:27:04.629 Mustafa Raja: Okay, I’ll do a new app, a new app in Heroku then, right?
1054 01:27:07.100 ⇒ 01:27:09.400 Samuel Roberts: Yeah, we’ll set up a new Heroku.
1055 01:27:16.830 ⇒ 01:27:18.420 Mustafa Raja: Let me get the topic.
1056 01:27:23.380 ⇒ 01:27:27.120 Samuel Roberts: I’m gonna have to go at some point, though, because I gotta switch into baby take care of moves.
1057 01:27:27.120 ⇒ 01:27:44.169 Uttam Kumaran: Okay. Yeah, I think if we can get this ho… like, I mean, I’ll… I’ll just be… I’m gonna stay up, but I guess, like, if we can get this hosted somewhere, Mustafa, where I can… Yeah, I’m going to host it, at least, like, start to test inputs and outputs.
1058 01:27:44.170 ⇒ 01:27:48.059 Samuel Roberts: Well, that’s… you could test that locally, that’s… that’s what we’ve been doing with the Mastra so far.
1059 01:27:48.240 ⇒ 01:27:54.750 Samuel Roberts: Which would let you iterate. So I… if we wanted a report.
1060 01:27:54.750 ⇒ 01:27:58.949 Uttam Kumaran: But I guess, like, walk me through that. So, like, for example, for me, I would basically…
1061 01:28:01.230 ⇒ 01:28:06.699 Uttam Kumaran: I’m going to just continue to test The inputs, outputs until, like.
1062 01:28:06.900 ⇒ 01:28:12.589 Uttam Kumaran: We basically nail down the fact that if it doesn’t know, it kicks it back.
1063 01:28:12.740 ⇒ 01:28:21.210 Uttam Kumaran: And if it knows it goes through, and for the one… and then the ones that it knows, it needs to do the right thing. Is that kind of like… that’s the…
1064 01:28:21.440 ⇒ 01:28:22.090 Uttam Kumaran: order of operations.
1065 01:28:22.090 ⇒ 01:28:27.270 Samuel Roberts: Okay, so… Yes, the…
1066 01:28:30.580 ⇒ 01:28:34.979 Samuel Roberts: for testing this, this, you want to test this, specifically the DBQuery tool?
1067 01:28:35.770 ⇒ 01:28:37.459 Uttam Kumaran: Yeah, cause…
1068 01:28:37.460 ⇒ 01:28:38.010 Samuel Roberts: Okay.
1069 01:28:39.030 ⇒ 01:28:42.720 Uttam Kumaran: I mean, I feel like this is, like, the main issue.
1070 01:28:43.330 ⇒ 01:28:51.840 Samuel Roberts: Yeah, I think, I mean, it’s a fair chunk of them, certainly. And the speed is also, that’s then, yeah. But what I’m saying is, like, you can test this locally, so if…
1071 01:28:52.270 ⇒ 01:28:56.770 Samuel Roberts: Do you have the Mastra Andy, open, Mustafa? Or is that…
1072 01:28:59.900 ⇒ 01:29:02.219 Samuel Roberts: So, the nice thing about the master dev
1073 01:29:02.640 ⇒ 01:29:06.060 Samuel Roberts: for when you’re doing it just locally, is it gives you this UI,
1074 01:29:06.210 ⇒ 01:29:09.140 Samuel Roberts: That you can test all the tools and agents and workflows.
1075 01:29:10.410 ⇒ 01:29:16.449 Samuel Roberts: So if we move that SQL the text SQL, Into here.
1076 01:29:16.450 ⇒ 01:29:19.289 Mustafa Raja: We live here, and then we’re just going to be able to talk to Tool directly.
1077 01:29:19.290 ⇒ 01:29:23.250 Samuel Roberts: And you can… you can pass things directly to that, you can also set it up so that the…
1078 01:29:23.360 ⇒ 01:29:28.029 Samuel Roberts: You know, you could see the outputs from one into the inputs of the other.
1079 01:29:28.990 ⇒ 01:29:33.800 Samuel Roberts: Without worrying about getting it up on Heroku right now, at least to dial in the prompt and, you know, whatever.
1080 01:29:34.460 ⇒ 01:29:39.220 Mustafa Raja: So we could trigger the tool directly, rather than having an agent go through it?
1081 01:29:40.210 ⇒ 01:29:42.429 Samuel Roberts: Well, you can do… you can test it both ways, but yes.
1082 01:29:42.430 ⇒ 01:29:45.149 Mustafa Raja: Yeah, yeah. I think we want to test it both ways for that purpose, but…
1083 01:29:48.040 ⇒ 01:29:50.799 Uttam Kumaran: I guess I haven’t brought this up locally, so…
1084 01:29:50.800 ⇒ 01:29:53.000 Samuel Roberts: Yeah, I think that would be…
1085 01:29:53.100 ⇒ 01:29:57.019 Samuel Roberts: more… because, like, getting it up on… I mean, you can get it up on Heroku and just test the…
1086 01:29:58.080 ⇒ 01:30:03.130 Uttam Kumaran: But then I guess, like, I would have to basically nail the prompt, and then put it back into the Python.
1087 01:30:06.860 ⇒ 01:30:07.680 Samuel Roberts: Right.
1088 01:30:08.810 ⇒ 01:30:12.449 Samuel Roberts: Yeah, I mean, if we want to replace it, we should just replace it, like, anyway, but…
1089 01:30:12.990 ⇒ 01:30:13.680 Mustafa Raja: I could just…
1090 01:30:13.680 ⇒ 01:30:15.130 Uttam Kumaran: place it. What, what.
1091 01:30:15.130 ⇒ 01:30:16.230 Mustafa Raja: Yeah, I could just source it.
1092 01:30:16.230 ⇒ 01:30:16.840 Samuel Roberts: Okay.
1093 01:30:17.790 ⇒ 01:30:24.189 Uttam Kumaran: So, you’re saying that I can just run the…
1094 01:30:31.030 ⇒ 01:30:38.000 Samuel Roberts: the… so we… for the migration, we were working on the monster code, so we… in the Andy… or in the ABC…
1095 01:30:38.660 ⇒ 01:30:45.579 Samuel Roberts: Abc Home & Commercials is an ABC Monster app, that’s what Casey had been working on previously, that Mustafa was adding a little bit to.
1096 01:30:47.230 ⇒ 01:31:00.590 Samuel Roberts: And then… but what we can add there now, if we want… like, this is still where we were just testing and making sure that things were responding well. Like, I think we tried, GPT 5.2, and it was… it was responding wildly differently, so we backed that off. The…
1097 01:31:01.490 ⇒ 01:31:03.540 Samuel Roberts: What we can do here is…
1098 01:31:04.390 ⇒ 01:31:08.819 Mustafa Raja: remove this, or replace this with whatever logic we have in windmill.
1099 01:31:09.660 ⇒ 01:31:13.900 Samuel Roberts: Exactly, exactly. The same thing, yeah. Yeah, and so then…
1100 01:31:14.600 ⇒ 01:31:18.690 Uttam Kumaran: But would you be able to call this piece from N8N?
1101 01:31:18.850 ⇒ 01:31:27.959 Samuel Roberts: Well, that’s one thing. I think for now, for testing locally, we could do that. What we could also then do is get this stood up on Heroku, expose that tool… Yeah.
1102 01:31:28.090 ⇒ 01:31:31.250 Samuel Roberts: And then we could pass that back to N8N, and so then.
1103 01:31:31.250 ⇒ 01:31:33.260 Uttam Kumaran: That’s what I would like. Yeah.
1104 01:31:33.810 ⇒ 01:31:38.219 Uttam Kumaran: is option 2. What I can do in the meantime…
1105 01:31:38.340 ⇒ 01:31:46.029 Uttam Kumaran: is try… for me to test locally is, like, I’ll just run that Python Like, on my machine.
1106 01:31:46.540 ⇒ 01:31:49.389 Samuel Roberts: Yeah, yeah, you shouldn’t have the cold startup plan, that should be better, so…
1107 01:31:49.740 ⇒ 01:31:53.280 Uttam Kumaran: Yeah, and so I’ll run and try to just get the prompt to work.
1108 01:31:54.090 ⇒ 01:31:54.680 Samuel Roberts: Okay.
1109 01:31:54.940 ⇒ 01:32:00.570 Uttam Kumaran: And my… Go through and try to fix… Yeah.
1110 01:32:01.770 ⇒ 01:32:03.460 Uttam Kumaran: Go through and just try to fix it.
1111 01:32:11.590 ⇒ 01:32:14.740 Samuel Roberts: Okay.
1112 01:32:15.220 ⇒ 01:32:29.350 Samuel Roberts: So I gotta, like I said, I gotta run, because my wife’s gotta get to work. I think, yeah, so get… so if you get this tool into here, that’ll be one, like, a good step forward anyway, and then get that stood up on Heroku with the right, endpoint.
1113 01:32:31.030 ⇒ 01:32:33.200 Uttam Kumaran: Is there any way we can…
1114 01:32:33.430 ⇒ 01:32:35.559 Uttam Kumaran: Make the switch tonight, or no?
1115 01:32:36.170 ⇒ 01:32:37.209 Samuel Roberts: The switch tonight for the…
1116 01:32:37.210 ⇒ 01:32:37.890 Mustafa Raja: Yeah, yeah.
1117 01:32:37.890 ⇒ 01:32:38.270 Samuel Roberts: betrayal.
1118 01:32:38.310 ⇒ 01:32:41.399 Mustafa Raja: Switched, like, directly to.
1119 01:32:41.400 ⇒ 01:32:45.229 Uttam Kumaran: Production query, DB toolk, is there any way we can actually
1120 01:32:45.500 ⇒ 01:32:49.229 Uttam Kumaran: I think so. Working and then switch it, like, or is that too risky?
1121 01:32:49.630 ⇒ 01:32:52.360 Samuel Roberts: I mean, it’s pretty risky, but, like.
1122 01:32:52.360 ⇒ 01:32:54.250 Mustafa Raja: Yeah, what we can do is…
1123 01:32:54.250 ⇒ 01:32:56.549 Samuel Roberts: tested a bunch against maybe what we already had.
1124 01:32:58.030 ⇒ 01:32:59.689 Mustafa Raja: Yeah, I could get it, I could get it…
1125 01:32:59.690 ⇒ 01:33:00.579 Samuel Roberts: If we feel comfortable.
1126 01:33:00.760 ⇒ 01:33:04.400 Mustafa Raja: Yeah, I could get it, up and running today, and.
1127 01:33:04.400 ⇒ 01:33:13.290 Uttam Kumaran: Well, if you can get up and running, Mustafa, and then just give me a way to test it, because I really care about an environment where I can redo the prompts.
1128 01:33:14.450 ⇒ 01:33:22.169 Uttam Kumaran: And so, ultimately, I’ll find the prompt that achieves what we need, and I will just swap that in, right? Basically.
1129 01:33:26.840 ⇒ 01:33:29.589 Uttam Kumaran: That’s… that’s basically right, right, guys? Like…
1130 01:33:29.870 ⇒ 01:33:31.260 Samuel Roberts: Yeah.
1131 01:33:31.620 ⇒ 01:33:34.779 Uttam Kumaran: Because what other variables do we have to work with, like… I mean.
1132 01:33:34.780 ⇒ 01:33:36.089 Mustafa Raja: The delivery structure.
1133 01:33:36.430 ⇒ 01:33:37.240 Uttam Kumaran: Huh?
1134 01:33:37.650 ⇒ 01:33:39.710 Mustafa Raja: The structure of the database itself, no?
1135 01:33:40.300 ⇒ 01:33:42.510 Uttam Kumaran: Not really, what can we do there?
1136 01:33:43.250 ⇒ 01:33:46.599 Samuel Roberts: There’s some weird stuff with different services being nested into others, I think.
1137 01:33:47.020 ⇒ 01:33:55.289 Mustafa Raja: So, I was thinking maybe… maybe tags, you know, this service, and then some tags missing, stuff like that.
1138 01:33:55.530 ⇒ 01:33:56.569 Mustafa Raja: If we want to do.
1139 01:33:56.570 ⇒ 01:33:58.559 Samuel Roberts: Yeah, I mean, that’s… I think it’s a little…
1140 01:33:59.620 ⇒ 01:34:13.099 Samuel Roberts: band-aid-y, but I’m worried about, like, the fact that mosquito system and mosquito misting and mosquito suppression are all somewhat related in a way I’m not necessarily sure I understand. Like, mosquito system, I’m assuming, is, like, the
1141 01:34:13.500 ⇒ 01:34:15.660 Samuel Roberts: top level, I guess? Because it’s all… I don’t know.
1142 01:34:15.660 ⇒ 01:34:16.030 Mustafa Raja: Yeah.
1143 01:34:16.030 ⇒ 01:34:23.099 Uttam Kumaran: So right now, there’s no… there is no mosquito system in here. There’s no mosquito, misting in here that we’re seeing.
1144 01:34:23.320 ⇒ 01:34:23.910 Samuel Roberts: Right, because that’s.
1145 01:34:23.910 ⇒ 01:34:27.020 Mustafa Raja: So if we… If you do things… Get rid of inspection.
1146 01:34:27.020 ⇒ 01:34:28.020 Samuel Roberts: actors.
1147 01:34:29.850 ⇒ 01:34:30.270 Mustafa Raja: Oh, yeah.
1148 01:34:30.270 ⇒ 01:34:35.670 Samuel Roberts: That was the issue that we had before, was it added misting when it really… it was wondering about inspectors, and inspectors are only…
1149 01:34:36.520 ⇒ 01:34:37.930 Samuel Roberts: mosquito system.
1150 01:34:38.260 ⇒ 01:34:41.000 Samuel Roberts: If you keep scrolling, you’ll see suppression, you’ll see misting as well.
1151 01:34:41.130 ⇒ 01:34:44.860 Samuel Roberts: But some are technicians, some are inspectors. The inspectors are only…
1152 01:34:46.690 ⇒ 01:34:47.300 Uttam Kumaran: But then…
1153 01:34:47.300 ⇒ 01:34:47.850 Mustafa Raja: limiting to…
1154 01:34:47.850 ⇒ 01:34:51.310 Uttam Kumaran: What I was questioning was… Tell me…
1155 01:34:51.520 ⇒ 01:34:58.300 Uttam Kumaran: Mosquito Misting System Estimate. So for system estimate, you need tax or inspectors.
1156 01:34:58.740 ⇒ 01:35:03.720 Samuel Roberts: Well, we need the inspectors, but the problem was that name did not include misting.
1157 01:35:03.940 ⇒ 01:35:07.219 Samuel Roberts: And that’s what this prompt chose to include.
1158 01:35:07.220 ⇒ 01:35:12.680 Uttam Kumaran: Wait, wait, wait. I guess that’s why I’m not… why didn’t the… misting…
1159 01:35:15.110 ⇒ 01:35:17.950 Uttam Kumaran: Well, is there, is there a…
1160 01:35:18.380 ⇒ 01:35:22.319 Uttam Kumaran: Yeah, I just want to see, like, is there… where is the misting here?
1161 01:35:22.320 ⇒ 01:35:25.509 Samuel Roberts: Scroll, yeah, scroll down… no, no, don’t scroll down.
1162 01:35:25.510 ⇒ 01:35:26.910 Mustafa Raja: Yeah, it’s limiting to…
1163 01:35:26.910 ⇒ 01:35:27.330 Uttam Kumaran: It’s limited.
1164 01:35:27.330 ⇒ 01:35:28.330 Mustafa Raja: hitting 200.
1165 01:35:28.640 ⇒ 01:35:29.520 Samuel Roberts: Oh, okay.
1166 01:35:29.880 ⇒ 01:35:30.720 Uttam Kumaran: Just type in miss.
1167 01:35:30.720 ⇒ 01:35:31.839 Mustafa Raja: I’m just going to do this, yeah.
1168 01:35:33.540 ⇒ 01:35:36.380 Samuel Roberts: Yeah, so there’s a service named Mosquito Misting Residential.
1169 01:35:37.130 ⇒ 01:35:43.220 Samuel Roberts: And if you notice, the only roles that… or the only role name that ever matches that is technician.
1170 01:35:43.740 ⇒ 01:35:44.969 Uttam Kumaran: Okay.
1171 01:35:44.970 ⇒ 01:35:53.200 Samuel Roberts: And so he was looking for an inspector, and so rather than do… and since he said, I think, mosquito misting system.
1172 01:35:54.320 ⇒ 01:35:54.990 Mustafa Raja: Yeah.
1173 01:35:55.180 ⇒ 01:35:58.370 Samuel Roberts: It’s not a service, technically, so it just searched sting.
1174 01:35:59.090 ⇒ 01:36:00.670 Mustafa Raja: Yeah, that is what I was saying.
1175 01:36:00.880 ⇒ 01:36:02.430 Mustafa Raja: That is why I was saying.
1176 01:36:02.430 ⇒ 01:36:09.550 Uttam Kumaran: Where, like, so if I build a prompt… so, let me get it, let me get it right. So in that situation, we should have asked him
1177 01:36:10.000 ⇒ 01:36:15.019 Uttam Kumaran: we should have said… we could have… we could have done one of many things. We could say, I’m sorry, there’s no inspectors.
1178 01:36:15.250 ⇒ 01:36:18.320 Uttam Kumaran: But… there is… like…
1179 01:36:18.320 ⇒ 01:36:19.050 Mustafa Raja: technician.
1180 01:36:19.220 ⇒ 01:36:20.470 Uttam Kumaran: technicians…
1181 01:36:21.250 ⇒ 01:36:23.709 Samuel Roberts: That would have been wrong, though, too.
1182 01:36:23.990 ⇒ 01:36:27.210 Uttam Kumaran: There… we can also say there is no mosquito mis…
1183 01:36:27.810 ⇒ 01:36:31.030 Uttam Kumaran: Like, mosquito misting system is vague.
1184 01:36:31.240 ⇒ 01:36:34.519 Uttam Kumaran: It’s either the mosquito system or mosquito misting.
1185 01:36:35.140 ⇒ 01:36:42.670 Samuel Roberts: Well, I don’t know if that’s vague, or if that’s just a, like, a thing that’s not captured in the database properly. Like, is that a normal way that they would say something?
1186 01:36:45.300 ⇒ 01:36:49.700 Uttam Kumaran: I mean, I think that’s just what, like, has to… we have to get that into the prompt, then.
1187 01:36:49.700 ⇒ 01:36:51.819 Samuel Roberts: Right, that’s why I’m not sure if it’s, like, a…
1188 01:36:52.010 ⇒ 01:36:55.079 Samuel Roberts: We need to verify what service it’s asking for.
1189 01:36:55.080 ⇒ 01:36:58.090 Uttam Kumaran: But this is… yeah, but that… I think we… I think we should.
1190 01:36:58.340 ⇒ 01:36:58.950 Samuel Roberts: Yeah.
1191 01:36:59.410 ⇒ 01:37:05.849 Uttam Kumaran: Like, I think we should say… he said mosquito misting system, it should kick back and say, like, did you mean…
1192 01:37:05.850 ⇒ 01:37:07.910 Samuel Roberts: Right, that’s what I’m saying, so we could do a first pass.
1193 01:37:07.910 ⇒ 01:37:08.480 Uttam Kumaran: Password.
1194 01:37:08.480 ⇒ 01:37:09.869 Samuel Roberts: Checks the service name.
1195 01:37:10.340 ⇒ 01:37:11.170 Uttam Kumaran: Yeah.
1196 01:37:11.170 ⇒ 01:37:12.730 Samuel Roberts: And then, maybe it even…
1197 01:37:13.000 ⇒ 01:37:16.710 Samuel Roberts: Verifies what it’s about to look up, which is one other step we could do.
1198 01:37:17.460 ⇒ 01:37:18.280 Samuel Roberts: like…
1199 01:37:18.490 ⇒ 01:37:21.229 Uttam Kumaran: But that seems extra, yeah, I don’t know.
1200 01:37:25.520 ⇒ 01:37:26.490 Uttam Kumaran: Yeah.
1201 01:37:31.280 ⇒ 01:37:37.859 Samuel Roberts: I mean, I think we could do a, like, a first pass where it looks at the systems, because, like, there’s a systems table, right? This is…
1202 01:37:38.460 ⇒ 01:37:45.079 Samuel Roberts: s.service name, yeah, so it’s coming from services. So there’s a way that it could verify against it.
1203 01:37:45.550 ⇒ 01:37:54.219 Samuel Roberts: Like, you asked for a service that we just don’t even have, or if it does, but this is a specific one where it has the service misting, it just doesn’t have it with the inspectors.
1204 01:37:55.630 ⇒ 01:37:58.490 Uttam Kumaran: Yeah. He didn’t necessarily ask for an inspector, he asked for…
1205 01:37:59.330 ⇒ 01:38:01.830 Samuel Roberts: What was the phrasing exactly?
1206 01:38:01.830 ⇒ 01:38:02.399 Uttam Kumaran: No, I see.
1207 01:38:02.400 ⇒ 01:38:02.840 Mustafa Raja: Yeah.
1208 01:38:02.840 ⇒ 01:38:07.780 Uttam Kumaran: I need to schedule a mosquito misting system estimate in Corpus.
1209 01:38:07.960 ⇒ 01:38:08.600 Samuel Roberts: Yeah.
1210 01:38:13.460 ⇒ 01:38:18.290 Mustafa Raja: So the inspector, the agent added itself then, right?
1211 01:38:20.780 ⇒ 01:38:23.310 Samuel Roberts: Right, because that’s who would do the estimate, like, it knows that, right?
1212 01:38:23.960 ⇒ 01:38:24.600 Uttam Kumaran: Yeah.
1213 01:38:24.600 ⇒ 01:38:25.670 Samuel Roberts: Prompt, I assume.
1214 01:38:26.470 ⇒ 01:38:27.190 Mustafa Raja: Yeah.
1215 01:38:27.650 ⇒ 01:38:33.600 Uttam Kumaran: That should be… I mean, alternatively, we can also have it give it the error information, like.
1216 01:38:34.910 ⇒ 01:38:38.839 Uttam Kumaran: I’m sorry, I couldn’t find… here’s what I… here’s what I’m looking for, here are.
1217 01:38:38.840 ⇒ 01:38:39.360 Samuel Roberts: Yeah.
1218 01:38:39.360 ⇒ 01:38:40.389 Uttam Kumaran: Yes, you could try.
1219 01:38:40.600 ⇒ 01:38:46.530 Samuel Roberts: That’s… that’s… yeah. Yeah, if we… if we have it fined… If it doesn’t find anything.
1220 01:38:47.580 ⇒ 01:38:53.729 Samuel Roberts: what are the… what are the reasons it could find nothing? It could find nothing because they straight up don’t service it, in which case there would be…
1221 01:38:54.050 ⇒ 01:38:56.660 Samuel Roberts: Because we did… service?
1222 01:38:57.210 ⇒ 01:39:01.409 Samuel Roberts: location, And… role, right?
1223 01:39:01.670 ⇒ 01:39:02.350 Uttam Kumaran: Yep.
1224 01:39:02.540 ⇒ 01:39:06.580 Samuel Roberts: And so, it could be, you know, based on, you know, it could… we could…
1225 01:39:07.250 ⇒ 01:39:11.559 Samuel Roberts: Reply with the fact that, like, we were not able to find any blank, blank, blanks.
1226 01:39:11.560 ⇒ 01:39:15.749 Uttam Kumaran: Well, I mean, it’s always a full name lookup, right? So…
1227 01:39:15.750 ⇒ 01:39:16.290 Samuel Roberts: Yeah.
1228 01:39:20.590 ⇒ 01:39:24.890 Uttam Kumaran: I mean, the alternative we could do is, like, if it’s unclear.
1229 01:39:25.350 ⇒ 01:39:27.979 Uttam Kumaran: It should pull up, and then be like.
1230 01:39:28.090 ⇒ 01:39:30.180 Uttam Kumaran: It just basically asks for clarity.
1231 01:39:30.370 ⇒ 01:39:32.920 Uttam Kumaran: So, basically, you’re right in that if…
1232 01:39:33.390 ⇒ 01:39:39.180 Uttam Kumaran: If we should give it the list of services, and then if it’s not able to identify, it should kick back and be like.
1233 01:39:39.740 ⇒ 01:39:41.460 Uttam Kumaran: What service is this?
1234 01:39:41.460 ⇒ 01:39:44.319 Samuel Roberts: Right, it should know what service it’s looking for.
1235 01:39:44.320 ⇒ 01:39:47.839 Uttam Kumaran: Not pick a misting… Yeah.
1236 01:39:47.840 ⇒ 01:39:50.230 Samuel Roberts: Versus service, or… yeah, okay.
1237 01:39:50.230 ⇒ 01:39:57.800 Uttam Kumaran: I think, Mustafa, if you can get me this in a way where I can, like, rapidly iterate on the prompt.
1238 01:39:59.440 ⇒ 01:40:01.619 Mustafa Raja: Yeah, I can do that. I had the problem.
1239 01:40:01.620 ⇒ 01:40:03.650 Uttam Kumaran: I’ll fix the problem tonight.
1240 01:40:04.650 ⇒ 01:40:11.620 Uttam Kumaran: And then, I will also… Give some suggestions on how do we layer on some more support here.
1241 01:40:11.910 ⇒ 01:40:13.410 Samuel Roberts: Yeah, I think…
1242 01:40:13.760 ⇒ 01:40:20.230 Samuel Roberts: And that’s something we can do with the monster, like, that’s what Mustafa and I were talking about, was part of the issue with some of this stuff, not related to the…
1243 01:40:20.460 ⇒ 01:40:26.510 Samuel Roberts: DB lookups, but like I said, there’s a ton in the context that really shouldn’t be part of the initial prompt.
1244 01:40:26.890 ⇒ 01:40:30.779 Samuel Roberts: But in NAM, the best way to do that was just to shove it in the prompt.
1245 01:40:30.930 ⇒ 01:40:39.360 Samuel Roberts: So we had talked about multiple agents that know, okay, they asked about cancellation, go to the cancellation flow agent, and it’ll respond with that.
1246 01:40:39.360 ⇒ 01:40:39.830 Uttam Kumaran: Yes.
1247 01:40:39.830 ⇒ 01:40:44.240 Samuel Roberts: I think we can do more, again, with this, too, where it does a first pass.
1248 01:40:44.370 ⇒ 01:40:54.089 Samuel Roberts: verifies the service name, kicks back if it doesn’t. Like, make sure it has the right information for making this query, or making this tool call, really.
1249 01:40:55.920 ⇒ 01:41:06.779 Mustafa Raja: I’m… I’m also wondering, like, for this, this service, Boo did not, you know, call the service exactly by its name, but by an alias.
1250 01:41:06.900 ⇒ 01:41:09.469 Mustafa Raja: If they… if they do this…
1251 01:41:09.470 ⇒ 01:41:10.089 Samuel Roberts: That’s the other thing.
1252 01:41:10.090 ⇒ 01:41:11.170 Mustafa Raja: the services.
1253 01:41:12.130 ⇒ 01:41:12.860 Samuel Roberts: Unless we could get.
1254 01:41:12.860 ⇒ 01:41:14.069 Mustafa Raja: The list of that.
1255 01:41:15.020 ⇒ 01:41:16.820 Mustafa Raja: If you could get a list of that…
1256 01:41:17.090 ⇒ 01:41:22.930 Mustafa Raja: You know, that might… that might really be, game-changing. Yeah, but we could even…
1257 01:41:23.780 ⇒ 01:41:24.550 Samuel Roberts: Whoa.
1258 01:41:26.170 ⇒ 01:41:29.360 Samuel Roberts: we might not need that, because, I mean, the LLM should be able to kind of
1259 01:41:30.210 ⇒ 01:41:35.100 Samuel Roberts: figure that out, but I think we didn’t provide it with all the services that… You know, it just…
1260 01:41:35.410 ⇒ 01:41:46.559 Mustafa Raja: Yeah, I guess even if L&M did zoom out a little, you know, after the first part, maybe zoom out on stuff, it might have found something interesting.
1261 01:41:46.560 ⇒ 01:41:53.099 Samuel Roberts: Right, right, so I feel like if it wasn’t… if it was only searching for… Validating the right service.
1262 01:41:53.750 ⇒ 01:41:57.939 Mustafa Raja: It probably would have gotten to a point when it would have said, like, are you looking for…
1263 01:41:58.300 ⇒ 01:42:00.500 Samuel Roberts: And then it would have said the system one.
1264 01:42:00.610 ⇒ 01:42:03.869 Samuel Roberts: And he might have said yes, and then it would have filled that improperly.
1265 01:42:04.720 ⇒ 01:42:05.860 Mustafa Raja: Potentially.
1266 01:42:05.880 ⇒ 01:42:17.330 Samuel Roberts: Oh, yeah, I think another step here is probably a smart thing, especially because, like, I imagine we’re crashing on service names. I don’t think we’ll really be having too many issues with location, because that’s pretty well organized.
1267 01:42:17.760 ⇒ 01:42:19.860 Samuel Roberts: And it, like I said, it clearly knew inspect.
1268 01:42:19.860 ⇒ 01:42:24.960 Mustafa Raja: Yeah, and they… yeah, and they don’t use any aliases for the locations also, so that… Exactly.
1269 01:42:24.960 ⇒ 01:42:26.349 Samuel Roberts: Yeah, and even…
1270 01:42:26.350 ⇒ 01:42:27.430 Mustafa Raja: Maybe he’s like photos.
1271 01:42:27.430 ⇒ 01:42:33.100 Samuel Roberts: Porpoise is, you know, the AI knows, can probably figure out, like, Yeah, okay.
1272 01:42:34.050 ⇒ 01:42:34.680 Uttam Kumaran: Okay.
1273 01:42:35.170 ⇒ 01:42:37.270 Samuel Roberts: Yeah, I think, I think…
1274 01:42:37.560 ⇒ 01:42:40.370 Samuel Roberts: you know, iterate on the prompt a little bit, but I think the…
1275 01:42:40.660 ⇒ 01:42:43.240 Samuel Roberts: Like, a more robust was another… there’s another step.
1276 01:42:43.710 ⇒ 01:42:55.189 Samuel Roberts: at least against service names, because that seems to be where this one broke down, but if we get a better sense from that spreadsheet that it’s doing other issues. But service names seems to be the, in my mind, the one that it has the least knowledge of, because.
1277 01:42:55.190 ⇒ 01:42:55.950 Uttam Kumaran: I agree.
1278 01:42:56.310 ⇒ 01:43:03.310 Samuel Roberts: branches and stuff, that’s places that it should have, and then it… I assume in the prompt is where it knows the difference between technicians and inspectors, so…
1279 01:43:03.910 ⇒ 01:43:04.540 Mustafa Raja: Yeah.
1280 01:43:05.180 ⇒ 01:43:11.750 Samuel Roberts: Maybe it is just a, like, do a quick search for service names that include any of those words or something, and pass that in, and, you know, it can…
1281 01:43:12.760 ⇒ 01:43:15.100 Samuel Roberts: See what it’s actually looking for, verify.
1282 01:43:17.300 ⇒ 01:43:18.160 Mustafa Raja: Yep. Okay.
1283 01:43:18.470 ⇒ 01:43:19.080 Uttam Kumaran: Okay.
1284 01:43:21.300 ⇒ 01:43:26.700 Uttam Kumaran: Okay, cool, so let me iterate on this. I don’t know, Mustafi, you want to just ping me when this is sort of, like…
1285 01:43:26.700 ⇒ 01:43:27.340 Mustafa Raja: Yes.
1286 01:43:27.340 ⇒ 01:43:32.550 Uttam Kumaran: posted, and then I’m gonna be up, so I’ll… I’ll keep working on this tonight, and I can send an email.
1287 01:43:32.780 ⇒ 01:43:33.390 Mustafa Raja: Okay.
1288 01:43:33.520 ⇒ 01:43:37.869 Samuel Roberts: Yeah, it’ll take me about, an hour or so.
1289 01:43:38.030 ⇒ 01:43:38.380 Uttam Kumaran: Okay.
1290 01:43:38.380 ⇒ 01:43:39.330 Mustafa Raja: helping you.
1291 01:43:39.620 ⇒ 01:43:43.020 Uttam Kumaran: And then tomorrow, yeah, I think…
1292 01:43:43.230 ⇒ 01:43:53.499 Uttam Kumaran: I’ll… I’m gonna… I’ll put a meeting on, and then let’s see how much of this migration we can… either part of it we can move over,
1293 01:43:54.840 ⇒ 01:43:57.950 Uttam Kumaran: Yeah, at least I want to see what we can do before…
1294 01:43:57.950 ⇒ 01:43:58.640 Samuel Roberts: Yeah.
1295 01:43:59.550 ⇒ 01:44:00.350 Uttam Kumaran: Like, Monday.
1296 01:44:00.350 ⇒ 01:44:01.100 Mustafa Raja: Yeah, what’s…
1297 01:44:01.100 ⇒ 01:44:09.090 Samuel Roberts: I bet we can get more than… than, like, we’ve been… because we… I’ve been a little more hands-off, just kind of guiding Casey and Mustafa, but I bet we can probably really lean in and…
1298 01:44:10.850 ⇒ 01:44:15.479 Samuel Roberts: It seems that, like, we can do some really more rigorous testing, too, now that we have some of these.
1299 01:44:16.000 ⇒ 01:44:17.730 Samuel Roberts: Things exported.
1300 01:44:19.150 ⇒ 01:44:21.080 Samuel Roberts: Comparing Nostra to NAN.
1301 01:44:22.450 ⇒ 01:44:23.060 Uttam Kumaran: Okay.
1302 01:44:23.310 ⇒ 01:44:24.509 Samuel Roberts: I’m, I’m… yeah.
1303 01:44:25.520 ⇒ 01:44:26.280 Samuel Roberts: Pretty.
1304 01:44:27.270 ⇒ 01:44:29.520 Samuel Roberts: I think we can make good progress there. Okay.
1305 01:44:29.520 ⇒ 01:44:33.070 Uttam Kumaran: Yeah, if we can even move parts of it, that would be really, really nice.
1306 01:44:33.070 ⇒ 01:44:36.249 Samuel Roberts: That’s kind of what… yeah, we had kind of broken it down into a few different…
1307 01:44:36.410 ⇒ 01:44:43.659 Samuel Roberts: parts of the agent, and one of the things we were… we weren’t necessarily gonna replace yet was the db query tool, because we knew that we were gonna get to that next, but we had already
1308 01:44:43.820 ⇒ 01:44:49.720 Samuel Roberts: Kind of scaffolded out the main context agent and the routing we had discussed.
1309 01:44:51.100 ⇒ 01:44:57.220 Samuel Roberts: Which are part of the NADN flows, but then we started talking about alternative
1310 01:44:57.680 ⇒ 01:45:05.679 Samuel Roberts: like, architectures for, like, you know, expert agents that know the right thing, and then the routing agent goes to them, but we can… we can hash that out later, but I think…
1311 01:45:06.220 ⇒ 01:45:10.729 Samuel Roberts: we were trying to do, like, a drop-in replacement initially, and I wanted to test a few other things.
1312 01:45:13.180 ⇒ 01:45:18.890 Uttam Kumaran: Okay, okay, perfect. Alright, so let’s, let’s do this today, and then we can talk tomorrow about some more information.
1313 01:45:18.890 ⇒ 01:45:19.250 Samuel Roberts: Sounds good.
1314 01:45:19.250 ⇒ 01:45:24.790 Amber Lin: Sounds good. Is Casey back tomorrow? He seems like he’s out until Wednesday.
1315 01:45:24.790 ⇒ 01:45:26.069 Samuel Roberts: I think he’s out till Wednesday, yeah.
1316 01:45:26.510 ⇒ 01:45:27.340 Amber Lin: Hmm.
1317 01:45:27.340 ⇒ 01:45:27.700 Uttam Kumaran: Okay.
1318 01:45:27.700 ⇒ 01:45:28.230 Amber Lin: trust.
1319 01:45:28.550 ⇒ 01:45:30.470 Uttam Kumaran: Yes.
1320 01:45:31.170 ⇒ 01:45:37.769 Amber Lin: Okay, and tomorrow morning, can you save the ABC until the end? Can I join in, like…
1321 01:45:37.770 ⇒ 01:45:42.839 Samuel Roberts: We were talking about a new meeting, so I was actually gonna ask, like, what is the… what’s, like, the earliest that…
1322 01:45:43.160 ⇒ 01:45:43.730 Samuel Roberts: Would be…
1323 01:45:43.730 ⇒ 01:45:59.690 Amber Lin: My next meeting is at 7.30, so I can try to get up at… and join about, like, 7, 10-ish, like, towards the end of the first meeting, if you can just reorder.
1324 01:45:59.810 ⇒ 01:46:02.200 Amber Lin: How you talk about things, is that alright?
1325 01:46:02.200 ⇒ 01:46:04.430 Uttam Kumaran: I mean, yeah, I guess it’s up to…
1326 01:46:04.570 ⇒ 01:46:08.599 Uttam Kumaran: It’s up to Sam. I just want to do something that’s, like, we can consistently meet.
1327 01:46:08.600 ⇒ 01:46:16.410 Samuel Roberts: Yeah, I can… I’m… Yeah. I mean, you probably need to be on the strategy, and… Analysis one, right?
1328 01:46:16.410 ⇒ 01:46:16.840 Amber Lin: Yeah.
1329 01:46:16.960 ⇒ 01:46:22.660 Amber Lin: But that one generally gets… like, latest, that gets done at
1330 01:46:22.930 ⇒ 01:46:28.749 Amber Lin: 8.30, if Robert talks about my clients early, I can get out closer to 8.
1331 01:46:29.010 ⇒ 01:46:36.929 Amber Lin: So, I’m usually free at my time, 8.30, which is Rob, your, like, PST 8.30, which is, like.
1332 01:46:37.910 ⇒ 01:46:38.370 Uttam Kumaran: Nope.
1333 01:46:38.370 ⇒ 01:46:43.450 Amber Lin: 10.30 Utum’s time, or, like… We can just do it right after.
1334 01:46:43.770 ⇒ 01:46:48.279 Samuel Roberts: I was… I was thinking that I’d just put an AI stand-up at 11.30 Eastern, which would be…
1335 01:46:48.660 ⇒ 01:46:51.580 Samuel Roberts: After all that, just to make sure we’re not conflicting.
1336 01:46:52.230 ⇒ 01:46:55.679 Samuel Roberts: Cool. And we’ll just check in tomorrow and then see if hopefully that time works going forward then.
1337 01:46:55.680 ⇒ 01:46:59.250 Uttam Kumaran: Yeah, let’s just do that, Sam, at least for the next, like, couple weeks, and then…
1338 01:46:59.250 ⇒ 01:47:00.620 Samuel Roberts: Yeah, that’s what, that’s… yeah, I…
1339 01:47:00.620 ⇒ 01:47:02.660 Uttam Kumaran: Gary will figure something out, yeah.
1340 01:47:02.660 ⇒ 01:47:03.510 Samuel Roberts: Okay, cool.
1341 01:47:03.740 ⇒ 01:47:12.279 Amber Lin: Yeah, tomorrow I’m… I won’t be able to make that time, I’m going to one of Yvette’s meetings, and then afterwards, I’ll be free.
1342 01:47:12.610 ⇒ 01:47:13.190 Uttam Kumaran: Okay.
1343 01:47:15.600 ⇒ 01:47:16.640 Uttam Kumaran: Okay, perfect.
1344 01:47:16.850 ⇒ 01:47:17.340 Mustafa Raja: Okay.
1345 01:47:17.340 ⇒ 01:47:17.820 Amber Lin: codes.
1346 01:47:17.820 ⇒ 01:47:18.310 Mustafa Raja: Thank you.
1347 01:47:18.550 ⇒ 01:47:19.510 Samuel Roberts: Alright, yeah.
1348 01:47:19.700 ⇒ 01:47:20.750 Samuel Roberts: Watch y’all later.
1349 01:47:21.860 ⇒ 01:47:22.490 Mustafa Raja: Yep.