Meeting Title: AI Service Daily Recap - Blockers + Realign Date: 2026-05-07 Meeting participants: Samuel Roberts, Mustafa Raja, Casie Aviles, Pranav
WEBVTT
1 00:00:21.800 ⇒ 00:00:22.770 Mustafa Raja: Hey…
2 00:00:23.780 ⇒ 00:00:24.870 Samuel Roberts: Hey, how’s it going?
3 00:00:27.010 ⇒ 00:00:28.359 Mustafa Raja: Good, how are you?
4 00:00:28.590 ⇒ 00:00:32.729 Samuel Roberts: Doing alright, just got those lawn zips in.
5 00:00:34.180 ⇒ 00:00:38.280 Samuel Roberts: We tested it, so, we had a pretty, pretty good day.
6 00:00:39.890 ⇒ 00:00:46.330 Mustafa Raja: Yeah, I got the email in pronounced… Draft.
7 00:00:46.810 ⇒ 00:00:47.270 Samuel Roberts: Oh, cool.
8 00:00:47.270 ⇒ 00:00:55.889 Mustafa Raja: No, I just need to work on the formatting, because it’s using Markdown. Oh. And it isn’t working in the… in the inbox, you know?
9 00:00:56.200 ⇒ 00:00:57.160 Samuel Roberts: Yeah.
10 00:00:57.190 ⇒ 00:00:59.969 Mustafa Raja: I’m wondering, yeah, girl.
11 00:01:00.120 ⇒ 00:01:04.860 Mustafa Raja: I’m wondering if there’s heading… if we can add headings and stuff.
12 00:01:06.650 ⇒ 00:01:08.079 Mustafa Raja: You know.
13 00:01:08.460 ⇒ 00:01:15.749 Samuel Roberts: Yeah, there’s a… I have not done a lot with emails, but I know there’s a library out there called React Email.
14 00:01:17.840 ⇒ 00:01:19.739 Samuel Roberts: It’s more for, like…
15 00:01:21.130 ⇒ 00:01:26.009 Samuel Roberts: I mean, it’ll help you format an email. I don’t know how pretty we want this to be, it could just be…
16 00:01:26.250 ⇒ 00:01:30.969 Samuel Roberts: you know, text. Essentially, an email is just HTML, right? So…
17 00:01:30.970 ⇒ 00:01:35.020 Mustafa Raja: Oh, yeah, yeah, I have worked with the HTML emails.
18 00:01:35.020 ⇒ 00:01:39.470 Samuel Roberts: Okay, so do you, like, there’s, like, a lot of little pitfalls with, like, tables and stuff.
19 00:01:39.470 ⇒ 00:01:40.310 Mustafa Raja: Yeah.
20 00:01:40.550 ⇒ 00:01:46.599 Samuel Roberts: But if we were just gonna do headers and stuff, we could probably just convert the markdown to HTML and paste it in, I would imagine.
21 00:01:47.310 ⇒ 00:01:48.060 Mustafa Raja: Yeah.
22 00:01:48.350 ⇒ 00:01:51.220 Mustafa Raja: Yeah, that’s… that’s going to be my next steps over here.
23 00:01:51.220 ⇒ 00:01:56.989 Samuel Roberts: Okay, yeah, that should be fine then, I think. As long as there’s, yeah, not crazy things like tables and stuff, then we may have to think about it a little bit more.
24 00:01:58.070 ⇒ 00:02:01.010 Mustafa Raja: For now, it doesn’t have any cables.
25 00:02:01.010 ⇒ 00:02:01.740 Samuel Roberts: Okay, cool.
26 00:02:01.740 ⇒ 00:02:07.240 Mustafa Raja: I don’t know, if future comparisons would… Add tables and stuff.
27 00:02:07.420 ⇒ 00:02:15.009 Samuel Roberts: Yeah, let’s, let’s, let’s just… if it’s just text, and lists and stuff, I would say just, yeah, you probably have to do a pre-process step, and then…
28 00:02:17.610 ⇒ 00:02:18.290 Mustafa Raja: Yes.
29 00:02:18.590 ⇒ 00:02:19.240 Samuel Roberts: Cool.
30 00:02:21.640 ⇒ 00:02:22.880 Samuel Roberts: How’s it going, guys?
31 00:02:24.270 ⇒ 00:02:25.150 Pranav: Hey, guys.
32 00:02:26.080 ⇒ 00:02:26.790 Mustafa Raja: Hey.
33 00:02:27.210 ⇒ 00:02:28.210 Samuel Roberts: How’s your day been?
34 00:02:31.720 ⇒ 00:02:33.359 Pranav: Pretty good, pretty good.
35 00:02:33.360 ⇒ 00:02:33.930 Samuel Roberts: Good.
36 00:02:35.950 ⇒ 00:02:38.839 Pranav: Yeah, okay. So…
37 00:02:39.340 ⇒ 00:02:52.800 Pranav: Sam, yeah, I saw your updates on the three tickets. That’s great. I think, that’s really good analysis. I think that’s really good for us to know. I think we’ve been noticing some issues with, like, the router, right? It’s causing…
38 00:02:52.960 ⇒ 00:02:59.459 Pranav: You know, I mean… there’s gonna be issues that we’re gonna… but I think,
39 00:02:59.670 ⇒ 00:03:08.419 Pranav: One thing that we were noticing was that there is, like, kind of these non-deterministic issues that we were noticing from at least a user’s perspective.
40 00:03:08.530 ⇒ 00:03:12.879 Pranav: But it’s looking like it’s not necessarily conflicting information all the time, which is what you showed.
41 00:03:13.220 ⇒ 00:03:13.760 Samuel Roberts: Yeah.
42 00:03:13.760 ⇒ 00:03:16.659 Pranav: So, yeah, that’s really good. I think…
43 00:03:16.660 ⇒ 00:03:34.810 Samuel Roberts: I started on the next part of that, which was, like, making the work I did a little more of a, okay, feed in a run that we saw was a problem, and we can run it with that history and all the context there, and then run it on its own, and that is how I kind of isolated it this time.
44 00:03:35.120 ⇒ 00:03:40.700 Samuel Roberts: But I pivoted to do the lawn care stuff, because I figured they’d want that sooner, so, now I can…
45 00:03:41.080 ⇒ 00:03:44.780 Samuel Roberts: Probably expand that, and that’ll give us, like, a quick check of…
46 00:03:44.910 ⇒ 00:03:47.910 Samuel Roberts: Okay, is it context that’s messing with this more than anything?
47 00:03:48.070 ⇒ 00:03:52.829 Samuel Roberts: Because that’s really, I think, where what you’re talking about with the non-determinism’s gonna come from, is, like.
48 00:03:53.300 ⇒ 00:03:57.370 Samuel Roberts: They’re not thinking about all the messages that they’ve already sent, necessarily.
49 00:03:58.590 ⇒ 00:04:03.960 Samuel Roberts: But sometimes they lean on that information, so we can’t just get rid of the memory, we want some of that, right?
50 00:04:04.890 ⇒ 00:04:08.329 Pranav: Yeah, I don’t really know how much of memory they’re using, though.
51 00:04:08.330 ⇒ 00:04:24.150 Samuel Roberts: Well, it’s only 5 messages, is what it is for the… Yeah. It’s not, like, a ton, but I think it’s enough that, like, when there’s a follow-up question, we need to make sure that there’s, like, something there. There’s also some stuff that I was seeing about, like, the user context, but I’m not really sure if that was…
52 00:04:24.230 ⇒ 00:04:28.749 Samuel Roberts: about it or not, but I think, yeah, if I put a script together, we can start to identify that more as more of these kind of…
53 00:04:28.980 ⇒ 00:04:31.680 Samuel Roberts: Non-deterministic things pop up.
54 00:04:32.270 ⇒ 00:04:33.580 Pranav: Yeah, sounds good.
55 00:04:34.730 ⇒ 00:04:38.549 Pranav: And so the status of those 3 tickets, are they complete?
56 00:04:39.170 ⇒ 00:04:44.590 Samuel Roberts: The, so the…
57 00:04:44.990 ⇒ 00:04:49.559 Samuel Roberts: isolating that, investigating that, I guess. Yeah, I put it in PR review,
58 00:04:49.800 ⇒ 00:05:02.949 Samuel Roberts: it’s not necessarily merged in yet, I wanted to, like, just, you know, get someone else’s eyes on it, obviously. But it’s just a… it essentially just ended up being a prompt change that worked with the scripts, but the scripts are there. And then the…
59 00:05:03.470 ⇒ 00:05:12.779 Samuel Roberts: The prompt non-determinism assessment, that’s what I started on, but then I switched over to the lawn care one, and those are all in…
60 00:05:12.890 ⇒ 00:05:16.950 Pranav: PR review? Can we just look at that one real quick, and we can just probably merge it?
61 00:05:16.950 ⇒ 00:05:19.690 Samuel Roberts: Yeah, cool, cool. Yeah, totally. I was just waiting, because I…
62 00:05:19.980 ⇒ 00:05:24.159 Samuel Roberts: Let me get that up. Where is GitHub?
63 00:05:30.870 ⇒ 00:05:39.630 Samuel Roberts: Okay, so… where is Zoom? Let me just share this real quick.
64 00:05:42.080 ⇒ 00:05:47.630 Samuel Roberts: Okay, so, it basically… I’ll probably build off of this for the next one.
65 00:05:47.860 ⇒ 00:05:51.369 Samuel Roberts: But, you see there’s just a few scripts here, effectively.
66 00:05:51.490 ⇒ 00:05:58.510 Samuel Roberts: One, I added the Cloud SQL proxy because I needed that, and thank you, Mustafa, for the work on Eden for that. I was basically able to reuse it.
67 00:05:58.640 ⇒ 00:06:09.339 Samuel Roberts: So that’s… that’s good. And then, this was, like, replay the specific one that I was trying to do, and this is… compare them after the fact, and make sure that the template still worked, and the…
68 00:06:09.770 ⇒ 00:06:14.129 Samuel Roberts: Other calculators still worked. And so the only, like, real change is essentially…
69 00:06:14.290 ⇒ 00:06:18.610 Samuel Roberts: this part of the prompt. Like, that’s what it came down to, which is not much, but,
70 00:06:18.720 ⇒ 00:06:19.730 Samuel Roberts: I think…
71 00:06:20.010 ⇒ 00:06:24.709 Samuel Roberts: this is really all we gotta worry about. The rest of them are things I’m gonna build off of for the next part, so…
72 00:06:24.710 ⇒ 00:06:27.439 Pranav: What are you calling a noun phrase?
73 00:06:28.740 ⇒ 00:06:34.189 Samuel Roberts: Something stated, like, cap… like, calculator. Like, it thought, for some reason that that… the,
74 00:06:34.480 ⇒ 00:06:39.090 Samuel Roberts: There’s another example that was…
75 00:06:39.260 ⇒ 00:06:42.780 Samuel Roberts: Let me see if it’s in here,
76 00:06:44.430 ⇒ 00:06:47.079 Samuel Roberts: There was something that should be a template.
77 00:06:47.680 ⇒ 00:06:51.830 Samuel Roberts: And it knew that, I don’t know if the example’s gonna be here.
78 00:06:52.420 ⇒ 00:06:56.730 Samuel Roberts: Because I think it was fetching them, yeah, okay. There was,
79 00:06:59.450 ⇒ 00:07:00.780 Samuel Roberts: Oh, actually, it might just be…
80 00:07:03.930 ⇒ 00:07:10.120 Samuel Roberts: Here, so, like, like, if they get revenue adjustments, right?
81 00:07:10.930 ⇒ 00:07:14.139 Samuel Roberts: This was the example that was in here, and that’s what it thought was…
82 00:07:14.360 ⇒ 00:07:20.410 Samuel Roberts: messing with it, whereas, like, calculator, it was like, oh, I gotta look for that word in vector query.
83 00:07:22.460 ⇒ 00:07:27.509 Samuel Roberts: So, like, instead of assuming it’s a template, it’s basically, like, let’s assume it’s not a template, check the vector.
84 00:07:28.130 ⇒ 00:07:35.149 Samuel Roberts: Otherwise, it should match, like, script, instructions, template kind of thing.
85 00:07:37.070 ⇒ 00:07:44.849 Pranav: Okay, so aside from just, tree calculator, did you test this against other things that are supposed to…
86 00:07:44.970 ⇒ 00:07:45.690 Pranav: You know…
87 00:07:45.690 ⇒ 00:07:53.029 Samuel Roberts: I tested against the other calculator. In fact, the AI initially want, like, had it hard-coded as, like, TreeCalculator as the example.
88 00:07:53.130 ⇒ 00:07:58.310 Samuel Roberts: But I pulled that out and tested it against the, holiday Lights Calculator.
89 00:08:00.750 ⇒ 00:08:04.600 Samuel Roberts: But that was the only one that I had, like, any sense of what was a good one to test with.
90 00:08:04.600 ⇒ 00:08:08.499 Pranav: Okay. So, yeah, by asking… by adding that part before.
91 00:08:09.150 ⇒ 00:08:13.739 Pranav: that’s going to prevent from… because we still want to be able to support template requests, right?
92 00:08:13.740 ⇒ 00:08:17.049 Samuel Roberts: And that’s… that’s what the other script did. It did… it checked for both.
93 00:08:17.050 ⇒ 00:08:18.070 Pranav: Perfect.
94 00:08:18.820 ⇒ 00:08:21.990 Samuel Roberts: Made sure lots of template… yeah, revenue adjustment still worked.
95 00:08:22.260 ⇒ 00:08:22.680 Pranav: Nice.
96 00:08:22.680 ⇒ 00:08:27.610 Samuel Roberts: And the Holiday Lights Calculator was the other one I tested. But we could always add more cases there, too, so…
97 00:08:27.890 ⇒ 00:08:30.399 Pranav: Okay, cool. No, no, no, this is… this is great, yeah, that’s…
98 00:08:30.400 ⇒ 00:08:34.270 Samuel Roberts: Yeah, no, it was… honestly, the AI was, like… We dug deep, and it…
99 00:08:34.570 ⇒ 00:08:41.470 Samuel Roberts: was able to pol… I figured it was a context thing, and sure enough, it was doing weird stuff without… that’s the problem with the LLMs, is like, you know, it’s just…
100 00:08:41.630 ⇒ 00:08:45.229 Samuel Roberts: you pass stuff in, you get stuff out, and so I was like, if we’re passing the same thing in, it shouldn’t…
101 00:08:45.360 ⇒ 00:08:47.929 Samuel Roberts: And then I saw the template, so, yeah.
102 00:08:48.260 ⇒ 00:08:49.529 Samuel Roberts: That’s fun, cool.
103 00:08:49.940 ⇒ 00:08:54.580 Samuel Roberts: Yeah, so if we’re good with that prompt change, like I said, I tested it against those two things, but.
104 00:08:55.540 ⇒ 00:08:57.310 Pranav: Yeah, we’re good with that.
105 00:08:57.310 ⇒ 00:08:57.720 Samuel Roberts: Cool.
106 00:08:57.720 ⇒ 00:09:02.230 Pranav: Oh, yeah, you said you’re gonna keep this open for just, for what exactly?
107 00:09:02.230 ⇒ 00:09:11.020 Samuel Roberts: No, no, I was gonna close it, but I’m saying I’ll build off of this, the stuff in the scripts, to put together the next, like, more generic tool for doing this kind of analysis.
108 00:09:11.020 ⇒ 00:09:14.419 Pranav: Yeah, that sounds great. Okay, cool. So yeah, this is… this is good to merge them.
109 00:09:15.380 ⇒ 00:09:15.819 Samuel Roberts: Let’s do it.
110 00:09:17.060 ⇒ 00:09:25.100 Pranav: Alright, yeah, so Mustafa, we got that scope. How are we looking for the… The automated email.
111 00:09:26.150 ⇒ 00:09:33.930 Mustafa Raja: Yeah, so, Irida and the workflow, and, in your Eden email, you can see the first draft.
112 00:09:35.130 ⇒ 00:09:36.360 Mustafa Raja: Take a look at the graph.
113 00:09:36.510 ⇒ 00:09:46.869 Mustafa Raja: I’d say that the formatting is something that I need to figure out, but you can take a look at the draft on, you know, the workflow is working, and…
114 00:09:47.530 ⇒ 00:09:52.530 Mustafa Raja: Once I get the formatting right, I can schedule it on a GCP scheduler.
115 00:09:53.520 ⇒ 00:09:54.100 Pranav: Okay.
116 00:09:54.840 ⇒ 00:09:57.689 Mustafa Raja: Let me know what would be a good time for it, you know?
117 00:09:58.360 ⇒ 00:09:59.260 Mustafa Raja: Yeah, so I think.
118 00:09:59.260 ⇒ 00:10:03.540 Pranav: Oh, this is like a once-a-week thing, so probably, Monday morning.
119 00:10:04.680 ⇒ 00:10:06.000 Mustafa Raja: Monday morning, okay.
120 00:10:06.000 ⇒ 00:10:06.610 Pranav: Yeah.
121 00:10:09.010 ⇒ 00:10:10.129 Mustafa Raja: Mountain time?
122 00:10:11.770 ⇒ 00:10:15.799 Pranav: Yeah, have it run, like, 6 AM.
123 00:10:16.150 ⇒ 00:10:18.070 Samuel Roberts: That’s what I was gonna say.
124 00:10:21.900 ⇒ 00:10:22.580 Mustafa Raja: Yeah.
125 00:10:22.850 ⇒ 00:10:26.079 Mustafa Raja: See, yeah, that, that, that, formatting, plus…
126 00:10:26.250 ⇒ 00:10:32.089 Mustafa Raja: Merging this PR and then getting it up and running, on Schedule A is my next steps.
127 00:10:33.110 ⇒ 00:10:34.620 Pranav: Okay, cool, that’s awesome.
128 00:10:34.890 ⇒ 00:10:35.550 Samuel Roberts: Yeah.
129 00:10:36.220 ⇒ 00:10:41.130 Pranav: Alright, Casey, how’s your progress been?
130 00:10:42.910 ⇒ 00:10:49.279 Casie Aviles: Yeah, I’ve just been working on the spreadsheets now, and I can share what they look like right now.
131 00:10:49.590 ⇒ 00:10:53.900 Casie Aviles: I’m currently just… Testing, doing some more tests.
132 00:10:54.380 ⇒ 00:10:57.630 Casie Aviles: As I’m finding some edge cases here and there,
133 00:11:02.140 ⇒ 00:11:08.870 Casie Aviles: Yeah, okay. So, for example, we have… so I created this, Google Drive here.
134 00:11:11.030 ⇒ 00:11:14.369 Casie Aviles: And it should be that no feedback reports.
135 00:11:15.440 ⇒ 00:11:21.999 Casie Aviles: And yeah, it’s really just all the records that did not get any thumbs result, and no detailed feedback, so…
136 00:11:22.850 ⇒ 00:11:27.130 Casie Aviles: And then this will, basically when on the time of trigger.
137 00:11:27.400 ⇒ 00:11:33.469 Casie Aviles: It will get the logs from that date and 7 days… or, yeah, 7 days back.
138 00:11:34.250 ⇒ 00:11:34.750 Pranav: Gotcha.
139 00:11:34.750 ⇒ 00:11:35.520 Casie Aviles: So…
140 00:11:36.000 ⇒ 00:11:43.329 Casie Aviles: This is what it looks like right now. There are… there’s just something weird with the… with how the dates are being encoded right now.
141 00:11:43.580 ⇒ 00:11:45.459 Casie Aviles: So, I’ll have to fix that.
142 00:11:45.990 ⇒ 00:11:46.319 Pranav: That’s fine.
143 00:11:46.320 ⇒ 00:11:46.860 Casie Aviles: readable.
144 00:11:46.860 ⇒ 00:11:49.899 Samuel Roberts: Is that a, just a formatting thing in Sheets?
145 00:11:51.130 ⇒ 00:11:53.979 Casie Aviles: Yeah, I think so, it might just be the…
146 00:11:56.050 ⇒ 00:11:57.220 Samuel Roberts: Yeah, just mind if I…
147 00:11:57.220 ⇒ 00:11:58.050 Casie Aviles: medical.
148 00:11:58.050 ⇒ 00:12:02.090 Samuel Roberts: Yeah, I think Excel does weird things as well, I just… I bet that carried over.
149 00:12:02.640 ⇒ 00:12:07.590 Samuel Roberts: I might even just be able to change the format into the date, but I don’t have a total read that.
150 00:12:08.290 ⇒ 00:12:11.219 Pranav: Yeah, but then every time you create a sheet, then you’re gonna have to do that, so…
151 00:12:11.220 ⇒ 00:12:15.500 Samuel Roberts: Yeah, so you might just want to paste it as, like, a timestamp or something.
152 00:12:15.730 ⇒ 00:12:17.280 Samuel Roberts: As a string, maybe.
153 00:12:18.970 ⇒ 00:12:27.180 Casie Aviles: Yeah, but if we… yeah, when we click format to date, it becomes… it shows, like, the right dates now, but yeah, I have to fix now, it’s being encoded.
154 00:12:27.640 ⇒ 00:12:28.520 Samuel Roberts: Okay, yeah.
155 00:12:28.520 ⇒ 00:12:32.820 Pranav: Cool. And then, also here, what we want is a section for…
156 00:12:33.230 ⇒ 00:12:43.699 Pranav: Honestly, what we need here, necessarily, is not necessarily that we get, like, the department or the thumbs result, because you’re already splitting this by department, right?
157 00:12:44.320 ⇒ 00:12:45.150 Casie Aviles: Yeah, yeah.
158 00:12:45.150 ⇒ 00:12:50.009 Pranav: Yeah, so… I mean, unless those are helpful for you in terms of debugging.
159 00:12:50.300 ⇒ 00:12:52.019 Pranav: We can, we can honestly, I mean…
160 00:12:52.560 ⇒ 00:12:57.159 Pranav: We could probably just remove that, is my… is my thinking.
161 00:12:58.080 ⇒ 00:12:58.850 Casie Aviles: Yeah, okay.
162 00:12:58.850 ⇒ 00:13:02.350 Pranav: And then detailed feedback is always gonna be empty, right, since there was no…
163 00:13:03.200 ⇒ 00:13:03.730 Casie Aviles: Yep.
164 00:13:03.730 ⇒ 00:13:04.209 Pranav: I was going.
165 00:13:04.210 ⇒ 00:13:05.310 Casie Aviles: It’s gonna be empty.
166 00:13:05.310 ⇒ 00:13:08.679 Pranav: Okay, cool. So, basically what I want here is just an…
167 00:13:09.210 ⇒ 00:13:15.480 Pranav: Like, two rows. Basically, one row saying thumbs up, thumbs down, which is gonna be the user’s input.
168 00:13:15.580 ⇒ 00:13:19.759 Pranav: And then… Yeah, so… we can probably just…
169 00:13:20.220 ⇒ 00:13:28.399 Pranav: yeah, have that on, like, column H or whatever. I’ll let you figure that out. And then one more row saying feedback.
170 00:13:28.540 ⇒ 00:13:36.730 Pranav: So, what is the explanation for what you gave? If it’s a thumbs up, then it doesn’t need feedback. If it’s thumbs down, then it’s required to have feedback.
171 00:13:39.690 ⇒ 00:13:43.960 Casie Aviles: Okay, wait, sorry, can you say that one more time?
172 00:13:44.230 ⇒ 00:13:52.250 Pranav: Yeah, sure. So, I just want to make sure, like, you kind of understand, like, what this is going to be doing. And so, we’re gonna be getting.
173 00:13:52.250 ⇒ 00:13:53.200 Casie Aviles: Yeah.
174 00:13:53.680 ⇒ 00:13:54.340 Pranav: Okay.
175 00:13:54.730 ⇒ 00:14:00.500 Casie Aviles: I remember that we have to, like, we need them to input the detailed feedback, right?
176 00:14:01.000 ⇒ 00:14:03.660 Casie Aviles: So we need a column for that.
177 00:14:04.820 ⇒ 00:14:07.490 Pranav: Oh, so is that what this detailed feedback column is?
178 00:14:08.270 ⇒ 00:14:11.060 Casie Aviles: Yeah, that’s… well, this is what’s…
179 00:14:11.510 ⇒ 00:14:17.609 Casie Aviles: coming from BigQuery, but I thought we could also just have it… have them write the feedback here.
180 00:14:17.610 ⇒ 00:14:24.460 Pranav: Yeah, yeah, that’s totally fine, actually. Yeah, we could do that. So then what you can do for thumbs result is…
181 00:14:25.210 ⇒ 00:14:28.429 Pranav: you can just leave… you can remove the none, I guess?
182 00:14:28.910 ⇒ 00:14:31.709 Pranav: And just… Keep it as empty as well.
183 00:14:32.600 ⇒ 00:14:33.300 Casie Aviles: Okay.
184 00:14:33.300 ⇒ 00:14:45.629 Pranav: Yeah. And then what would be good there is if we can… instead of having them type yes or no, or thumbs up, thumbs down, just having a dropdown where they can just, like, select one. So it’s like an enum.
185 00:14:46.690 ⇒ 00:14:47.619 Casie Aviles: Okay, okay.
186 00:14:47.620 ⇒ 00:14:48.489 Pranav: Yeah, that would be great.
187 00:14:48.490 ⇒ 00:14:49.210 Casie Aviles: That’s good.
188 00:14:50.480 ⇒ 00:14:51.000 Pranav: Okay, cool.
189 00:14:51.000 ⇒ 00:14:54.859 Samuel Roberts: Is there other… is this all the metadata we’re getting through department on here?
190 00:14:55.390 ⇒ 00:14:58.169 Samuel Roberts: Is this A through G? Is there anything else?
191 00:15:00.490 ⇒ 00:15:06.240 Casie Aviles: I mean, from Bigger, there are… there’s probably more columns there, like, the…
192 00:15:06.240 ⇒ 00:15:06.930 Samuel Roberts: Yeah.
193 00:15:07.470 ⇒ 00:15:08.810 Samuel Roberts: What I was finding…
194 00:15:08.810 ⇒ 00:15:10.079 Casie Aviles: Constitution ID.
195 00:15:10.280 ⇒ 00:15:22.999 Samuel Roberts: Yeah, I think if we at least keep those tied to these, that will also help, because, like, the script I was running earlier, I was able to just, like, pull the context and look back in the conversation and stuff pretty easily, and so…
196 00:15:23.130 ⇒ 00:15:28.220 Samuel Roberts: I think if we’re gonna start using this for, like, more testing and evals and have a decent data set.
197 00:15:28.420 ⇒ 00:15:35.410 Samuel Roberts: We might want to be able to at least correlate that to the previous runs and the context that actually was fed in and stuff.
198 00:15:36.780 ⇒ 00:15:37.850 Samuel Roberts: If that makes sense.
199 00:15:38.760 ⇒ 00:15:41.469 Pranav: Wait, how are you thinking about using this as evals?
200 00:15:41.940 ⇒ 00:15:46.630 Samuel Roberts: If they’re, like, if they’re… if we’re getting feedback that is, like, thumbs up, right?
201 00:15:50.430 ⇒ 00:15:54.500 Samuel Roberts: We could be pulling that out, and if they’re getting… giving good feedback on…
202 00:15:55.050 ⇒ 00:16:01.009 Samuel Roberts: You know, neutral, or no thumbs up, or no thumbs, or thumbs down in the feedback.
203 00:16:01.110 ⇒ 00:16:04.590 Samuel Roberts: Then over time, that’ll start to become like…
204 00:16:05.090 ⇒ 00:16:07.800 Samuel Roberts: Like, a known knowledge set that we have.
205 00:16:08.450 ⇒ 00:16:09.910 Pranav: Right.
206 00:16:09.910 ⇒ 00:16:10.720 Samuel Roberts: that, like.
207 00:16:10.720 ⇒ 00:16:11.370 Pranav: Yeah.
208 00:16:11.540 ⇒ 00:16:18.080 Samuel Roberts: If we have that tied to… like, if we can, like, take this and, like, whatever they’ve added to this and annotate it effectively…
209 00:16:19.740 ⇒ 00:16:24.789 Samuel Roberts: we can use the run ID, or the execution ID, or whatever it is that corresponds
210 00:16:25.200 ⇒ 00:16:30.920 Samuel Roberts: And kind of run a similar script to what I was doing earlier, where we can eval against that.
211 00:16:31.510 ⇒ 00:16:37.610 Pranav: Oh, I see. So then, instead of having the input as the reference, we can have the…
212 00:16:37.940 ⇒ 00:16:42.449 Pranav: the run ID as a reference for, like, what could be, like, a golden dataset.
213 00:16:42.760 ⇒ 00:16:51.169 Samuel Roberts: Yeah, I think because, like, the thing about what I was finding earlier was that when you say, like, how are rewards, whatever, calculated, it’s getting past a bunch of other stuff.
214 00:16:52.210 ⇒ 00:16:53.550 Samuel Roberts: Usually, in the last.
215 00:16:53.550 ⇒ 00:16:53.910 Pranav: Who knows?
216 00:16:53.910 ⇒ 00:17:04.330 Samuel Roberts: So, like, my thought is that this is good, but I think knowing what the rest of the context might have been, if we really need to dig into something, like, or if we run into some non-determinism kind of stuff again.
217 00:17:04.660 ⇒ 00:17:10.300 Samuel Roberts: But I want to be able to correlate whatever these entries are back to BigQuery, and then BigQuery can help me
218 00:17:10.540 ⇒ 00:17:13.510 Samuel Roberts: figure out the rest of the context that was passed, if that makes sense.
219 00:17:13.770 ⇒ 00:17:16.959 Pranav: Cool. Yeah, yeah. I mean, shouldn’t hurt to bring that information.
220 00:17:16.960 ⇒ 00:17:19.159 Samuel Roberts: That’s what I’m saying, just, like, yeah, don’t drop the rest of the meta.
221 00:17:19.640 ⇒ 00:17:27.140 Samuel Roberts: like, execution, stuff like that, so that if at some point we see a real, you know… Otherwise, like, digging through finding this input and that date is a little bit of a pain in the butt.
222 00:17:29.030 ⇒ 00:17:35.460 Pranav: Okay. Yeah, so I also, since this is gonna be client-facing, I don’t want to just put, like, a bunch of metrics in here, so I think…
223 00:17:35.460 ⇒ 00:17:37.589 Samuel Roberts: Sure, sure, sure. No, I would just include the… whatever…
224 00:17:37.590 ⇒ 00:17:38.880 Pranav: It’s got, like, the runway beat.
225 00:17:39.450 ⇒ 00:17:46.269 Pranav: Yeah, yeah, let’s just get the run ID, so then we can… using just that, we can pull what’s relevant from BigQuery.
226 00:17:46.990 ⇒ 00:17:49.490 Samuel Roberts: Exactly, exactly. Just so that it’s easier to tie back.
227 00:17:49.490 ⇒ 00:17:59.149 Pranav: Yeah, so, and also, Casey, I think it would be helpful as well if we can… yeah, of the columns we don’t remove, like, input and output, username.
228 00:17:59.720 ⇒ 00:18:01.160 Pranav: Date as well.
229 00:18:01.320 ⇒ 00:18:06.270 Pranav: Is there a way to lock that? So they can’t modify those things, or…
230 00:18:07.360 ⇒ 00:18:08.350 Pranav: Is that…
231 00:18:08.350 ⇒ 00:18:13.079 Casie Aviles: I haven’t… I don’t… I’m not aware if we could lock with specific columns.
232 00:18:13.350 ⇒ 00:18:15.150 Casie Aviles: I… Yeah.
233 00:18:15.150 ⇒ 00:18:17.269 Samuel Roberts: We can, like, freeze certain things, I don’t know.
234 00:18:19.430 ⇒ 00:18:19.840 Pranav: Okay.
235 00:18:19.840 ⇒ 00:18:20.390 Samuel Roberts: Fair enough.
236 00:18:21.470 ⇒ 00:18:23.940 Pranav: Yeah, it’s worth doing just, like, a quick search to see if that’s.
237 00:18:23.940 ⇒ 00:18:27.069 Samuel Roberts: Oh, that’s for scrolling, that’s right, okay, I might be locked in.
238 00:18:27.070 ⇒ 00:18:28.200 Pranav: Maybe I’ll just do that right now.
239 00:18:28.370 ⇒ 00:18:32.199 Samuel Roberts: Protect Sheets and range. Yeah, data, protect sheets and range, yeah.
240 00:18:32.810 ⇒ 00:18:34.840 Pranav: Oh, so you can do that, okay, perfect.
241 00:18:37.190 ⇒ 00:18:38.880 Samuel Roberts: Yeah, exactly, that’s exactly what I just saw.
242 00:18:41.270 ⇒ 00:18:42.669 Casie Aviles: Okay, yeah, I’ll do that.
243 00:18:42.790 ⇒ 00:18:43.679 Samuel Roberts: That’s smart, yeah.
244 00:18:43.680 ⇒ 00:18:44.749 Casie Aviles: Bye, blah, blah.
245 00:18:45.440 ⇒ 00:18:46.120 Pranav: Yeah.
246 00:18:50.130 ⇒ 00:18:51.380 Pranav: Okay. Cool, guys.
247 00:18:51.380 ⇒ 00:18:52.290 Casie Aviles: Okay, yeah.
248 00:18:52.570 ⇒ 00:18:53.280 Samuel Roberts: Sweet.
249 00:18:56.140 ⇒ 00:19:05.130 Pranav: Sounds good. Doesn’t sound like there’s any blockers then for right now. Mustafa, you’ll keep working on that email, Casey, you’ll keep working on this.
250 00:19:05.530 ⇒ 00:19:11.039 Pranav: Yeah, sounds like we’re in a good spot. And then, yeah, Sam, you’ll just continue working on that script.
251 00:19:11.410 ⇒ 00:19:12.839 Samuel Roberts: Yeah, yeah, that’s the next thing.
252 00:19:13.490 ⇒ 00:19:15.949 Pranav: Awesome. Yeah, no, this is great progress, guys. Cool.
253 00:19:15.950 ⇒ 00:19:16.510 Samuel Roberts: Yeah.
254 00:19:18.110 ⇒ 00:19:19.220 Pranav: Alright, anything else?
255 00:19:19.220 ⇒ 00:19:20.890 Samuel Roberts: I’ll later.
256 00:19:20.890 ⇒ 00:19:21.520 Pranav: Alright, perfect.
257 00:19:21.750 ⇒ 00:19:23.429 Pranav: Yeah, we’ll talk soon. See ya.
258 00:19:23.630 ⇒ 00:19:24.680 Samuel Roberts: Alright, bye.