Meeting Title: AI-PR-Review-Automation-Feedback-Brainstorming Date: 2024-09-26 Meeting participants: Nicolas Sucari, Joshuadeveyra, Uttam Kumaran, Brian Pei, Ryan Luke Daque, Patrick Trainer
WEBVTT
1 00:05:28.650 ⇒ 00:05:29.510 Patrick Trainer: Yo.
2 00:05:34.130 ⇒ 00:05:35.713 Ryan Luke Daque: Hello! Hello! How’s it going?
3 00:05:36.240 ⇒ 00:05:37.680 Patrick Trainer: Going good! How are you.
4 00:05:38.560 ⇒ 00:05:40.090 Ryan Luke Daque: Doing well, so far.
5 00:05:40.350 ⇒ 00:05:41.250 Patrick Trainer: Heard that.
6 00:05:42.440 ⇒ 00:05:43.410 Ryan Luke Daque: Nice.
7 00:05:44.010 ⇒ 00:05:44.610 Patrick Trainer: That.
8 00:06:31.580 ⇒ 00:06:33.169 Ryan Luke Daque: How’s the week going pat.
9 00:06:34.210 ⇒ 00:06:38.214 Patrick Trainer: Week’s been good. Yeah. Can’t complain too much. It’s
10 00:06:39.250 ⇒ 00:06:39.910 Patrick Trainer: it.
11 00:06:40.160 ⇒ 00:06:42.870 Patrick Trainer: The humidity and temperature
12 00:06:42.900 ⇒ 00:06:45.809 Patrick Trainer: dropped a good like 5, 10 degrees, and.
13 00:06:46.020 ⇒ 00:06:46.300 Ryan Luke Daque: Wow!
14 00:06:46.300 ⇒ 00:06:47.619 Patrick Trainer: Stay that way for
15 00:06:47.660 ⇒ 00:06:51.800 Patrick Trainer: like at least the next week. And so it’s feeling amazing.
16 00:06:52.370 ⇒ 00:06:53.550 Ryan Luke Daque: Nice.
17 00:06:53.550 ⇒ 00:06:54.650 Patrick Trainer: Yeah, it’s
18 00:06:56.140 ⇒ 00:06:58.171 Patrick Trainer: yeah, really, really, nice.
19 00:06:59.250 ⇒ 00:07:01.890 Ryan Luke Daque: No longer raining and and stuff.
20 00:07:02.050 ⇒ 00:07:04.580 Patrick Trainer: Yeah, no, not not quite yet.
21 00:07:05.040 ⇒ 00:07:06.050 Ryan Luke Daque: Yeah, cool.
22 00:07:06.260 ⇒ 00:07:06.930 Patrick Trainer: Yeah.
23 00:07:07.310 ⇒ 00:07:08.690 Patrick Trainer: And it’s
24 00:07:08.760 ⇒ 00:07:11.839 Patrick Trainer: we looked out. There’s like a huge hurricane
25 00:07:11.910 ⇒ 00:07:13.669 Patrick Trainer: coming. But it’s
26 00:07:13.820 ⇒ 00:07:15.620 Patrick Trainer: gonna hit Florida
27 00:07:15.690 ⇒ 00:07:17.920 Patrick Trainer: instead of us. And
28 00:07:17.930 ⇒ 00:07:19.450 Patrick Trainer: it’s weird
29 00:07:19.700 ⇒ 00:07:30.730 Patrick Trainer: stressing out for a little bit, but it’s it’s taking a turn to the east. Gonna hit them via. It would have been a big, real big pain in the ass. But it’s gonna be a
30 00:07:31.660 ⇒ 00:07:35.781 Patrick Trainer: quite big for the people that it does hit.
31 00:07:36.660 ⇒ 00:07:38.437 Ryan Luke Daque: Yeah, that sucks.
32 00:07:49.430 ⇒ 00:07:58.570 Ryan Luke Daque: Wonder if Tom’s joining. He seems to be online, though, so I guess he’ll be able to join a bit. Let’s probably wait for a few minutes.
33 00:07:59.100 ⇒ 00:07:59.630 Patrick Trainer: Yeah.
34 00:09:54.780 ⇒ 00:10:01.439 Ryan Luke Daque: Yeah, it looks like, but I’m still in a different call. Maybe he’ll be joining in like 2 min or so.
35 00:10:01.820 ⇒ 00:10:06.269 Ryan Luke Daque: so maybe we can just probably start like discussing.
36 00:10:09.110 ⇒ 00:10:16.579 Ryan Luke Daque: the workflow, I guess, like, basically, the purpose of this call was just to gather like feedback and like brainstorm, some
37 00:10:16.630 ⇒ 00:10:19.699 Ryan Luke Daque: ideas as to like how we can utilize.
38 00:10:19.710 ⇒ 00:10:22.089 Ryan Luke Daque: you know. Chat, Gpt, or whatever
39 00:10:22.340 ⇒ 00:10:25.700 Ryan Luke Daque: Api AI Api like we can use.
40 00:10:26.640 ⇒ 00:10:30.539 Ryan Luke Daque: Yeah, let me probably go ahead and share my screen.
41 00:10:38.180 ⇒ 00:10:39.490 Ryan Luke Daque: okay.
42 00:10:42.040 ⇒ 00:10:46.500 Ryan Luke Daque: yeah. So I created this like notion document, which is basically just
43 00:10:47.400 ⇒ 00:10:53.779 Ryan Luke Daque: a summary of what we did for this specific workflow. And we just followed what the blog said.
44 00:10:53.930 ⇒ 00:10:55.417 Ryan Luke Daque: and this has like,
45 00:10:57.140 ⇒ 00:10:58.220 Ryan Luke Daque: this is the
46 00:10:58.610 ⇒ 00:11:01.077 Ryan Luke Daque: they had the. They have their like
47 00:11:01.460 ⇒ 00:11:03.590 Ryan Luke Daque: Github repo, which was
48 00:11:03.670 ⇒ 00:11:06.470 Ryan Luke Daque: basically doing the
49 00:11:07.060 ⇒ 00:11:10.137 Ryan Luke Daque: the code review already. Right? They have this like,
50 00:11:12.430 ⇒ 00:11:15.390 Ryan Luke Daque: they have a already a set prompt
51 00:11:16.210 ⇒ 00:11:18.260 Ryan Luke Daque: to forward that
52 00:11:18.720 ⇒ 00:11:20.699 Ryan Luke Daque: the AI to do the
53 00:11:20.890 ⇒ 00:11:21.980 Ryan Luke Daque: review.
54 00:11:23.530 ⇒ 00:11:24.214 Ryan Luke Daque: And
55 00:11:25.280 ⇒ 00:11:34.169 Ryan Luke Daque: so this is basically, if if we can copy this kind of Repost repository, we can have our own prompt depending on what we wanted to do like
56 00:11:34.180 ⇒ 00:11:37.210 Ryan Luke Daque: one of the things that I I was thinking of was like
57 00:11:39.790 ⇒ 00:11:43.989 Ryan Luke Daque: utilizing AI to investigate failing workflows, for example.
58 00:11:44.880 ⇒ 00:11:45.666 Ryan Luke Daque: or like.
59 00:11:46.170 ⇒ 00:11:51.620 Ryan Luke Daque: yeah, just coding style checks. And just like sequel, fluff and stuff like that. But for
60 00:11:52.250 ⇒ 00:11:53.570 Ryan Luke Daque: anything else.
61 00:11:55.190 ⇒ 00:12:02.459 Ryan Luke Daque: Or like, maybe even this would be a great one. Because we’re we’re not doing this at the moment like automated release notes generation.
62 00:12:02.830 ⇒ 00:12:09.959 Ryan Luke Daque: So like we can, we can have releases versioning and stuff like that for our repositories. But
63 00:12:10.210 ⇒ 00:12:13.590 Ryan Luke Daque: this is probably not that important at the moment.
64 00:12:13.990 ⇒ 00:12:15.069 Ryan Luke Daque: But yeah.
65 00:12:16.370 ⇒ 00:12:17.759 Ryan Luke Daque: what do you think.
66 00:12:19.380 ⇒ 00:12:20.759 Patrick Trainer: I think this is good.
67 00:12:29.400 ⇒ 00:12:37.200 Ryan Luke Daque: I guess, for now, though like, do we want this? Do we want to go live on this one with like the automated
68 00:12:37.260 ⇒ 00:12:40.890 Ryan Luke Daque: Code review, using chat Gpt. 4. Oh.
69 00:12:41.020 ⇒ 00:12:43.359 Ryan Luke Daque: for all our Prs or something.
70 00:12:43.850 ⇒ 00:12:46.759 Ryan Luke Daque: cause we can have this trigger on Pr
71 00:12:46.850 ⇒ 00:12:48.100 Ryan Luke Daque: right. But.
72 00:12:48.610 ⇒ 00:12:50.239 Patrick Trainer: Yeah, I think
73 00:12:50.830 ⇒ 00:12:51.760 Patrick Trainer: it.
74 00:12:53.420 ⇒ 00:12:58.639 Patrick Trainer: I think one thing that we might want to do is modify the prompt
75 00:12:58.830 ⇒ 00:13:00.060 Patrick Trainer: a bit or.
76 00:13:00.060 ⇒ 00:13:00.930 Ryan Luke Daque: Right.
77 00:13:00.930 ⇒ 00:13:07.609 Patrick Trainer: The system prompt a bit just to either make it a bit more concise or like. One thing that I noticed
78 00:13:08.170 ⇒ 00:13:09.242 Patrick Trainer: was that it?
79 00:13:09.790 ⇒ 00:13:12.260 Patrick Trainer: It seems to like loop through it twice.
80 00:13:14.060 ⇒ 00:13:19.320 Patrick Trainer: it it! When I was looking at the that pr it like.
81 00:13:20.080 ⇒ 00:13:28.739 Ryan Luke Daque: You mean the 4? 0, 1, the the there’s like I tested it twice, right, like one of the 1st one was using 3.5.
82 00:13:29.400 ⇒ 00:13:30.150 Patrick Trainer: Okay.
83 00:13:30.460 ⇒ 00:13:31.250 Ryan Luke Daque: And oh, I was.
84 00:13:31.250 ⇒ 00:13:34.066 Patrick Trainer: Gonna say, it looked like it did the same
85 00:13:36.430 ⇒ 00:13:41.970 Patrick Trainer: the same suggestion twice or the same file, like you would run through it twice. But maybe I’m just like.
86 00:13:42.250 ⇒ 00:13:44.019 Patrick Trainer: maybe I just read that wrong.
87 00:13:44.430 ⇒ 00:13:50.390 Ryan Luke Daque: Yeah, maybe that’s because, like the 1st one, it it was using. We were using the 3.5 version.
88 00:13:50.390 ⇒ 00:13:51.180 Patrick Trainer: Okay. Yeah.
89 00:13:51.509 ⇒ 00:14:00.070 Ryan Luke Daque: And then it did the code again like review again for the same file for the 4 0 version. When I tested it the second time.
90 00:14:00.330 ⇒ 00:14:01.090 Patrick Trainer: Right.
91 00:14:03.180 ⇒ 00:14:06.583 Patrick Trainer: Yeah, I think it’s I think it’s
92 00:14:07.230 ⇒ 00:14:09.166 Patrick Trainer: I mean one awesome
93 00:14:10.620 ⇒ 00:14:14.530 Patrick Trainer: But if we did want to modify the prompt at all.
94 00:14:14.530 ⇒ 00:14:15.250 Ryan Luke Daque: Yeah, I would say.
95 00:14:15.250 ⇒ 00:14:16.000 Patrick Trainer: Say.
96 00:14:16.670 ⇒ 00:14:19.650 Patrick Trainer: maybe, for like conciseness.
97 00:14:19.790 ⇒ 00:14:23.910 Patrick Trainer: just to like, not output such like a huge response.
98 00:14:24.020 ⇒ 00:14:27.564 Patrick Trainer: I don’t think like it’s not a make or break anything but
99 00:14:28.280 ⇒ 00:14:29.460 Patrick Trainer: just to.
100 00:14:29.870 ⇒ 00:14:33.850 Patrick Trainer: I don’t know. Make the review a bit more like manageable.
101 00:14:34.240 ⇒ 00:14:34.880 Ryan Luke Daque: Right.
102 00:14:35.020 ⇒ 00:14:39.279 Patrick Trainer: But then there is like the argument of like having a more complete review.
103 00:14:40.090 ⇒ 00:14:44.010 Patrick Trainer: The other part that I was thinking it may be
104 00:14:44.450 ⇒ 00:14:49.719 Patrick Trainer: instead of just having it trigger on each pull request. Just make it like a manual trigger
105 00:14:50.070 ⇒ 00:14:52.220 Patrick Trainer: that way. We
106 00:14:53.120 ⇒ 00:14:58.129 Patrick Trainer: like cause not every pull, request, or every commit like we’re going to want to like, run.
107 00:14:58.130 ⇒ 00:14:58.890 Ryan Luke Daque: Hmm.
108 00:14:58.890 ⇒ 00:15:00.490 Patrick Trainer: And so
109 00:15:01.050 ⇒ 00:15:05.798 Patrick Trainer: that gives us, like the optionality of running or not running
110 00:15:07.160 ⇒ 00:15:10.359 Patrick Trainer: just so we’re not like constantly burning through
111 00:15:10.510 ⇒ 00:15:12.520 Patrick Trainer: doing the same stuff.
112 00:15:12.790 ⇒ 00:15:13.540 Ryan Luke Daque: Yeah.
113 00:15:13.750 ⇒ 00:15:16.159 Patrick Trainer: And like, I know that.
114 00:15:16.970 ⇒ 00:15:24.739 Patrick Trainer: like at the end, like I’ll I’ll submit multiple. You submit multiple commits to an individual pull request.
115 00:15:24.970 ⇒ 00:15:25.530 Ryan Luke Daque: Right.
116 00:15:25.530 ⇒ 00:15:27.860 Patrick Trainer: Gonna want it to rerun on every single commit.
117 00:15:27.980 ⇒ 00:15:33.550 Patrick Trainer: So like, if you save those or have like checkpoints of like, okay, I actually need it
118 00:15:33.650 ⇒ 00:15:37.619 Patrick Trainer: right that like at here than to do that.
119 00:15:38.720 ⇒ 00:15:44.070 Patrick Trainer: that’s just kind of how I’m thinking about it. And then we can also like, modify the prompt into
120 00:15:44.220 ⇒ 00:15:45.660 Patrick Trainer: like a structure
121 00:15:45.700 ⇒ 00:15:49.429 Patrick Trainer: of how we want to it to respond.
122 00:15:52.380 ⇒ 00:15:56.700 Ryan Luke Daque: Yeah, that makes sense like, based on the repository and like what we are
123 00:15:56.830 ⇒ 00:16:00.169 Ryan Luke Daque: using it for, like, if it’s dbt, then it has, like.
124 00:16:00.360 ⇒ 00:16:07.469 Ryan Luke Daque: we have to modify the the prompt. So it will be like more on the sequel stuff and stuff like that
125 00:16:08.000 ⇒ 00:16:09.739 Ryan Luke Daque: or the yaml files.
126 00:16:09.820 ⇒ 00:16:12.249 Ryan Luke Daque: Yeah, so, but yeah, makes sense.
127 00:16:12.470 ⇒ 00:16:14.510 Uttam Kumaran: And we already go through the workflow part.
128 00:16:14.710 ⇒ 00:16:16.399 Uttam Kumaran: Sorry, I know. I joined late.
129 00:16:16.910 ⇒ 00:16:20.350 Ryan Luke Daque: No problem. I think, yeah, I can go through it
130 00:16:20.920 ⇒ 00:16:21.990 Ryan Luke Daque: again.
131 00:16:22.270 ⇒ 00:16:28.310 Ryan Luke Daque: So I did create this notion notion document on like the workflow. Basically.
132 00:16:28.770 ⇒ 00:16:31.399 Ryan Luke Daque: yeah, so yeah, there’s like
133 00:16:31.900 ⇒ 00:16:35.349 Ryan Luke Daque: 6 steps here. But essentially, we are just
134 00:16:35.880 ⇒ 00:16:38.660 Ryan Luke Daque: using the Github action workflow.
135 00:16:40.610 ⇒ 00:16:45.170 Ryan Luke Daque: and this is currently based on how the repository structured.
136 00:16:45.845 ⇒ 00:16:47.179 Ryan Luke Daque: The one that
137 00:16:49.010 ⇒ 00:16:50.989 Ryan Luke Daque: bit too mad created right.
138 00:16:50.990 ⇒ 00:16:51.700 Uttam Kumaran: Yeah.
139 00:16:51.700 ⇒ 00:16:55.103 Ryan Luke Daque: So yeah, all the variables here, like,
140 00:16:56.890 ⇒ 00:16:57.490 Uttam Kumaran: Oh, okay.
141 00:16:57.490 ⇒ 00:17:03.630 Ryan Luke Daque: These stuff we can. We can always change like the engine, or like
142 00:17:03.750 ⇒ 00:17:06.299 Ryan Luke Daque: the Max tokens and and stuff like that.
143 00:17:06.589 ⇒ 00:17:12.600 Ryan Luke Daque: But currently using this, we we won’t be able to change like what the
144 00:17:14.150 ⇒ 00:17:16.039 Ryan Luke Daque: what the prompt is.
145 00:17:16.069 ⇒ 00:17:18.930 Ryan Luke Daque: So if if we want it to be more
146 00:17:20.790 ⇒ 00:17:21.550 Ryan Luke Daque: like
147 00:17:22.210 ⇒ 00:17:23.760 Ryan Luke Daque: using our own.
148 00:17:23.760 ⇒ 00:17:27.589 Uttam Kumaran: Oh, I see, I see. I see. Yeah, we should just fork the whole thing.
149 00:17:27.599 ⇒ 00:17:29.559 Ryan Luke Daque: Right. You can just fork this.
150 00:17:29.950 ⇒ 00:17:30.440 Uttam Kumaran: Yeah, we could.
151 00:17:30.440 ⇒ 00:17:31.609 Ryan Luke Daque: Probably just whoop.
152 00:17:31.610 ⇒ 00:17:40.929 Uttam Kumaran: Yeah, exactly like we’ll fork the whole thing. And then, how? So how is this? Okay? So it’s getting called like, externally, you’re just basically importing you’re calling this action from is.
153 00:17:41.560 ⇒ 00:17:42.250 Ryan Luke Daque: Right.
154 00:17:42.250 ⇒ 00:17:44.650 Uttam Kumaran: So, yeah, we would just have our own internal action.
155 00:17:46.010 ⇒ 00:17:49.041 Uttam Kumaran: And so you said, we’re doing it for
156 00:17:49.640 ⇒ 00:17:58.679 Uttam Kumaran: Yes, I agree with pat in that it should be something more on Trigger, I think. Record having it required on the first.st
157 00:17:58.850 ⇒ 00:18:04.220 Uttam Kumaran: When you open a pr is fine. I also like.
158 00:18:04.270 ⇒ 00:18:08.179 Uttam Kumaran: I think this is gonna go into more pat, like having really clear
159 00:18:08.250 ⇒ 00:18:15.769 Uttam Kumaran: like Pr messages and issues attached to Prs, otherwise, it’s gonna have, like, basically no context on like what the goal is.
160 00:18:16.228 ⇒ 00:18:20.709 Uttam Kumaran: So I actually. And this is where Miguel, I think we’re basically gonna have, like
161 00:18:20.790 ⇒ 00:18:41.470 Uttam Kumaran: some sort of steps here, whereas, like, if it doesn’t have, if it doesn’t have like, what is a Pr, I want us to think so. I’m like my mind’s kind of going crazy. I think if I want this to basically emulate, like what an actual pr reviewer does right? And what do you look for? You look for effective titles, you look for, you know, really good pr descriptions.
162 00:18:41.510 ⇒ 00:18:57.969 Uttam Kumaran: You look to reference the issue that was trying to be solved. And then you look at the code right? So I want to kind of break this down step by step, into what a Pr review process is, and the more well defined, that is, for us the better. This, the A, this agent, this process, is gonna work
163 00:18:58.090 ⇒ 00:19:05.009 Uttam Kumaran: right? So I think that’s 1 clear thing we can do is think about what are the different review steps and in what order that needs to go
164 00:19:05.080 ⇒ 00:19:08.630 Uttam Kumaran: right. And the nice thing about Pr view is what this will start.
165 00:19:08.940 ⇒ 00:19:26.970 Uttam Kumaran: Maybe it’s more like, Hey, fix this, fix this. But as we learn how to fix more of these, we’ll then move to an area where the agent itself actually does those things for you. For example, if you already have access to the issue and the code change, then you can probably write the the Pr title better. You can probably already write the Pr. Description pretty good.
166 00:19:27.100 ⇒ 00:19:33.170 Uttam Kumaran: right? So this will start more as like feedback to the developer and then evolve into actually handling some of that stuff.
167 00:19:33.520 ⇒ 00:19:38.469 Uttam Kumaran: So the so I will kind of want to break this down into what are the different
168 00:19:38.600 ⇒ 00:19:40.190 Uttam Kumaran: parts of the review
169 00:19:40.280 ⇒ 00:19:42.390 Uttam Kumaran: and attack each of those
170 00:19:42.440 ⇒ 00:19:44.239 Uttam Kumaran: you know isolated.
171 00:19:44.766 ⇒ 00:19:56.950 Uttam Kumaran: isolated meeting kind of the way these, like AI things work is. You just have prompts for each different area. And we will call the a variety of prompts.
172 00:19:56.980 ⇒ 00:19:58.620 Uttam Kumaran: you know, based on what
173 00:19:58.680 ⇒ 00:20:00.770 Uttam Kumaran: just a specific thing we’re targeting
174 00:20:00.900 ⇒ 00:20:07.610 Uttam Kumaran: like you don’t like for effective. We will have like guidelines on what is an effective Pr title stuff like that.
175 00:20:07.630 ⇒ 00:20:10.394 Uttam Kumaran: But addition, like we’re we’re doing things about
176 00:20:10.870 ⇒ 00:20:15.079 Uttam Kumaran: you know, code quality and stuff like that where we we’re trying to leverage SQL, Fluff.
177 00:20:15.190 ⇒ 00:20:25.919 Uttam Kumaran: sequel. Fluff will help here for a bit. But there’s gonna be some stuff where? I think it’s going to be way easier just to use this to basically do like basic code maintenance and quality.
178 00:20:27.340 ⇒ 00:20:33.139 Uttam Kumaran: but I think the toughest part of all those 3 that I mentioned is going to be actually giving guidance on the code.
179 00:20:33.657 ⇒ 00:20:41.520 Uttam Kumaran: Which is where you’re gonna you may need context on the repo or parts of the repo. You may need context on the issue.
180 00:20:43.080 ⇒ 00:20:43.715 Uttam Kumaran: and
181 00:20:44.670 ⇒ 00:20:48.795 Uttam Kumaran: we, that’s gonna be the part. I think that really moved the needle for us.
182 00:20:50.450 ⇒ 00:20:59.579 Ryan Luke Daque: Yeah, I definitely agree. Like, currently, this is very generic, like how it’s doing the review, right? It’s like looking at all the files
183 00:21:00.020 ⇒ 00:21:01.949 Ryan Luke Daque: that were changed, and then.
184 00:21:02.810 ⇒ 00:21:06.440 Ryan Luke Daque: like giving feedback on that specific file.
185 00:21:07.040 ⇒ 00:21:07.660 Uttam Kumaran: Yeah.
186 00:21:07.980 ⇒ 00:21:08.730 Ryan Luke Daque: Yeah.
187 00:21:08.960 ⇒ 00:21:10.900 Ryan Luke Daque: And then very
188 00:21:11.390 ⇒ 00:21:12.130 Ryan Luke Daque: like.
189 00:21:12.710 ⇒ 00:21:16.449 Ryan Luke Daque: since it’s a generic. Prompt. It’s also like getting giving.
190 00:21:16.450 ⇒ 00:21:19.999 Uttam Kumaran: Yeah, we’ll we will make this purely focused on
191 00:21:20.560 ⇒ 00:21:24.440 Uttam Kumaran: right cause. You can. You can tell that we’ll have different types of review
192 00:21:24.490 ⇒ 00:21:35.339 Uttam Kumaran: bots, or whatever right, some stuff that’s focused on data models, some stuff that’s focused on python. Our ability to like basically atomize this problem helps us make this like way more flexible.
193 00:21:35.740 ⇒ 00:21:38.340 Uttam Kumaran: So the one thing I think we should think about
194 00:21:40.020 ⇒ 00:21:45.020 Uttam Kumaran: from like our engineering perspective, is like, what do we expect in a great Pr review.
195 00:21:45.070 ⇒ 00:21:47.959 Uttam Kumaran: and then the task is to replicate that.
196 00:21:48.570 ⇒ 00:21:49.910 Ryan Luke Daque: Right, right.
197 00:21:49.910 ⇒ 00:21:50.910 Uttam Kumaran: You know
198 00:21:53.810 ⇒ 00:22:00.700 Uttam Kumaran: but otherwise, like I mean, I’m glad it works really well. I think we definitely should fork it into our own, and then we should think about how we
199 00:22:01.160 ⇒ 00:22:07.089 Uttam Kumaran: basically atomize this problem into a couple of key things. The other thing is like, I
200 00:22:07.320 ⇒ 00:22:09.178 Uttam Kumaran: I wonder how we can.
201 00:22:10.150 ⇒ 00:22:16.079 Uttam Kumaran: yeah, do the re review process like, let’s say you get the initial review, you make a change
202 00:22:16.400 ⇒ 00:22:24.789 Uttam Kumaran: like, I don’t know how creative we want to get. Because, yeah, I don’t want this to be on every commit I almost want to be, you know. I almost would prefer it’s like
203 00:22:25.260 ⇒ 00:22:32.990 Uttam Kumaran: it gets triggered when you request the user. Like, there is a brain forward. Reviewer is like an actual use like service account user. Right?
204 00:22:33.695 ⇒ 00:22:39.050 Uttam Kumaran: I would prefer. I don’t know what the mechanics of that are, but like that seems like the best form factor.
205 00:22:39.720 ⇒ 00:22:45.330 Uttam Kumaran: Because, for example, you can request that reviewer can be on every Pr, and you can have a human reviewer.
206 00:22:45.410 ⇒ 00:22:49.099 Uttam Kumaran: And then maybe some stuff. It’s just goes to that. And it gets approved.
207 00:22:50.400 ⇒ 00:22:55.970 Uttam Kumaran: because you also want an approval like you want an approval flow. If the agent is like this looks pretty good.
208 00:22:56.480 ⇒ 00:22:59.340 Uttam Kumaran: and then it should. It should like check mark and be like, you’re good to go
209 00:22:59.410 ⇒ 00:23:04.923 Uttam Kumaran: right like that’s you want, like, I think the problem with a lot of AI agents now is
210 00:23:05.240 ⇒ 00:23:10.699 Uttam Kumaran: They don’t go far enough to replicating the actual like. What we would do on a Pr review.
211 00:23:10.920 ⇒ 00:23:17.580 Uttam Kumaran: Right? Like leaving a comment is one thing, but it should be like comment. And that’s a blocking that’s like a blocking problem.
212 00:23:17.790 ⇒ 00:23:19.529 Uttam Kumaran: Or that’s like a common. But
213 00:23:19.550 ⇒ 00:23:20.830 Uttam Kumaran: it’s still approved
214 00:23:21.120 ⇒ 00:23:25.799 Uttam Kumaran: right? So that those are the things that that seem like really great opportunities for us.
215 00:23:26.610 ⇒ 00:23:29.210 Uttam Kumaran: I don’t know, Brian. What do you think?
216 00:23:29.690 ⇒ 00:23:31.180 Uttam Kumaran: Or or Miguel?
217 00:23:37.270 ⇒ 00:23:39.460 Uttam Kumaran: You guys are on mute if you’re talking.
218 00:23:39.890 ⇒ 00:23:40.843 joshuadeveyra: Oh, yeah,
219 00:23:42.550 ⇒ 00:23:49.680 joshuadeveyra: I I would approach similarly to how we approached Hpi. To be honest, like, as you mentioned, you know, atomize it.
220 00:23:53.030 ⇒ 00:23:54.132 Uttam Kumaran: Do you think,
221 00:23:54.800 ⇒ 00:24:01.659 Uttam Kumaran: Miguel? We could do this easily like, instead of calling one prompt like, would we use an assistant?
222 00:24:01.680 ⇒ 00:24:08.469 Uttam Kumaran: Because this isn’t. Gonna we’re. I don’t think it’s necessary here to go through relevance as like cause we don’t need any front end.
223 00:24:08.860 ⇒ 00:24:10.379 Uttam Kumaran: So we like.
224 00:24:10.430 ⇒ 00:24:16.339 Uttam Kumaran: But I don’t have much experience doing like using open AI assistance. Is that what we would try to do.
225 00:24:17.715 ⇒ 00:24:19.500 joshuadeveyra: Yeah, the only
226 00:24:19.520 ⇒ 00:24:24.699 joshuadeveyra: thing I think the the only issue will be like, how will we get the data to the assistant?
227 00:24:25.270 ⇒ 00:24:26.850 joshuadeveyra: And if it’s like multiple.
228 00:24:26.850 ⇒ 00:24:27.170 Uttam Kumaran: Agents.
229 00:24:27.170 ⇒ 00:24:32.809 joshuadeveyra: It has to be relevance, because, as Openai assistance doesn’t support multiple agents, yet.
230 00:24:33.530 ⇒ 00:24:36.360 Uttam Kumaran: So this would be called from the workflow.
231 00:24:36.400 ⇒ 00:24:39.450 Uttam Kumaran: The workflow actually passes it via an Api call.
232 00:24:40.230 ⇒ 00:24:42.709 Uttam Kumaran: So see that there’s that python code that
233 00:24:43.660 ⇒ 00:24:46.850 Uttam Kumaran: Brian went through that python code actually has
234 00:24:46.870 ⇒ 00:24:50.789 Uttam Kumaran: has a has an Api call to open AI
235 00:24:50.800 ⇒ 00:24:52.470 Uttam Kumaran: that passes in the prompt.
236 00:24:54.710 ⇒ 00:24:56.285 Ryan Luke Daque: Oh, yeah, this. This one.
237 00:25:01.740 ⇒ 00:25:04.369 Uttam Kumaran: So if if you wanted to do this via
238 00:25:04.410 ⇒ 00:25:09.280 Uttam Kumaran: relevance, and we would just call the relevance, we would, whatever web, hook, or whatever we would do.
239 00:25:09.630 ⇒ 00:25:11.670 joshuadeveyra: Yeah. Yeah.
240 00:25:11.850 ⇒ 00:25:13.749 joshuadeveyra: I mean, we could create the
241 00:25:13.860 ⇒ 00:25:14.680 joshuadeveyra: wait.
242 00:25:15.510 ⇒ 00:25:17.409 joshuadeveyra: Yeah, we don’t need Zapier for this.
243 00:25:19.170 ⇒ 00:25:27.809 joshuadeveyra: like, my only concern would be like, for example, we create like a Pr, right? How will we get that data? But it seems like we already have that down.
244 00:25:28.560 ⇒ 00:25:29.320 joshuadeveyra: Yeah.
245 00:25:29.320 ⇒ 00:25:30.290 Uttam Kumaran: That’s fine.
246 00:25:31.920 ⇒ 00:25:38.409 joshuadeveyra: And then from there, you know. It just all depends on like what the guidelines are, what our guidelines are.
247 00:25:44.730 ⇒ 00:25:47.230 Uttam Kumaran: Okay, cool, I guess, from from the
248 00:25:47.330 ⇒ 00:25:48.390 Uttam Kumaran: like
249 00:25:48.520 ⇒ 00:25:54.158 Uttam Kumaran: other aes folks in the call, I guess. Pat or Brian like, what do you think about
250 00:25:55.910 ⇒ 00:25:58.460 Uttam Kumaran: like, what are your thoughts on, like what we could
251 00:25:58.540 ⇒ 00:26:04.279 Uttam Kumaran: potentially like review and fix versus like? What may be more hard to do in terms of the actual code itself.
252 00:26:04.770 ⇒ 00:26:07.520 Uttam Kumaran: Cause, I feel like I’m confident that we can get
253 00:26:07.720 ⇒ 00:26:13.519 Uttam Kumaran: the form factor fine, but wondering, like what what kind of thing that this would catch versus
254 00:26:14.130 ⇒ 00:26:16.080 Uttam Kumaran: what we may have difficulty with.
255 00:26:22.040 ⇒ 00:26:23.600 Patrick Trainer: One thing
256 00:26:23.610 ⇒ 00:26:27.519 Patrick Trainer: might be like we want might wanna limit the
257 00:26:28.240 ⇒ 00:26:31.510 Patrick Trainer: review to just the diffs
258 00:26:32.260 ⇒ 00:26:36.980 Patrick Trainer: of like between the commits and the the head branch rather than.
259 00:26:37.160 ⇒ 00:26:37.780 Ryan Luke Daque: Like a whole.
260 00:26:37.780 ⇒ 00:26:41.960 Patrick Trainer: Entire file. Yeah, because, like, there’s gonna be a lot of
261 00:26:42.780 ⇒ 00:26:47.609 Patrick Trainer: like, it’s gonna waste cycles on stuff that’s either already been reviewed
262 00:26:47.670 ⇒ 00:26:53.559 Patrick Trainer: or like isn’t necessarily relevant to the change itself.
263 00:26:56.670 ⇒ 00:27:03.120 Patrick Trainer: and like, yeah, like, if you were to make a commit, and then it comments on stuff that
264 00:27:04.350 ⇒ 00:27:07.240 Patrick Trainer: like I wrote a month ago like that doesn’t.
265 00:27:07.240 ⇒ 00:27:07.890 Ryan Luke Daque: Yeah, it’s not.
266 00:27:07.890 ⇒ 00:27:15.939 Patrick Trainer: Relevant to like what’s going on. So yeah, we’d wanna we’d wanna probably look at just the diffs between the the commit and the the head branch. There.
267 00:27:15.940 ⇒ 00:27:19.239 Ryan Luke Daque: What if it’s a new file, though, like.
268 00:27:19.240 ⇒ 00:27:21.179 Patrick Trainer: I mean, if it’s net new file. Yeah, that.
269 00:27:21.180 ⇒ 00:27:21.590 Ryan Luke Daque: Right.
270 00:27:21.590 ⇒ 00:27:24.032 Patrick Trainer: That’s a that’s still a diff.
271 00:27:24.440 ⇒ 00:27:25.210 Uttam Kumaran: Yeah.
272 00:27:25.630 ⇒ 00:27:29.890 Patrick Trainer: Yeah, we’d we’d want to limit it there in. That
273 00:27:30.090 ⇒ 00:27:36.430 Patrick Trainer: is probably just is going to involve. I mean, I believe all of this is going to involve forking the
274 00:27:36.570 ⇒ 00:27:39.880 Patrick Trainer: the various action in modifying that.
275 00:27:40.400 ⇒ 00:27:44.930 Patrick Trainer: So I think, like a good starting point, would be like.
276 00:27:45.400 ⇒ 00:27:51.269 Patrick Trainer: let’s let’s fork it into Brainforge. And then let’s get like.
277 00:27:52.150 ⇒ 00:27:56.269 Patrick Trainer: basically like a board together of like features
278 00:27:56.400 ⇒ 00:27:57.640 Patrick Trainer: that we want
279 00:27:58.225 ⇒ 00:28:02.380 Patrick Trainer: and ideas and like, put these all down and then
280 00:28:03.741 ⇒ 00:28:07.590 Patrick Trainer: and then that actually like would fit into
281 00:28:07.820 ⇒ 00:28:09.899 Patrick Trainer: like a normal sprint cycle
282 00:28:10.250 ⇒ 00:28:11.899 Patrick Trainer: to where we can like.
283 00:28:11.900 ⇒ 00:28:12.190 Uttam Kumaran: Yeah.
284 00:28:12.190 ⇒ 00:28:14.289 Patrick Trainer: Knock out this feature, knock out that.
285 00:28:14.330 ⇒ 00:28:16.109 Patrick Trainer: do this, and then.
286 00:28:16.250 ⇒ 00:28:18.589 Patrick Trainer: and then continue to iterate on it.
287 00:28:19.960 ⇒ 00:28:22.109 Patrick Trainer: I think like what
288 00:28:22.530 ⇒ 00:28:26.640 Patrick Trainer: Ryan’s already done. This is like perfect, like
289 00:28:26.720 ⇒ 00:28:31.149 Patrick Trainer: poc. And this is like the thumbs up to like, yeah, let’s.
290 00:28:31.340 ⇒ 00:28:31.690 Uttam Kumaran: Yeah.
291 00:28:31.690 ⇒ 00:28:33.689 Patrick Trainer: Go. Let’s go down the rabbit hole
292 00:28:33.900 ⇒ 00:28:35.280 Patrick Trainer: and
293 00:28:36.850 ⇒ 00:28:38.770 Patrick Trainer: productize this. Basically.
294 00:28:42.710 ⇒ 00:28:45.379 Patrick Trainer: yeah, yeah, those are my thoughts.
295 00:28:49.470 ⇒ 00:28:51.750 Uttam Kumaran: I’m gonna call him Brian, cause he’s been quiet
296 00:28:51.770 ⇒ 00:28:56.769 Uttam Kumaran: because I and also because I texted him was like, Are you alive? And he’s like he’s like, Yeah, I’m alive.
297 00:28:56.890 ⇒ 00:29:01.819 Uttam Kumaran: But he’s he said. He doesn’t use Chat Gbt in real life. But I guess my my question is more about like.
298 00:29:02.170 ⇒ 00:29:04.850 Uttam Kumaran: does this seem like this would
299 00:29:05.500 ⇒ 00:29:09.439 Uttam Kumaran: like? Does this seem possible? Or if also, I’m like as
300 00:29:09.780 ⇒ 00:29:21.279 Uttam Kumaran: I like picking on people, because, especially if you don’t know what AI can do. That’s great, because I want to know, like, what would be magic if this did, that, you would be like, Okay, shit. This is like crazy
301 00:29:21.570 ⇒ 00:29:27.909 Uttam Kumaran: like looking for commas and stuff is one thing. But if there’s 1 thing this this AI agent could do, that you’d be like
302 00:29:28.690 ⇒ 00:29:33.020 Uttam Kumaran: no way or like this like, really saved me a ton of time like what would it be.
303 00:29:46.520 ⇒ 00:29:47.690 Uttam Kumaran: It’s a hard question.
304 00:29:48.070 ⇒ 00:29:49.510 Uttam Kumaran: I know we already went through a bunch of stuff.
305 00:29:51.350 ⇒ 00:29:55.222 Brian Pei: I don’t use AI or chat gpt like you, said
306 00:29:55.840 ⇒ 00:30:01.110 Brian Pei: and this is, I’m separating this from just like a SQL. Inter, which
307 00:30:01.330 ⇒ 00:30:04.784 Brian Pei: usually solves some of my problems.
308 00:30:05.650 ⇒ 00:30:08.579 Brian Pei: I just I don’t. I haven’t seen
309 00:30:08.760 ⇒ 00:30:11.900 Brian Pei: an example output. I feel like I need to see
310 00:30:12.360 ⇒ 00:30:14.359 Brian Pei: on a Pr of what this
311 00:30:14.540 ⇒ 00:30:15.859 Brian Pei: will do
312 00:30:15.950 ⇒ 00:30:17.819 Brian Pei: to get ideas.
313 00:30:18.390 ⇒ 00:30:20.359 Brian Pei: I was kind of just looking at Code. There.
314 00:30:21.090 ⇒ 00:30:22.656 Uttam Kumaran: Yeah let me pull up
315 00:30:24.310 ⇒ 00:30:28.050 Nicolas Sucari: The output should be like a comment in the Pr. With some like
316 00:30:28.600 ⇒ 00:30:32.179 Nicolas Sucari: comments on the different changes of that
317 00:30:32.220 ⇒ 00:30:33.600 Nicolas Sucari: comments right.
318 00:30:34.940 ⇒ 00:30:36.850 Uttam Kumaran: Yeah, let me. I’m gonna pull up
319 00:30:37.230 ⇒ 00:30:38.589 Uttam Kumaran: an example.
320 00:30:39.740 ⇒ 00:30:44.449 Uttam Kumaran: And, Brian, you can. I think, like, let’s say this because it’s a good question.
321 00:30:48.290 ⇒ 00:30:58.150 Uttam Kumaran: so this is like, this is, this is all like 1st proof, proof of concept. But basically this Pr has, like some changes in it. This is from like a while ago. And this is like
322 00:30:58.510 ⇒ 00:31:00.820 Uttam Kumaran: in this Pr in this pr.
323 00:31:00.880 ⇒ 00:31:03.219 Uttam Kumaran: the Github action is actually commenting
324 00:31:03.390 ⇒ 00:31:04.790 Uttam Kumaran: suggestions
325 00:31:04.860 ⇒ 00:31:07.679 Uttam Kumaran: for every single file on like what to change.
326 00:31:08.190 ⇒ 00:31:09.270 Uttam Kumaran: So
327 00:31:09.640 ⇒ 00:31:13.719 Uttam Kumaran: I don’t know. Ryan. Is this a good example? Should I keep scrolling down? Is there like a better one?
328 00:31:14.766 ⇒ 00:31:24.440 Ryan Luke Daque: Yeah. The the 1st tests were like using the 3.5 version of chat gpt, so that’s why it’s like pretty short and and stuff. And like the the newer ones like this one.
329 00:31:24.540 ⇒ 00:31:30.940 Ryan Luke Daque: it’s using the 4 0 version. And it’s pretty detailed and like it’s checking everything basically.
330 00:31:31.750 ⇒ 00:31:35.705 Uttam Kumaran: So, Brian, take a look at this. So imagine this, said my name.
331 00:31:36.140 ⇒ 00:31:37.979 Uttam Kumaran: and I gave this review.
332 00:31:38.730 ⇒ 00:31:41.530 Uttam Kumaran: I would say, this is pretty detailed.
333 00:31:42.020 ⇒ 00:31:45.369 Uttam Kumaran: but I guess give me your 1st reaction. I don’t know
334 00:31:45.390 ⇒ 00:31:46.990 Uttam Kumaran: but this is like an example.
335 00:31:48.280 ⇒ 00:31:50.670 Brian Pei: 1st reaction is.
336 00:31:51.140 ⇒ 00:31:54.630 Brian Pei: if I have a big Pr, and
337 00:31:55.220 ⇒ 00:32:02.370 Brian Pei: I’m the let’s say I’m the only contractor working on a client, so I don’t have somebody who can actually do peer review.
338 00:32:03.425 ⇒ 00:32:04.969 Brian Pei: This is great
339 00:32:06.680 ⇒ 00:32:07.810 Brian Pei: for
340 00:32:07.920 ⇒ 00:32:13.079 Brian Pei: for quick fixes I pro like, if I’m doing one or 2 line changes.
341 00:32:13.190 ⇒ 00:32:15.290 Brian Pei: I probably wouldn’t read this
342 00:32:15.560 ⇒ 00:32:16.460 Brian Pei: but
343 00:32:17.100 ⇒ 00:32:19.856 Brian Pei: for, let’s say this, this coffee client
344 00:32:21.100 ⇒ 00:32:26.779 Brian Pei: when I have like 15 new Dbt models in one Pr
345 00:32:28.420 ⇒ 00:32:31.160 Brian Pei: code formatting code. Formatting is great.
346 00:32:31.510 ⇒ 00:32:33.030 Brian Pei: Air handling is great.
347 00:32:34.600 ⇒ 00:32:37.070 Brian Pei: I mean, yeah, this is all stuff that I guess I would ask
348 00:32:37.120 ⇒ 00:32:44.009 Brian Pei: a peer, a human being peer reviewer to look at for me, so it’s nicely summarized
349 00:32:44.980 ⇒ 00:32:46.879 Brian Pei: would use thumbs up
350 00:32:47.680 ⇒ 00:32:49.669 Brian Pei: would not use, for
351 00:32:50.130 ⇒ 00:32:52.260 Brian Pei: it’s Friday night, and the client says.
352 00:32:52.260 ⇒ 00:32:52.940 Uttam Kumaran: Yeah. Ordered.
353 00:32:52.940 ⇒ 00:32:55.670 Brian Pei: Imagine I gotta push something really quick, and I’m just gonna merge it.
354 00:32:56.250 ⇒ 00:32:58.330 Uttam Kumaran: And that’s exactly like
355 00:32:58.670 ⇒ 00:33:02.450 Uttam Kumaran: what I’m talking about is that we? And you know, I think
356 00:33:03.520 ⇒ 00:33:11.530 Uttam Kumaran: I was talking to someone about this is within data. Our slas are a lot quicker than in software, sometimes.
357 00:33:11.590 ⇒ 00:33:14.740 Uttam Kumaran: you know. And so we need to make sure that this isn’t.
358 00:33:15.270 ⇒ 00:33:21.330 Uttam Kumaran: I mean, similar to how it’s like when we had elementary. And it was yelling all the time very similar problem, where it’s like, you can’t cry Wolf, at everything.
359 00:33:21.390 ⇒ 00:33:39.280 Uttam Kumaran: But also, the nice thing is we can have those expectations right? If if it’s a 1, if it’s like a 1 line, change then, and the chat, and then, but also what you what you expect is that. Imagine, Brian, you could get reviews on like 2 line changes, and all the review could say is like, you’re good.
360 00:33:39.450 ⇒ 00:33:58.799 Uttam Kumaran: Right? That’s actually what I want to think about is like there are gonna be opportunities where we may have not got a review, or what you said, which is like you’re the only one on the client. None of us have, like great context on everything that’s going on. And so for us to provide a review. It’s maybe like, it’s like way too much to ask for.
361 00:33:59.160 ⇒ 00:34:00.650 Uttam Kumaran: This is where I think.
362 00:34:00.920 ⇒ 00:34:04.400 Uttam Kumaran: having something like this. Running is actually super helpful.
363 00:34:04.760 ⇒ 00:34:13.400 Uttam Kumaran: The other thing is in. And I think this is going to be what we have to figure out. Miguel is like, how does how do we provide the context of the repository.
364 00:34:13.830 ⇒ 00:34:15.840 Uttam Kumaran: I think, providing the diff
365 00:34:15.980 ⇒ 00:34:21.969 Uttam Kumaran: is a good start. But I want to stretch us to figure out, how do we actually give
366 00:34:22.110 ⇒ 00:34:25.329 Uttam Kumaran: the entire repository context
367 00:34:25.944 ⇒ 00:34:30.909 Uttam Kumaran: and make that available for the agent to consider. And when giving the review.
368 00:34:32.949 ⇒ 00:34:35.260 Uttam Kumaran: I think there are ways to do this.
369 00:34:35.757 ⇒ 00:34:41.210 Uttam Kumaran: And I don’t think this all needs to happen in in context, like, I don’t think this is like passing it every time.
370 00:34:41.730 ⇒ 00:34:50.570 Uttam Kumaran: but I want that to be our like stretch goal. Actually, in that, when we build the agent for a repo, it has context of everything.
371 00:34:50.679 ⇒ 00:35:05.679 Uttam Kumaran: The other thing that we’ll start to layer in is even business context right? Imagine the agent who’s reviewing not only knows what good sequel looks like, what Snowflake is, but it also knows, like what the clients said in the most recent meeting.
372 00:35:05.770 ⇒ 00:35:15.299 Uttam Kumaran: and what that, how, how that has any, how, that has relevance with the code that’s being pushed right like. That’s how far I want us to stretch on something like this.
373 00:35:15.653 ⇒ 00:35:21.460 Uttam Kumaran: And so this goes into kind of like what me and Miguel are already trying to work on, which is like a hub for every client.
374 00:35:21.670 ⇒ 00:35:27.520 Uttam Kumaran: But that hub not only includes the meetings that we have, but includes the code. We’re writing internal conversations.
375 00:35:29.190 ⇒ 00:35:35.640 Uttam Kumaran: and so I think this is a great start. I think we’re probably months away from what I just said.
376 00:35:35.960 ⇒ 00:35:42.030 Uttam Kumaran: But I think, even in probably a day’s worth of changes, we can modify this to be super focused on SQL.
377 00:35:42.090 ⇒ 00:35:45.689 Uttam Kumaran: To tackle all of this in one message.
378 00:35:46.360 ⇒ 00:35:51.062 Uttam Kumaran: to actually like, for example, doing things like, Oh, you should.
379 00:35:51.930 ⇒ 00:36:01.440 Uttam Kumaran: you should put these into constants like that’s not. That’s not really gonna be a huge problem for us. So some of these are nice to have. Some of these are like required changes.
380 00:36:02.950 ⇒ 00:36:07.630 Uttam Kumaran: and those are things I think I want to get over the hump of Brian’s like.
381 00:36:08.020 ⇒ 00:36:13.770 Uttam Kumaran: Some of these are good. Some of these are not good. I want it to go from. Some of these are good, some of these not good, like all these are great suggestions.
382 00:36:14.150 ⇒ 00:36:19.729 Uttam Kumaran: And then I wanted to go to like holy shit like it basically found something that I had no idea about
383 00:36:20.246 ⇒ 00:36:24.540 Uttam Kumaran: and then the next holy shit moment is like it, writes the Pr, right?
384 00:36:24.877 ⇒ 00:36:42.710 Uttam Kumaran: So those are the those are kind of the the levels I want this to go for. But I think, even in this conversation, seeing how this could do reviews for very small prs, that maybe we would have no review process on before, and this can do it in seconds. And catch things like commas, or like random small things.
385 00:36:42.720 ⇒ 00:36:46.496 Uttam Kumaran: and then also for big for Pr, where we don’t have
386 00:36:46.910 ⇒ 00:36:49.339 Uttam Kumaran: We don’t have a what do you call it?
387 00:36:50.350 ⇒ 00:36:55.359 Uttam Kumaran: What is it called when you have like. We don’t have like. And it’s not contingency, but almost like,
388 00:36:56.430 ⇒ 00:37:03.230 Uttam Kumaran: you have, you have overlap. You have multiple people working on the same stuff where someone can give a review. We have this agent do that right? So I think that’s great.
389 00:37:05.330 ⇒ 00:37:08.636 Uttam Kumaran: So I think we have like enough takeaways
390 00:37:09.370 ⇒ 00:37:11.579 Uttam Kumaran: to kind of like start on some stuff.
391 00:37:12.040 ⇒ 00:37:15.830 Uttam Kumaran: I think we could take the transcript from this meeting and basically summarize that and
392 00:37:16.180 ⇒ 00:37:19.060 Uttam Kumaran: think about the roadmap for this project.
393 00:37:19.200 ⇒ 00:37:22.669 Uttam Kumaran: I think tomorrow, Pat, when we talk, we could talk about this underneath
394 00:37:22.780 ⇒ 00:37:32.490 Uttam Kumaran: all that stuff, because I kind of want you to guide that. And then we can think about like, how do we want to dedicate time towards this? But this is sick. I love this.
395 00:37:32.860 ⇒ 00:37:34.430 Patrick Trainer: Yeah, I’m pretty pumped about this.
396 00:37:34.924 ⇒ 00:37:43.330 Ryan Luke Daque: This Pr, even this reviewer even has suggested improvements to the code which it already created. The
397 00:37:43.540 ⇒ 00:37:47.110 Ryan Luke Daque: improved code, basically removing the duplicates.
398 00:37:47.450 ⇒ 00:37:48.640 Ryan Luke Daque: She’s pretty
399 00:37:48.690 ⇒ 00:37:50.390 Ryan Luke Daque: crazy if you ask me.
400 00:37:50.970 ⇒ 00:37:51.620 Uttam Kumaran: Yeah.
401 00:37:52.790 ⇒ 00:37:56.840 Uttam Kumaran: I mean, and to give you a suggestion to give you a feeling of like how this
402 00:37:57.280 ⇒ 00:38:00.390 Uttam Kumaran: right now, there are companies, their entire, like
403 00:38:00.670 ⇒ 00:38:08.570 Uttam Kumaran: AI companies that are built on providing us with just this thing. And we did it in like a Github action, right? So like
404 00:38:08.900 ⇒ 00:38:13.749 Uttam Kumaran: the stuff that we’re able to do, and just very simply is super available to us
405 00:38:13.850 ⇒ 00:38:19.059 Uttam Kumaran: and the quality of life improvements that we’re gonna have for anybody at Brainforge that does engineering.
406 00:38:19.240 ⇒ 00:38:21.109 Uttam Kumaran: I think it’s gonna be tremendous.
407 00:38:21.550 ⇒ 00:38:40.229 Uttam Kumaran: The last thing I’ll say is, not everyone at the company that comes on is gonna be at the level of understanding that, like, even in the level of skill set of me, Pat Ryan, and Brian, in terms of sequel, right? And our job is to get them to that level.
408 00:38:40.270 ⇒ 00:38:46.170 Uttam Kumaran: And I think tools like this really help right? Like, if we have people that are junior working on Prs.
409 00:38:47.630 ⇒ 00:38:54.649 Uttam Kumaran: going through and getting an approval from the the Gpt Bot 1st is a great step for them to work, Async
410 00:38:55.066 ⇒ 00:38:58.619 Uttam Kumaran: and it’s a great way for them to level up right. And
411 00:38:58.640 ⇒ 00:39:12.379 Uttam Kumaran: we’re slowly gonna become a factory for like great engineers. And I think this is a great way of of in in the actual process, allowing them assistance where we’re not gonna have to scale us right? And I think
412 00:39:12.480 ⇒ 00:39:21.040 Uttam Kumaran: what we found is constantly that like finding people like us is gonna continue to be difficult. So our goal is gonna be to level people up. And I think this is a great way of
413 00:39:21.070 ⇒ 00:39:26.440 Uttam Kumaran: codifying our understanding of what great code is into an agent like this, and having it run these Pr cycles.
414 00:39:28.700 ⇒ 00:39:33.120 Uttam Kumaran: cool. Anything. I know we’re a little bit over time anything else we wanted to chat about.
415 00:39:34.000 ⇒ 00:39:35.489 Brian Pei: Actually just had a thought.
416 00:39:35.630 ⇒ 00:39:38.319 Brian Pei: I will actually try to have
417 00:39:38.828 ⇒ 00:39:44.919 Brian Pei: a real good thought for this something that I would that, I think, would be really cool.
418 00:39:45.746 ⇒ 00:39:47.280 Brian Pei: If this
419 00:39:48.260 ⇒ 00:39:49.470 Brian Pei: entity
420 00:39:49.770 ⇒ 00:39:54.249 Brian Pei: I don’t know what AI is. If this entity has already like, read through
421 00:39:54.310 ⇒ 00:39:57.560 Brian Pei: all of our repos that are in production.
422 00:39:57.630 ⇒ 00:40:00.109 Brian Pei: And I’m on this coffee client, and I’m
423 00:40:00.160 ⇒ 00:40:16.590 Brian Pei: writing you know, order line items from scratch and I and I put that in a Pr. It would be cool if it says something like, Hey, in pool parts to go. It actually joins on date and customer here, and you don’t join on that. Would you consider joining on.
424 00:40:16.630 ⇒ 00:40:18.690 Brian Pei: or something like that where it’s like, Yeah.
425 00:40:18.690 ⇒ 00:40:19.160 Patrick Trainer: Oh!
426 00:40:19.393 ⇒ 00:40:27.580 Brian Pei: Shopify objects all have the same name. So it’s like, Hey, you’ve done this a couple of times before you might be missing this join, or you might be missing this column. That’d be right.
427 00:40:27.580 ⇒ 00:40:34.130 Patrick Trainer: Offer offers advice for like modularization, or keeping your code dry and shit like that. Yeah.
428 00:40:34.130 ⇒ 00:40:40.829 Ryan Luke Daque: I guess that that’s where the hub comes in like the context comes in like giving it context beforehand
429 00:40:40.910 ⇒ 00:40:44.719 Ryan Luke Daque: to make it decide what it wants to show us
430 00:40:45.314 ⇒ 00:40:49.140 Ryan Luke Daque: but the other I don’t know how we can do that, but that’s
431 00:40:50.010 ⇒ 00:40:50.480 Ryan Luke Daque: that’s.
432 00:40:50.480 ⇒ 00:40:57.059 Uttam Kumaran: Yeah, I mean, like we, because we had this, me and Brian had this conversation today where I was like, oh, we just. We did the similar thing in pool parts.
433 00:40:57.100 ⇒ 00:41:03.300 Uttam Kumaran: I think we’re I mean again, Miguel, this goes all to the stuff we’re gonna have to figure out with like having vector, stores for everything.
434 00:41:03.400 ⇒ 00:41:31.870 Uttam Kumaran: but also like that shared learning is already. It’s already happens between us and the company. So this is just like, how do we make that shared knowledge available to the bot. The nice thing is, the bot only needs to know about like SQL. Files. So there’s already we can reduce the scope a lot it needs to know about SQL files. Dbt already has a relationships so it it can know what the relationships are. Right. So the more we layer on, and the more and the more we build on top of it, I think it’ll actually work really well.
435 00:41:32.432 ⇒ 00:41:37.109 Uttam Kumaran: And then the last thing, I think. Yeah, the the last thing I’ll say and is.
436 00:41:37.340 ⇒ 00:41:38.135 Uttam Kumaran: are
437 00:41:39.620 ⇒ 00:41:43.310 Uttam Kumaran: Then this is probably gonna be the one painful part of this is our
438 00:41:43.330 ⇒ 00:41:44.690 Uttam Kumaran: issues.
439 00:41:44.780 ⇒ 00:41:47.620 Uttam Kumaran: I have to be really clear about what we’re trying to do.
440 00:41:47.650 ⇒ 00:41:49.560 Uttam Kumaran: because otherwise I don’t.
441 00:41:49.790 ⇒ 00:41:54.590 Uttam Kumaran: Are. These are the issues or the pull request. Description has to be super clear.
442 00:41:56.130 ⇒ 00:42:00.620 Uttam Kumaran: otherwise, I think the AI is just not. It’s not gonna be able to fill that gap up
443 00:42:00.660 ⇒ 00:42:01.690 Uttam Kumaran: easily.
444 00:42:02.050 ⇒ 00:42:06.100 Uttam Kumaran: I think, for for some stuff it’s gonna be able to infer what’s going on.
445 00:42:06.540 ⇒ 00:42:11.669 Uttam Kumaran: But my hunch is that the better our issues or pr descriptions are, the better
446 00:42:12.020 ⇒ 00:42:13.650 Uttam Kumaran: the responses will be.
447 00:42:14.780 ⇒ 00:42:15.747 Ryan Luke Daque: Yeah, that makes sense.
448 00:42:16.480 ⇒ 00:42:16.855 Ryan Luke Daque: But
449 00:42:17.230 ⇒ 00:42:20.119 Patrick Trainer: I was gonna say, was in response to
450 00:42:20.160 ⇒ 00:42:26.109 Patrick Trainer: Udem. What you said about like all it needs to really like know about it are those SQL files.
451 00:42:26.270 ⇒ 00:42:33.870 Patrick Trainer: And if you I mean even take that a step forward. Dbt is already creating that like that, manifest that graph. So.
452 00:42:33.870 ⇒ 00:42:34.210 Uttam Kumaran: Yeah.
453 00:42:34.210 ⇒ 00:42:40.429 Patrick Trainer: Already like. It has all that SQL. Text, and it has all of the relations, and how they all
454 00:42:40.450 ⇒ 00:42:42.340 Patrick Trainer: like it has the dag.
455 00:42:42.450 ⇒ 00:42:46.679 Patrick Trainer: And so you feed that big Json blob into it.
456 00:42:47.000 ⇒ 00:42:53.589 Patrick Trainer: and then it understands the context of the entire project in in relation to the change.
457 00:42:55.110 ⇒ 00:42:55.800 Patrick Trainer: So.
458 00:42:56.540 ⇒ 00:43:00.660 Uttam Kumaran: So I wonder if it’s actually that we should send in instead of yeah.
459 00:43:00.660 ⇒ 00:43:03.520 Patrick Trainer: Yeah, yeah, I mean, and that may be
460 00:43:03.760 ⇒ 00:43:05.850 Patrick Trainer: like the way that I would see that work
461 00:43:05.930 ⇒ 00:43:07.480 Patrick Trainer: would be
462 00:43:07.560 ⇒ 00:43:10.269 Patrick Trainer: like when this.
463 00:43:10.870 ⇒ 00:43:13.080 Patrick Trainer: when the job kicks off.
464 00:43:13.670 ⇒ 00:43:14.140 Patrick Trainer: Yeah.
465 00:43:14.140 ⇒ 00:43:15.599 Uttam Kumaran: It somewhere. It it.
466 00:43:15.600 ⇒ 00:43:17.790 Patrick Trainer: Compiles the Dbt project.
467 00:43:18.060 ⇒ 00:43:23.470 Patrick Trainer: grabs that manifest file and then uses that manifest file as context
468 00:43:23.500 ⇒ 00:43:25.090 Patrick Trainer: to the
469 00:43:26.300 ⇒ 00:43:30.509 Patrick Trainer: Chat Gpt, or to the yeah to the Gpt. Api call.
470 00:43:31.580 ⇒ 00:43:33.519 Uttam Kumaran: Yeah, I mean, I think
471 00:43:33.600 ⇒ 00:43:38.250 Uttam Kumaran: I think, what we’re gonna yeah, I think either that or what we’re gonna try to do is basically
472 00:43:40.440 ⇒ 00:43:46.600 Uttam Kumaran: like. And this is what we’re gonna I’m trying. I’m probably have to look into this weekend. But we’re gonna have vector, stores for every client
473 00:43:46.850 ⇒ 00:43:59.479 Uttam Kumaran: where we shove all the info into, and then the vector, store gets is available to any tool related to the client. So if it’s like a hey, I need to cause. The way that these AI agents work is. It’s like
474 00:43:59.620 ⇒ 00:44:01.030 Uttam Kumaran: I have a I get a
475 00:44:01.080 ⇒ 00:44:04.779 Uttam Kumaran: the way the thing is like, I have a question about code. I should go pull.
476 00:44:04.810 ⇒ 00:44:07.180 Uttam Kumaran: I should go pull the code
477 00:44:07.300 ⇒ 00:44:10.079 Uttam Kumaran: from the vector, store right? And then it asks a question.
478 00:44:10.130 ⇒ 00:44:20.849 Uttam Kumaran: The part right now that it’s trouble is, I think, for some use cases. The context window is going to be big enough, especially now. And I think with Gemini and some stuff we could probably fit the entire manifest in there
479 00:44:21.310 ⇒ 00:44:25.899 Uttam Kumaran: or even to slim down, because maybe some of the stuff is like, not that useful. But
480 00:44:26.180 ⇒ 00:44:35.410 Uttam Kumaran: I do think at some point, if we wanted to have more contacts, or even, for example, Pat, I wanted to load in the entire Snowflake docs.
481 00:44:36.820 ⇒ 00:44:45.709 Uttam Kumaran: I want all the Snowflake docs, all the Dbt docs, and basically any materials on writing sequel. Best practices to sit somewhere
482 00:44:46.150 ⇒ 00:45:00.599 Uttam Kumaran: right? Because, like, that’s also that’s how we get beyond us, which is like, Oh, for example, Brian would be like, Oh, you could use like qualify here, or something like that. Instead of this. It’s like, Oh, shit. Yeah. I totally could have done that. That’s where I wanted to basically have all that information.
483 00:45:02.000 ⇒ 00:45:08.050 Uttam Kumaran: and then you know where that goes to help is like, we’re not only building this Pr about anytime. You have a question about how to do something.
484 00:45:08.490 ⇒ 00:45:11.019 Uttam Kumaran: and which is when you have an issue.
485 00:45:11.435 ⇒ 00:45:21.920 Uttam Kumaran: When you create the issue, a suggestion will get made about what kind of code, what kind of documents you should go right. Those are the kind of things where we build up this like repository of knowledge.
486 00:45:22.212 ⇒ 00:45:27.789 Uttam Kumaran: And that’s the thing. Why, we’re not. We don’t. We can’t find this pro like, I can’t find this product off the shelf.
487 00:45:28.424 ⇒ 00:45:33.469 Uttam Kumaran: Which is building this because every development environment is different.
488 00:45:33.480 ⇒ 00:45:38.860 Uttam Kumaran: There’s some people that are trying to do this for front, end and back end. But in our world this is not happening that easily.
489 00:45:38.860 ⇒ 00:45:39.460 Patrick Trainer: Doesn’t exist.
490 00:45:39.460 ⇒ 00:45:40.900 Uttam Kumaran: Especially in data modeling.
491 00:45:41.218 ⇒ 00:45:46.879 Uttam Kumaran: and so like, I want us to try to build this and basically like, see how we can.
492 00:45:47.200 ⇒ 00:45:52.840 Uttam Kumaran: If you, as a solo ae can take on these projects and and like crush.
493 00:45:52.940 ⇒ 00:46:00.159 Uttam Kumaran: then it’s it’s great right. And I think Brian is a Brian on on Javi. Coffee is a great use case.
494 00:46:00.602 ⇒ 00:46:02.130 Uttam Kumaran: To try to nail.
495 00:46:02.210 ⇒ 00:46:12.369 Uttam Kumaran: Yeah, especially because he has no context of the AI stuff which makes it even better, actually, for me. Because he’s really looking, not from a lens of like
496 00:46:13.020 ⇒ 00:46:13.980 Uttam Kumaran: there’s no
497 00:46:15.070 ⇒ 00:46:19.030 Uttam Kumaran: like. Okay? Well, like, let’s cut it some slack. It like really needs to work
498 00:46:19.160 ⇒ 00:46:27.050 Uttam Kumaran: right? So that’s like, kind of like what I’m very interested in. So let’s let’s take that unless anyone else has anything else. Like, let’s take
499 00:46:27.070 ⇒ 00:46:33.630 Uttam Kumaran: the action items that we talked about today. We’ll take this meeting transcript, and then we’ll kind of create a couple of
500 00:46:33.720 ⇒ 00:46:36.769 Uttam Kumaran: next steps, and maybe we can talk about that tomorrow, Pat.
501 00:46:36.920 ⇒ 00:46:42.679 Uttam Kumaran: and let’s see again, like I think, we’ll find out where the time comes from to dedicate to stuff like this. But this is dope
502 00:46:43.072 ⇒ 00:46:48.340 Uttam Kumaran: and we’ll plan this out. I think this is a great 1st start. I think at minimum, you know, we’ll fork this and do stuff. But
503 00:46:48.400 ⇒ 00:47:07.510 Uttam Kumaran: before we do anything else, we can kind of set set a roadmap, I think, for Miguel, too. We’re gonna we’re gonna have people in the company that have used github and stuff maybe individually or on teams. But some of the stuff where we’ve done actions, and you have this, these sort of like overarching concepts. I I want everybody to kind of be familiar with
504 00:47:07.630 ⇒ 00:47:32.809 Uttam Kumaran: right? I want us on the development side to understand how it impacts us. I want folks on the AI side to know like how they can configure this. And they’re like, for example, Miguel is super strong at like prompt engineering and writing these really great prompts. And so there’s a portion of that that’s here. I want people like Nico to be able to understand, like how this helps up speed projects, how we can communicate, how this helps clients get better work from us.
505 00:47:32.830 ⇒ 00:47:38.739 Uttam Kumaran: And then for me, I want to know like how I can sell this, you know. So that’s the that’s the 4 pillars. Right? So
506 00:47:38.870 ⇒ 00:47:40.260 Uttam Kumaran: so let’s go with that.
507 00:47:43.360 ⇒ 00:47:44.140 Patrick Trainer: Roll Todd.
508 00:47:44.830 ⇒ 00:47:47.609 Uttam Kumaran: Cool look he’s just been. He’s just been sleeping in the background here.
509 00:47:48.680 ⇒ 00:47:49.969 Uttam Kumaran: totally passed out.
510 00:47:50.280 ⇒ 00:47:51.300 Uttam Kumaran: Yeah. So.
511 00:47:51.960 ⇒ 00:47:53.029 Patrick Trainer: My dog tooth.
512 00:47:53.240 ⇒ 00:47:54.870 Uttam Kumaran: Nice. Yeah.
513 00:47:55.070 ⇒ 00:47:56.639 Uttam Kumaran: Oh, he’s alive.
514 00:47:56.880 ⇒ 00:47:58.040 Uttam Kumaran: You live.
515 00:48:00.610 ⇒ 00:48:03.360 Uttam Kumaran: Okay, cool guys. Anything else.
516 00:48:03.360 ⇒ 00:48:03.890 Ryan Luke Daque: Yes.
517 00:48:06.010 ⇒ 00:48:06.630 Patrick Trainer: That’s it.
518 00:48:07.200 ⇒ 00:48:07.520 Ryan Luke Daque: Yeah.
519 00:48:07.840 ⇒ 00:48:08.580 Uttam Kumaran: Awesome.
520 00:48:08.870 ⇒ 00:48:10.220 Patrick Trainer: Gonna go get some lunch.
521 00:48:11.140 ⇒ 00:48:13.460 Uttam Kumaran: Thanks, guys. I’ll talk to you soon.
522 00:48:13.460 ⇒ 00:48:14.060 Ryan Luke Daque: See you.
523 00:48:14.060 ⇒ 00:48:14.500 Nicolas Sucari: Alright, bye.
524 00:48:14.500 ⇒ 00:48:15.520 joshuadeveyra: You guys, bye-bye.