Meeting Title: Slack → Supabase Pipeline Date: 2025-07-07 Meeting participants: Uttam Kumaran, Mustafa Raja, Awaish Kumar, Casie Aviles, Ryan Luke Daque, Rico Rejoso
WEBVTT
1 00:04:38.080 ⇒ 00:04:39.450 Awaish Kumar: Yeah. Hello.
2 00:04:41.290 ⇒ 00:04:42.130 Ryan Luke Daque: Hi, everyone!
3 00:04:43.560 ⇒ 00:04:44.539 Ryan Luke Daque: How’s it going.
4 00:04:45.190 ⇒ 00:04:46.539 Mustafa Raja: Hey! How are you on?
5 00:04:48.290 ⇒ 00:04:49.450 Ryan Luke Daque: All good.
6 00:04:54.090 ⇒ 00:04:55.220 Ryan Luke Daque: Okay.
7 00:04:55.540 ⇒ 00:04:56.510 Awaish Kumar: So.
8 00:05:02.940 ⇒ 00:05:04.349 Mustafa Raja: Should be paying with them.
9 00:05:07.300 ⇒ 00:05:12.368 Awaish Kumar: I don’t know. He he was interested in joining. We can wait for a few minutes.
10 00:05:13.460 ⇒ 00:05:15.709 Awaish Kumar: So look like, did you?
11 00:05:16.060 ⇒ 00:05:20.710 Awaish Kumar: What did you made any changes. After our discussion.
12 00:05:21.490 ⇒ 00:05:29.196 Ryan Luke Daque: Yeah, I I did. But I was gonna try in local. But I wasn’t. I wasn’t successful.
13 00:05:30.130 ⇒ 00:05:32.029 Ryan Luke Daque: but I guess I can push my
14 00:05:32.520 ⇒ 00:05:36.400 Ryan Luke Daque: changes. Let me just share my screen so everybody can see.
15 00:05:39.410 ⇒ 00:05:41.370 Ryan Luke Daque: Can you see my screen.
16 00:05:43.990 ⇒ 00:05:46.523 Ryan Luke Daque: So basically, what I did
17 00:05:48.040 ⇒ 00:05:50.448 Ryan Luke Daque: initially was to update the
18 00:05:51.500 ⇒ 00:05:54.945 Ryan Luke Daque: the config file that we had currently
19 00:05:56.240 ⇒ 00:06:04.040 Ryan Luke Daque: But this time I added all the the rest of the clients that we have, including.
20 00:06:04.280 ⇒ 00:06:11.509 Ryan Luke Daque: like grouping all of the channels for each client, for example, lead, and we have, like 4 channels for lead and web.
21 00:06:11.830 ⇒ 00:06:15.639 Ryan Luke Daque: The client Eden Channel external Eden data team
22 00:06:15.750 ⇒ 00:06:17.120 Ryan Luke Daque: and the rest of the other
23 00:06:17.540 ⇒ 00:06:22.589 Ryan Luke Daque: hidden channels looks like. So I did that for the rest of the
24 00:06:23.370 ⇒ 00:06:33.630 Ryan Luke Daque: clients, and also added something like that for internal, and then group it by whatever grouping
25 00:06:33.930 ⇒ 00:06:43.190 Ryan Luke Daque: that most likely represents these like, for example, engineering would be like anything that has like engineering before prefix
26 00:06:44.060 ⇒ 00:06:48.249 Ryan Luke Daque: like sales would be all the sales prefix channels as well.
27 00:06:48.410 ⇒ 00:06:55.659 Ryan Luke Daque: Like project management would be like, project channels. AI, anything with AI, so does this.
28 00:06:57.000 ⇒ 00:06:59.179 Ryan Luke Daque: does this make sense for the grouping?
29 00:07:02.230 ⇒ 00:07:03.110 Ryan Luke Daque: So?
30 00:07:03.750 ⇒ 00:07:08.720 Ryan Luke Daque: But yeah, I did this. I also added, I, I did not include the
31 00:07:09.010 ⇒ 00:07:12.618 Ryan Luke Daque: other channels like the vendor channels or external
32 00:07:14.218 ⇒ 00:07:17.010 Ryan Luke Daque: what? What other stuff do we have?
33 00:07:18.720 ⇒ 00:07:23.070 Ryan Luke Daque: Yeah, like vendor channels. For instance, do we need those?
34 00:07:23.470 ⇒ 00:07:24.460 Ryan Luke Daque: Alright?
35 00:07:25.260 ⇒ 00:07:26.990 Ryan Luke Daque: We can skip those for now.
36 00:07:27.580 ⇒ 00:07:30.270 Ryan Luke Daque: You can easily add them here. Anyway.
37 00:07:30.270 ⇒ 00:07:32.760 Uttam Kumaran: I don’t think we need the vendor channels at all.
38 00:07:33.300 ⇒ 00:07:40.299 Uttam Kumaran: I think just the core clients. And then the the table, the basically the channels that are most
39 00:07:40.960 ⇒ 00:07:41.880 Uttam Kumaran: talk, then.
40 00:07:43.280 ⇒ 00:07:44.100 Ryan Luke Daque: Okay.
41 00:07:45.120 ⇒ 00:07:45.810 Ryan Luke Daque: Gotcha.
42 00:07:45.810 ⇒ 00:07:46.590 Awaish Kumar: You know.
43 00:07:47.480 ⇒ 00:07:49.000 Ryan Luke Daque: So, yeah.
44 00:07:49.960 ⇒ 00:07:57.010 Awaish Kumar: Like the purpose is to just this script. So the only thing right now missing is
45 00:07:57.120 ⇒ 00:08:01.359 Awaish Kumar: that it does not create the table. So like you have.
46 00:08:01.360 ⇒ 00:08:01.850 Ryan Luke Daque: Yeah.
47 00:08:01.850 ⇒ 00:08:03.489 Awaish Kumar: Feature functionality right now.
48 00:08:03.620 ⇒ 00:08:07.319 Awaish Kumar: So after that, like, it should be easy just to just.
49 00:08:07.460 ⇒ 00:08:12.940 Awaish Kumar: I had a new client and channel, so it will just go and create table and supervis.
50 00:08:13.420 ⇒ 00:08:14.070 Ryan Luke Daque: Right.
51 00:08:14.280 ⇒ 00:08:24.990 Ryan Luke Daque: So I added, this function which creates a table if it doesn’t exist, and and when, where, when it calls the right to super base function. It calls that
52 00:08:25.190 ⇒ 00:08:30.619 Ryan Luke Daque: function so that if there’s any new table that doesn’t exist in super base, it creates it based on
53 00:08:30.740 ⇒ 00:08:34.929 Ryan Luke Daque: by the current schema that we have from start to base.
54 00:08:35.688 ⇒ 00:08:40.591 Ryan Luke Daque: I wanted to test this locally. But yeah, I wasn’t able to, because I didn’t have the
55 00:08:41.440 ⇒ 00:08:45.329 Ryan Luke Daque: credentials within environment variables. So maybe
56 00:08:45.670 ⇒ 00:08:48.759 Ryan Luke Daque: maybe I can just push this and we can test it.
57 00:08:48.760 ⇒ 00:08:51.470 Awaish Kumar: You can get those from Dexter Cloud.
58 00:08:51.926 ⇒ 00:08:52.840 Ryan Luke Daque: I see.
59 00:08:56.650 ⇒ 00:09:01.820 Ryan Luke Daque: And also what I did here was for, at least for the current compliance
60 00:09:02.070 ⇒ 00:09:06.159 Ryan Luke Daque: labels that we had. I just changed the naming convention, so that it would
61 00:09:06.280 ⇒ 00:09:15.230 Ryan Luke Daque: follow the new naming convention that we are. With that I updated in our now. It already.
62 00:09:15.230 ⇒ 00:09:15.840 Awaish Kumar: Okay.
63 00:09:17.460 ⇒ 00:09:23.529 Ryan Luke Daque: Yeah, basically it in in the script. It has this naming convention, like client underscore
64 00:09:23.660 ⇒ 00:09:29.040 Ryan Luke Daque: the client name underscore messages. And if it’s internal. It’s internal underscore. The internal.
65 00:09:31.490 ⇒ 00:09:31.950 Awaish Kumar: Evening.
66 00:09:31.950 ⇒ 00:09:38.359 Ryan Luke Daque: Under. Yeah, something like that. So I just changed the current one, so that once the pipeline is up
67 00:09:38.520 ⇒ 00:09:41.559 Ryan Luke Daque: is in production, it would just load.
68 00:09:42.140 ⇒ 00:09:42.490 Awaish Kumar: And then.
69 00:09:43.430 ⇒ 00:09:44.689 Ryan Luke Daque: So yeah,
70 00:09:46.270 ⇒ 00:09:54.790 Awaish Kumar: So yeah, just for everybody. The flow is like, so we have polyatomic which is.
71 00:09:55.310 ⇒ 00:09:58.670 Awaish Kumar: And there’s a connection between polyatomic and S. 3.
72 00:09:58.800 ⇒ 00:10:02.330 Awaish Kumar: So if I can share my screen, oh.
73 00:10:05.560 ⇒ 00:10:10.879 Awaish Kumar: so the data so basically. So
74 00:10:11.000 ⇒ 00:10:17.900 Awaish Kumar: it goes into internal AI bucket. And here we have a slack folder. Inside this slack folder we have tools.
75 00:10:18.200 ⇒ 00:10:22.930 Awaish Kumar: select messages and select matches, messages, transform folders.
76 00:10:23.820 ⇒ 00:10:26.929 Awaish Kumar: So basically, the slack messages folder is
77 00:10:27.080 ⇒ 00:10:30.539 Awaish Kumar: where the polyatomic is storing the data which is
78 00:10:31.604 ⇒ 00:10:37.290 Awaish Kumar: coming from slack channels. So it’s in this format. So we only have the 4 files here.
79 00:10:37.720 ⇒ 00:10:40.379 Awaish Kumar: and it’s not divided by the client being.
80 00:10:40.570 ⇒ 00:10:42.400 Awaish Kumar: And now the second.
81 00:10:42.640 ⇒ 00:10:53.439 Awaish Kumar: the the script which looks short. What it is doing is basically reads this data from this select messages folder and transforms it
82 00:10:53.570 ⇒ 00:10:55.810 Awaish Kumar: per client basis.
83 00:10:55.950 ⇒ 00:10:58.269 Awaish Kumar: So now it should be
84 00:10:58.750 ⇒ 00:11:08.709 Awaish Kumar: in this format. So now we have a select messages transformed for each client. So for Eden it goes into the Eden folder.
85 00:11:09.510 ⇒ 00:11:16.489 Awaish Kumar: and for, like each of these clients, it goes into their own client folder.
86 00:11:16.590 ⇒ 00:11:19.099 Awaish Kumar: and we store it as a
87 00:11:19.350 ⇒ 00:11:25.570 Awaish Kumar: with the d snapshot. So it it retains the version of the
88 00:11:26.692 ⇒ 00:11:32.509 Awaish Kumar: messages. So it it. So we have version history of all our slack channels in s. 3.
89 00:11:33.170 ⇒ 00:11:34.789 Awaish Kumar: So this is one part.
90 00:11:35.410 ⇒ 00:11:41.800 Awaish Kumar: and this script is doing so. It reads from S. 3 stores back to S. 3 here, and then
91 00:11:41.990 ⇒ 00:11:46.119 Awaish Kumar: the second part part of it is that it also stores in super base.
92 00:11:46.410 ⇒ 00:11:51.269 Awaish Kumar: So in the super base, we have a project called Internal Slack.
93 00:11:51.910 ⇒ 00:11:57.246 Awaish Kumar: In this project, we basically have all ours like
94 00:11:59.300 ⇒ 00:12:09.290 Awaish Kumar: we have all these tables called the end, all all the tables which end with messages. They are basically populated by this script.
95 00:12:09.580 ⇒ 00:12:14.350 Awaish Kumar: So we do have few other tables here which are which ends with transformed.
96 00:12:14.520 ⇒ 00:12:22.080 Awaish Kumar: But they are being basically used by Miguel. And he add some embeddings there. So
97 00:12:22.260 ⇒ 00:12:30.669 Awaish Kumar: basically this. What this script is doing is basically just creates all these tables which end with messages. And it has basically
98 00:12:30.800 ⇒ 00:12:35.049 Awaish Kumar: the data from all the channels for this specific client.
99 00:12:35.440 ⇒ 00:12:37.000 Awaish Kumar: So client Eden.
100 00:12:37.440 ⇒ 00:12:44.650 Awaish Kumar: we have the data here for all the messages, and we can differentiate between channels by using the
101 00:12:44.920 ⇒ 00:12:47.569 Awaish Kumar: channel Id and
102 00:12:48.870 ⇒ 00:12:53.980 Awaish Kumar: we. And it will have like if if Eden has 5 channels. So all of that data is coming into this table.
103 00:12:54.740 ⇒ 00:12:58.409 Awaish Kumar: So that’s basically the the core architecture.
104 00:13:08.180 ⇒ 00:13:13.099 Awaish Kumar: So for Luke, like, I think there’s 1 more blocker.
105 00:13:13.350 ⇒ 00:13:18.300 Awaish Kumar: So so basically, you have updated the the this
106 00:13:21.030 ⇒ 00:13:31.760 Awaish Kumar: like these old, like, the table, exist these existing tables with the new naming. And you also added the updated the script so that it creates tables if they don’t exist.
107 00:13:32.334 ⇒ 00:13:38.509 Awaish Kumar: So the super way. Side is good. So but now we you also have to take care of this thing, or
108 00:13:38.780 ⇒ 00:13:43.150 Awaish Kumar: where we are storing this data into S. 3 as well.
109 00:13:43.800 ⇒ 00:13:45.679 Awaish Kumar: so I’m not sure how it’s going to change.
110 00:13:45.680 ⇒ 00:13:47.530 Uttam Kumaran: That’s just creating the folders.
111 00:13:48.680 ⇒ 00:13:52.080 Awaish Kumar: Yeah, it will create a folder with a client name
112 00:13:52.580 ⇒ 00:13:56.849 Awaish Kumar: and just stores the data in a in a format like
113 00:13:57.250 ⇒ 00:13:59.649 Awaish Kumar: as a with the date and the time. Yeah.
114 00:14:02.370 ⇒ 00:14:05.739 Ryan Luke Daque: Is that coming from the script, or that’s from Polytoni.
115 00:14:05.740 ⇒ 00:14:17.130 Awaish Kumar: It’s coming from this script. So yeah, this slack messages folder. This is where polytomic is storing the data. And this slack messages transform. This is basically
116 00:14:17.340 ⇒ 00:14:19.470 Awaish Kumar: being populated by this script.
117 00:14:19.740 ⇒ 00:14:23.720 Awaish Kumar: So so I’m I’m like for for the new
118 00:14:24.010 ⇒ 00:14:26.519 Awaish Kumar: clients and for the new internal
119 00:14:27.191 ⇒ 00:14:39.860 Awaish Kumar: channels like the departments. I I don’t think there’s anything to do, because it will just create a folder and store the data. The script should just work fine, but for the existing ones.
120 00:14:40.246 ⇒ 00:14:57.360 Awaish Kumar: It’s going to affect, because now the naming you have changed the naming. So instead of Idn, it will have a client ident name right, so it will create a new folder instead of storing into the same. So we might have to rename the existing ones, and then run the script and verify.
121 00:14:59.360 ⇒ 00:15:00.010 Ryan Luke Daque: Okay.
122 00:15:05.530 ⇒ 00:15:09.599 Awaish Kumar: Yeah, that’s it. If there’s any more questions, please let me know.
123 00:15:16.090 ⇒ 00:15:20.340 Ryan Luke Daque: Would you be able to test the script? A wish on your end?
124 00:15:21.970 ⇒ 00:15:23.430 Ryan Luke Daque: Yeah, I I can.
125 00:15:25.990 ⇒ 00:15:28.939 Ryan Luke Daque: Yeah, I just. I just push the changes to them.
126 00:15:30.180 ⇒ 00:15:37.900 Ryan Luke Daque: The pr, yeah, let me try to update the names of the folders in S 3.
127 00:15:38.220 ⇒ 00:15:46.010 Awaish Kumar: Okay, okay, do that, and then let me know. I will test the script locally, and then that
128 00:15:47.280 ⇒ 00:15:50.020 Awaish Kumar: I’ll review it and we can merge it today.
129 00:15:51.340 ⇒ 00:15:52.250 Ryan Luke Daque: Sounds good.
130 00:15:56.730 ⇒ 00:16:00.119 Uttam Kumaran: Anyone else have any questions. On the whole slack pipeline.
131 00:16:07.160 ⇒ 00:16:08.690 Casie Aviles: So far.
132 00:16:08.860 ⇒ 00:16:14.189 Awaish Kumar: Me, yeah, like, this is like 50% of the pipeline.
133 00:16:15.000 ⇒ 00:16:25.569 Awaish Kumar: 50% is like embeddings, and all is is another pipeline which is created by Miguel.
134 00:16:29.990 ⇒ 00:16:33.319 Uttam Kumaran: Like the when it’s within super base.
135 00:16:33.960 ⇒ 00:16:35.735 Uttam Kumaran: I guess. There you go.
136 00:16:37.160 ⇒ 00:16:39.940 Uttam Kumaran: I don’t know, Miguel. I don’t think Miguel’s on this one. But
137 00:16:40.391 ⇒ 00:16:47.028 Uttam Kumaran: yeah, I guess I know. Remember, he had a he had one where it’s within super base sort of doing
138 00:16:47.730 ⇒ 00:16:50.030 Uttam Kumaran: like the pre filtering process right.
139 00:16:51.399 ⇒ 00:16:58.980 Awaish Kumar: Yes, but it’s it’s like the the pipeline is in Dexter. It’s called embedding and the the transform jobs.
140 00:16:59.130 ⇒ 00:17:03.010 Awaish Kumar: But that basically does the transformation on super base.
141 00:17:04.910 ⇒ 00:17:07.352 Uttam Kumaran: So once you get it into
142 00:17:08.440 ⇒ 00:17:14.229 Uttam Kumaran: once you get it into S. 3, though that’s all we need for Luke’s task.
143 00:17:14.510 ⇒ 00:17:19.450 Uttam Kumaran: cause he’s purely running some analytics on top of the slack data.
144 00:17:20.126 ⇒ 00:17:32.189 Uttam Kumaran: Not using it for anything in AI, although the AI team will certainly need it. So I think maybe one question. And I think, Mustafa, this is probably for you. Then, once this data is in here.
145 00:17:32.810 ⇒ 00:17:36.579 Uttam Kumaran: I think there may be some requirement on your side.
146 00:17:37.093 ⇒ 00:17:44.079 Uttam Kumaran: To update Miguel’s scripts because he’s running some sort of pre filtering process on top of this data.
147 00:17:46.600 ⇒ 00:17:51.200 Mustafa Raja: Yeah, let me let me look into the database structure.
148 00:17:51.740 ⇒ 00:17:52.360 Uttam Kumaran: Okay.
149 00:17:59.940 ⇒ 00:18:01.960 Awaish Kumar: The project called Internal Slack.
150 00:18:02.160 ⇒ 00:18:02.750 Mustafa Raja: Yeah.
151 00:18:03.470 ⇒ 00:18:10.106 Mustafa Raja: So so I see that there are 2 2 tables for each client.
152 00:18:10.440 ⇒ 00:18:10.840 Awaish Kumar: Yes.
153 00:18:10.840 ⇒ 00:18:18.680 Mustafa Raja: The client name, and then messages, and then transformed. The both of these are being populated by the script.
154 00:18:19.490 ⇒ 00:18:27.520 Awaish Kumar: No, I the the message, the tables which end with string messages.
155 00:18:27.630 ⇒ 00:18:47.339 Awaish Kumar: They are being populated by the the script which I which Luke just showed about it’s called slack data transformation pipeline. This basically just stores the data as it comes from slack. So messages, text channel id user id whatever there is and username.
156 00:18:47.540 ⇒ 00:18:54.540 Awaish Kumar: Then there is another table which ends with performed, that is basically being created by the
157 00:18:54.690 ⇒ 00:19:02.719 Awaish Kumar: this script written by Miguel. So it it will have embeddings and other like, yeah, I deleted.
158 00:19:02.720 ⇒ 00:19:03.490 Mustafa Raja: Okay.
159 00:19:03.490 ⇒ 00:19:04.030 Awaish Kumar: Thanks. Yeah.
160 00:19:04.630 ⇒ 00:19:11.289 Mustafa Raja: Okay, okay, for my task. I I would only need the messages table that Luke’s script would give
161 00:19:12.155 ⇒ 00:19:15.010 Mustafa Raja: for the for the transform thing.
162 00:19:16.138 ⇒ 00:19:26.329 Mustafa Raja: Maybe we should. Update Miguel on this if we need it. I don’t. I don’t know if we need it, or where are we using this transformed table thing.
163 00:19:26.950 ⇒ 00:19:27.510 Uttam Kumaran: Okay.
164 00:19:27.510 ⇒ 00:19:30.799 Mustafa Raja: Maybe this is for the for the client hubs actually.
165 00:19:33.950 ⇒ 00:19:38.350 Uttam Kumaran: Yeah, the the it is gonna be for client hubs. I just didn’t know if you needed it. If not.
166 00:19:38.350 ⇒ 00:19:41.929 Mustafa Raja: That’s fine. I don’t. I don’t need the transformed thing.
167 00:19:41.930 ⇒ 00:19:42.315 Uttam Kumaran: Okay.
168 00:19:45.180 ⇒ 00:19:45.730 Mustafa Raja: Yeah.
169 00:19:49.150 ⇒ 00:19:51.190 Uttam Kumaran: Okay, okay, great.
170 00:19:51.850 ⇒ 00:19:54.410 Uttam Kumaran: Anything else. We wanna chat through.
171 00:19:59.300 ⇒ 00:20:05.028 Uttam Kumaran: Okay, perfect. So yeah, Luke, if you wanna make that Pr, and then a waste can probably review it.
172 00:20:05.790 ⇒ 00:20:11.989 Uttam Kumaran: and then, yeah, I think part of the new client onboarding will, or
173 00:20:12.480 ⇒ 00:20:19.030 Uttam Kumaran: I think periodically, we may have to start to add these for new channels. I’m not sure we have to think some process.
174 00:20:22.120 ⇒ 00:20:27.350 Ryan Luke Daque: Yeah, I just committed the changes to the Pr, so always can review.
175 00:20:27.790 ⇒ 00:20:31.510 Ryan Luke Daque: And then we can test it out. And yeah should be good.
176 00:20:32.470 ⇒ 00:20:33.290 Awaish Kumar: Okay. Okay.
177 00:20:34.780 ⇒ 00:20:35.540 Awaish Kumar: Thank you.
178 00:20:36.250 ⇒ 00:20:36.840 Ryan Luke Daque: Thanks.
179 00:20:36.840 ⇒ 00:20:38.690 Ryan Luke Daque: Thank you. Talk soon.
180 00:20:40.280 ⇒ 00:20:41.130 Rico Rejoso: Thanks guys.
181 00:20:41.490 ⇒ 00:20:41.980 Uttam Kumaran: Bye.