Meeting Title: MatterMore x Brainforge | Standup Date: 2025-05-12 Meeting participants: Annie Yu, Luke Daque, Trevor Cohen, Amber Lin
WEBVTT
1 00:01:22.000 ⇒ 00:01:23.329 Amber Lin: Hi Trevor.
2 00:01:46.980 ⇒ 00:01:47.730 Trevor Cohen: Hello!
3 00:01:49.930 ⇒ 00:01:50.750 Amber Lin: I’m
4 00:01:54.980 ⇒ 00:01:56.710 Amber Lin: pretty good slow start to my life.
5 00:01:56.710 ⇒ 00:01:59.539 Amber Lin: I hear you wait. Oh, now I can. Okay, that was me.
6 00:02:00.660 ⇒ 00:02:02.830 Amber Lin: What about you? How’s your weekend been.
7 00:02:02.960 ⇒ 00:02:07.009 Trevor Cohen: It was good. It’s my birthday, this weekend. So I did some some birthday stuff. It was fun.
8 00:02:07.010 ⇒ 00:02:09.083 Amber Lin: Happy birthday.
9 00:02:10.120 ⇒ 00:02:14.270 Trevor Cohen: Thanks. It was my sister’s birthday, too, because we’re twins. So we did. So we did puppy yoga together.
10 00:02:14.513 ⇒ 00:02:15.000 Amber Lin: Oh, wow!
11 00:02:15.000 ⇒ 00:02:18.780 Luke Daque: Oh, that’s cool, happy birthday to you and your sister. Then.
12 00:02:19.130 ⇒ 00:02:25.489 Amber Lin: Is your sister? Does your sister live close by, or it’s like, Are you guys traveling to me? Oh.
13 00:02:25.490 ⇒ 00:02:28.359 Trevor Cohen: Yeah. She lives in Brooklyn also she lives in Wizard Terrace.
14 00:02:29.690 ⇒ 00:02:30.360 Trevor Cohen: Yeah.
15 00:02:30.360 ⇒ 00:02:31.090 Amber Lin: Yeah.
16 00:02:31.400 ⇒ 00:02:32.700 Amber Lin: Very. Fun.
17 00:02:32.700 ⇒ 00:02:39.440 Trevor Cohen: And we’re supposed to have dinner, but she also had her bachelorette party on Saturday night, so she was exhausted. So
18 00:02:39.440 ⇒ 00:02:43.479 Trevor Cohen: she’s is. She’s get. She’s getting married on her birthday.
19 00:02:43.780 ⇒ 00:02:46.880 Trevor Cohen: No. Just had her bachelor party. But she she’s
20 00:02:47.610 ⇒ 00:02:49.689 Trevor Cohen: She’s getting married in like a couple weeks.
21 00:02:51.050 ⇒ 00:02:59.040 Trevor Cohen: Yeah, I have a lot of stuff to prepare. I need a I’m playing violin when she walks down the aisle, and I have to make Ray’s speech so it’s getting I’m getting.
22 00:02:59.040 ⇒ 00:03:00.250 Amber Lin: Oh!
23 00:03:00.250 ⇒ 00:03:00.950 Trevor Cohen: Yeah.
24 00:03:01.320 ⇒ 00:03:03.210 Amber Lin: How long have you played ball? Then.
25 00:03:04.970 ⇒ 00:03:16.609 Trevor Cohen: I played. I started when I was young. I like when I was like 4 or 5, but I I hadn’t. I kind of played until college, and then hadn’t really for a while. And now I just fiddle around from time to time.
26 00:03:16.610 ⇒ 00:03:24.210 Amber Lin: I see. I see. Yeah, I also played until college. And I was like, Nope, I’m not bringing this on the plane with me. So.
27 00:03:24.210 ⇒ 00:03:25.420 Trevor Cohen: Violin, cool.
28 00:03:25.420 ⇒ 00:03:34.699 Amber Lin: Yeah, I used to play in the orchestra, but then I was like, I’m not taking this with me on the plane. I had already a lot of years carrying the case.
29 00:03:35.250 ⇒ 00:03:35.830 Trevor Cohen: Familiar.
30 00:03:35.830 ⇒ 00:03:36.195 Amber Lin: Yes.
31 00:03:36.560 ⇒ 00:03:38.419 Trevor Cohen: There’s my case right there.
32 00:03:38.770 ⇒ 00:03:40.180 Amber Lin: Oh, wow!
33 00:03:40.180 ⇒ 00:03:42.400 Trevor Cohen: But I just got my bow rehaired, so I’m ready.
34 00:03:43.370 ⇒ 00:03:44.520 Amber Lin: Exciting!
35 00:03:44.520 ⇒ 00:03:45.140 Trevor Cohen: Yeah.
36 00:03:46.362 ⇒ 00:03:50.419 Amber Lin: Okay, let me let me pull up Marlin here.
37 00:03:51.798 ⇒ 00:03:58.150 Amber Lin: I guess Luke and Annie would. Do you want to share some updates on our end first, st as I pull it up.
38 00:04:01.990 ⇒ 00:04:08.740 Luke Daque: Sure. Yeah. So just a quick update. We already recreated the scripts for.
39 00:04:09.310 ⇒ 00:04:14.119 Luke Daque: yeah, for for the synthetic data. So we can get we we get, we got like all
40 00:04:14.650 ⇒ 00:04:20.010 Luke Daque: user ids in the same basically.
41 00:04:20.610 ⇒ 00:04:28.429 Luke Daque: we we can basically join all the tables that we currently have with the user. Ids, yeah. So we should be ready for that.
42 00:04:28.710 ⇒ 00:04:35.429 Luke Daque: So the only I guess Blocker at the moment is still big query. It looks like we can access bigquery. But
43 00:04:35.970 ⇒ 00:04:41.119 Luke Daque: not the project itself. So we we aren’t able to create tables in bigquery.
44 00:04:41.410 ⇒ 00:04:46.659 Trevor Cohen: Wait, didn’t I? Didn’t. We saw that last time when I slacked you guys the link to the project like Annie? Didn’t you see it.
45 00:04:47.860 ⇒ 00:05:01.577 Luke Daque: Yeah, we we were able to access. It looks like we can see the project, but we can select it from the dropdown, it seems and when we try to like create a table, I can maybe show you.
46 00:05:01.920 ⇒ 00:05:03.530 Trevor Cohen: Sorry this has been such a pain.
47 00:05:04.340 ⇒ 00:05:05.830 Luke Daque: Yeah, no worries.
48 00:05:06.730 ⇒ 00:05:08.770 Luke Daque: Victory a second here.
49 00:05:12.410 ⇒ 00:05:17.199 Amber Lin: I’m saying, if we if it doesn’t work, we should just contact support. Get that figured out.
50 00:05:17.780 ⇒ 00:05:19.299 Luke Daque: Yeah, I think so.
51 00:05:19.300 ⇒ 00:05:23.669 Trevor Cohen: And if you can see it, but not make tables, maybe you have the wrong permission, so I’ll pull up mine also.
52 00:05:26.450 ⇒ 00:05:27.180 Luke Daque: Can’t.
53 00:06:01.160 ⇒ 00:06:06.969 Trevor Cohen: Okay, yeah. Cause you guys have data owner, do you have data editor? But maybe that just doesn’t let you make tables. Let me see.
54 00:06:06.970 ⇒ 00:06:09.008 Luke Daque: I sent you like a list of
55 00:06:10.660 ⇒ 00:06:15.380 Luke Daque: roles that we can pro probably try like a data viewer
56 00:06:15.810 ⇒ 00:06:22.639 Luke Daque: job, user, bigquery data viewer, bigquery job, user bigquery data editor, which I think.
57 00:06:22.640 ⇒ 00:06:30.090 Trevor Cohen: Yeah, yeah, you have data viewer and data editor. But maybe you need data, too. I’m not sure.
58 00:06:31.110 ⇒ 00:06:34.940 Luke Daque: Yeah, let’s try that. And maybe job user as well to run queries.
59 00:06:34.940 ⇒ 00:06:35.670 Trevor Cohen: Okay.
60 00:06:36.343 ⇒ 00:06:39.579 Luke Daque: A viewer do we have, like a viewer or something?
61 00:06:41.050 ⇒ 00:06:45.409 Luke Daque: Not bigquery viewer, but like viewer of the project itself.
62 00:06:47.900 ⇒ 00:06:52.350 Trevor Cohen: Oh, oh, huh! Yeah, okay, I’ll give you that.
63 00:06:54.090 ⇒ 00:06:59.339 Luke Daque: Can you try giving it to the matter more, you know. So like the the brain, for just matter more.
64 00:06:59.340 ⇒ 00:06:59.840 Trevor Cohen: Yeah, yeah.
65 00:06:59.840 ⇒ 00:07:02.590 Luke Daque: That’s that’s what I currently need at the moment.
66 00:07:11.520 ⇒ 00:07:16.069 Luke Daque: And sharing my screen, you can see.
67 00:07:22.660 ⇒ 00:07:24.140 Trevor Cohen: Oh, yeah, sorry. Hold on.
68 00:07:24.410 ⇒ 00:07:25.540 Trevor Cohen: Okay. I’m looking.
69 00:07:27.060 ⇒ 00:07:33.780 Luke Daque: So this was the problem that we have. We can’t create the table because, like it says, I don’t know if you can see.
70 00:07:33.780 ⇒ 00:07:34.770 Trevor Cohen: What does it say?
71 00:07:35.260 ⇒ 00:07:38.470 Luke Daque: Says you must select a project from the top action bar.
72 00:07:38.750 ⇒ 00:07:39.299 Trevor Cohen: Oh, you!
73 00:07:39.790 ⇒ 00:07:47.780 Luke Daque: It’s it’s this one. Yeah. So I can see we can see the project here in the explorer. But it doesn’t show up here.
74 00:07:47.780 ⇒ 00:07:50.870 Trevor Cohen: Okay. And you can see the data also. Right? You just can’t make a table.
75 00:07:51.830 ⇒ 00:08:01.309 Luke Daque: No, I we can’t see anything. This is, we can only see the synthetic data set. This is something that I created through a query, just to test we can. It looks like, but we can.
76 00:08:01.567 ⇒ 00:08:09.030 Trevor Cohen: I mean for me. That’s the only thing that’s in that repository, so that so it looks like you can see it, which is good, but let me
77 00:08:09.310 ⇒ 00:08:10.629 Trevor Cohen: give me one. Sec.
78 00:08:10.970 ⇒ 00:08:11.560 Luke Daque: Sure.
79 00:08:37.960 ⇒ 00:08:39.619 Trevor Cohen: Alright, so viewer.
80 00:08:43.360 ⇒ 00:08:50.420 Luke Daque: I guess, project. I don’t know what viewer let me.
81 00:08:55.520 ⇒ 00:08:59.290 Trevor Cohen: And then a query
82 00:09:02.300 ⇒ 00:09:02.970 Trevor Cohen: and.
83 00:09:21.910 ⇒ 00:09:22.590 Luke Daque: Right.
84 00:09:32.620 ⇒ 00:09:37.310 Trevor Cohen: Okay, try refreshing. Now see if that does anything. Oh, sorry! Which one are you using.
85 00:09:37.820 ⇒ 00:09:43.070 Luke Daque: I’m currently in the brain for chat matter more. But I can try switching to my.
86 00:09:43.070 ⇒ 00:09:44.180 Trevor Cohen: Yeah, try yours.
87 00:09:54.710 ⇒ 00:10:01.040 Trevor Cohen: Okay, were you? Were you able to see the the data set, though at least from yours.
88 00:10:02.730 ⇒ 00:10:03.929 Luke Daque: Let’s see.
89 00:10:10.150 ⇒ 00:10:11.040 Luke Daque: Oh, it’s.
90 00:10:19.450 ⇒ 00:10:21.170 Trevor Cohen: You might have to log out of the other one.
91 00:10:22.250 ⇒ 00:10:22.900 Luke Daque: Yeah.
92 00:10:31.470 ⇒ 00:10:35.659 Luke Daque: let me go to the the link again.
93 00:10:57.400 ⇒ 00:10:58.020 Trevor Cohen: Okay.
94 00:11:01.310 ⇒ 00:11:09.297 Trevor Cohen: okay? So I. So I just gave it the data viewer, and and viewer
95 00:11:10.280 ⇒ 00:11:15.260 Trevor Cohen: But let’s add something else. Hold on.
96 00:11:22.720 ⇒ 00:11:25.179 Trevor Cohen: I’m just gonna make it. I’m gonna give data owner.
97 00:11:26.350 ⇒ 00:11:28.020 Luke Daque: Okay, screen.
98 00:11:28.490 ⇒ 00:11:31.359 Trevor Cohen: Okay, try refreshing. Now see what happens.
99 00:11:41.700 ⇒ 00:11:45.510 Trevor Cohen: Alright. It’s still not there. But do you want to try creating a table? See if that works.
100 00:11:46.070 ⇒ 00:11:47.380 Luke Daque: Yeah, let’s try that
101 00:11:50.850 ⇒ 00:11:53.030 Luke Daque: just kind of using the Csv file.
102 00:11:56.760 ⇒ 00:11:58.670 Luke Daque: You’ll you’ll have to weird.
103 00:11:58.670 ⇒ 00:12:00.790 Trevor Cohen: You have to give it a table name. And
104 00:12:01.780 ⇒ 00:12:03.880 Trevor Cohen: yeah, and all the text schema.
105 00:12:04.370 ⇒ 00:12:11.940 Trevor Cohen: okay, got it? Got it? Okay? So let me let me just give it to the brain forge user, cause it the groups. When I give it to a group, I think it takes some time to update.
106 00:12:12.510 ⇒ 00:12:13.190 Luke Daque: Okay.
107 00:12:14.100 ⇒ 00:12:15.549 Trevor Cohen: Let me just do that.
108 00:12:26.040 ⇒ 00:12:28.810 Trevor Cohen: okay, so yeah, try that one.
109 00:12:30.900 ⇒ 00:12:32.770 Luke Daque: Okay, trying. I’m trying to log in.
110 00:12:41.570 ⇒ 00:12:42.370 Trevor Cohen: Yay!
111 00:12:42.890 ⇒ 00:12:45.890 Luke Daque: There’s matter more. AI. Is this the same thing.
112 00:12:45.890 ⇒ 00:12:49.729 Trevor Cohen: Yeah, yeah, that’s it’s matter where AI should be. And then.
113 00:12:50.180 ⇒ 00:12:53.470 Luke Daque: The project? Is this one the exemplary.
114 00:12:54.250 ⇒ 00:12:54.810 Luke Daque: It’s.
115 00:12:56.750 ⇒ 00:13:01.599 Trevor Cohen: No go to click on Mattermore. AI, see what happens.
116 00:13:04.030 ⇒ 00:13:06.079 Trevor Cohen: Just clicking on Mattermore. AI.
117 00:13:07.550 ⇒ 00:13:08.089 Luke Daque: That’s 1.
118 00:13:09.170 ⇒ 00:13:12.709 Trevor Cohen: Okay, interesting. Alright. So let me see. Maybe you need project level.
119 00:13:13.370 ⇒ 00:13:17.810 Luke Daque: Yeah, maybe maybe project viewer or something project related.
120 00:13:19.580 ⇒ 00:13:21.470 Trevor Cohen: God damn it! So annoying!
121 00:13:23.500 ⇒ 00:13:25.649 Trevor Cohen: I’m just gonna ask Chat Gbt real quick.
122 00:13:57.750 ⇒ 00:14:00.669 Luke Daque: Maybe viewer or project, viewer.
123 00:14:02.910 ⇒ 00:14:04.670 Trevor Cohen: There’s a project viewer.
124 00:14:06.650 ⇒ 00:14:14.780 Luke Daque: Roles viewer. Just viewer actually looks like in the I’m trying. I’m looking at the app in docs.
125 00:14:33.700 ⇒ 00:14:34.909 Trevor Cohen: Okay. Give me one. Sec.
126 00:14:51.720 ⇒ 00:14:53.539 Trevor Cohen: Okay. Alright. Try that.
127 00:14:58.950 ⇒ 00:15:00.290 Luke Daque: Try refresh
128 00:15:05.230 ⇒ 00:15:11.410 Luke Daque: extension. Just this one, just the just showing them my 1st project. So.
129 00:15:11.410 ⇒ 00:15:12.160 Trevor Cohen: Damn it.
130 00:15:13.660 ⇒ 00:15:17.860 Luke Daque: Yeah, we can. Yeah, I guess we can ask support or something.
131 00:15:34.690 ⇒ 00:15:38.300 Luke Daque: Wonder if we create a table using the my 1st project?
132 00:15:55.820 ⇒ 00:15:57.159 Luke Daque: Looks like it.
133 00:15:58.498 ⇒ 00:16:00.379 Trevor Cohen: Yeah. Let me try. One more thing.
134 00:16:05.970 ⇒ 00:16:07.000 Luke Daque: Yeah.
135 00:16:08.260 ⇒ 00:16:10.259 Luke Daque: For some reason this worked.
136 00:16:10.490 ⇒ 00:16:11.790 Luke Daque: So I
137 00:16:12.120 ⇒ 00:16:21.030 Luke Daque: I’m in the my 1st project project. And then I was able to create the list events, table under matter more, and analytics.
138 00:16:21.200 ⇒ 00:16:22.919 Trevor Cohen: Whoa! Wait! Whoa!
139 00:16:25.050 ⇒ 00:16:26.315 Luke Daque: Okay, it’s
140 00:16:27.040 ⇒ 00:16:31.952 Luke Daque: Because now, like, I’ve selected the my 1st project. And then if I create a table,
141 00:16:32.260 ⇒ 00:16:32.860 Trevor Cohen: No.
142 00:16:33.310 ⇒ 00:16:36.789 Luke Daque: Okay, like the the project. I guess.
143 00:16:37.000 ⇒ 00:16:38.990 Luke Daque: I guess this kind of works already.
144 00:16:39.200 ⇒ 00:16:45.290 Trevor Cohen: That works. Okay. Great. Alright. Let me delete the cause. I hold on.
145 00:16:47.290 ⇒ 00:16:49.749 Luke Daque: Other roles. I guess that you added.
146 00:16:49.990 ⇒ 00:16:52.960 Trevor Cohen: I just wanna delete the yeah refresh it again.
147 00:16:53.070 ⇒ 00:16:54.040 Trevor Cohen: Okay? And.
148 00:16:54.040 ⇒ 00:16:54.600 Luke Daque: Okay.
149 00:16:57.710 ⇒ 00:16:58.270 Trevor Cohen: Yeah.
150 00:16:58.270 ⇒ 00:17:01.639 Luke Daque: So I’ll be. I’ll be in here and let’s go. Okay, today.
151 00:17:01.640 ⇒ 00:17:02.610 Trevor Cohen: And that.
152 00:17:03.270 ⇒ 00:17:05.500 Luke Daque: Oh, I can see matter more.
153 00:17:05.500 ⇒ 00:17:12.039 Trevor Cohen: Yeah, I just, I don’t want you to be able to see that. So let me see if I can guarantee that. I mean, it’s okay. It’s okay. But
154 00:17:15.079 ⇒ 00:17:16.190 Trevor Cohen: give me one. Sec.
155 00:17:19.359 ⇒ 00:17:20.260 Trevor Cohen: The.
156 00:17:43.790 ⇒ 00:17:46.540 Luke Daque: So, yeah, it looks like it was able to create.
157 00:17:47.970 ⇒ 00:18:00.589 Trevor Cohen: Okay? Good. Alright. So that gives you what you need. I’m gonna see if I can just limit the data because we don’t have any client data sets there. But I just it would be best if you didn’t just see our whole client list. So I’m gonna figure that out, at least, for now, you guys can, it looks like you have what you need.
158 00:18:01.390 ⇒ 00:18:02.080 Luke Daque: Sure.
159 00:18:02.520 ⇒ 00:18:03.049 Trevor Cohen: Okay. Cool.
160 00:18:03.050 ⇒ 00:18:03.720 Luke Daque: Sounds good.
161 00:18:03.720 ⇒ 00:18:04.430 Trevor Cohen: Alright, great.
162 00:18:09.100 ⇒ 00:18:09.960 Luke Daque: Nice.
163 00:18:10.300 ⇒ 00:18:16.107 Luke Daque: Yeah, we’ll we’ll do that. We’ll continue adding all the tables and then viewing some joins.
164 00:18:16.560 ⇒ 00:18:19.459 Luke Daque: did we have the deep blue project set up, already.
165 00:18:20.850 ⇒ 00:18:22.639 Luke Daque: I think so right.
166 00:18:23.050 ⇒ 00:18:23.600 Trevor Cohen: I I mean.
167 00:18:23.600 ⇒ 00:18:24.919 Luke Daque: Or who would you?
168 00:18:25.170 ⇒ 00:18:31.059 Trevor Cohen: Which part I mean, I haven’t. I haven’t like started doing stuff in that yet. But yeah.
169 00:18:31.060 ⇒ 00:18:33.190 Luke Daque: Did you subject.
170 00:18:33.190 ⇒ 00:18:34.040 Trevor Cohen: Tractor.
171 00:18:34.530 ⇒ 00:18:35.309 Trevor Cohen: If you go to that.
172 00:18:35.310 ⇒ 00:18:35.800 Luke Daque: Activity.
173 00:18:35.800 ⇒ 00:18:37.040 Trevor Cohen: Projects, yeah.
174 00:18:37.040 ⇒ 00:18:43.109 Luke Daque: Gotcha, and this is Dbt cloud or Dbt. Core. You created this in DVD core.
175 00:18:44.190 ⇒ 00:18:44.960 Trevor Cohen: Say it again.
176 00:18:45.680 ⇒ 00:18:52.890 Luke Daque: Did you create the project using Dbt cloud? Or did you just use the Id, the core? One dbt, core.
177 00:18:52.890 ⇒ 00:18:57.250 Trevor Cohen: Yeah, it’s Dbt core, it’s running. And it’s it’s I’m posting it in Google Cloud Run.
178 00:18:58.010 ⇒ 00:18:59.190 Luke Daque: I see.
179 00:18:59.550 ⇒ 00:19:00.663 Luke Daque: Cool and
180 00:19:02.280 ⇒ 00:19:06.469 Luke Daque: Are you using any service account for this to connect to bigquery.
181 00:19:07.340 ⇒ 00:19:08.090 Trevor Cohen: Yes.
182 00:19:08.300 ⇒ 00:19:09.270 Trevor Cohen: Yeah.
183 00:19:10.368 ⇒ 00:19:17.270 Luke Daque: Should we be using that? If if so, then I guess we need access to the service account? Json, file right.
184 00:19:17.670 ⇒ 00:19:24.857 Trevor Cohen: No, you should use. No, you should use your own accounts. Connect to bigquery like you have. But the the point is that
185 00:19:25.610 ⇒ 00:19:39.080 Trevor Cohen: Dbt. Is hosted in Cloud Run, and that cloud run app itself like the the permissions, like access, is handled through Cloud Run. So I I created a dedicated service account for the Dbt. App. So.
186 00:19:39.080 ⇒ 00:19:39.670 Luke Daque: Gotcha.
187 00:19:39.670 ⇒ 00:19:44.850 Trevor Cohen: What happens is that like you can test any of the queries that you want to run
188 00:19:45.060 ⇒ 00:19:45.800 Trevor Cohen: against?
189 00:19:48.500 ⇒ 00:20:03.149 Trevor Cohen: Well, tell me what you need to do. Because basically, the way that it’s set up is that when for like production data, what’s gonna happen is that the airflow app is gonna run and like pull data from our sort of like our client sources. And then there’s an Api endpoint that triggers the Tpt app to run
190 00:20:03.370 ⇒ 00:20:08.220 Trevor Cohen: for a particular data set. And so, yeah.
191 00:20:09.090 ⇒ 00:20:09.600 Luke Daque: Gotcha.
192 00:20:10.110 ⇒ 00:20:10.860 Trevor Cohen: So I guess you you.
193 00:20:10.860 ⇒ 00:20:11.709 Luke Daque: Needed to create.
194 00:20:11.710 ⇒ 00:20:14.550 Trevor Cohen: Against against the synthetic data set right?
195 00:20:15.230 ⇒ 00:20:23.519 Luke Daque: Right? Yeah, like, if we can create a dev, I’ll have to like, create my profile, Saml, and connect it to the dev environment. I guess.
196 00:20:24.840 ⇒ 00:20:28.869 Luke Daque: Because it looks like the default. One would be the production one using the
197 00:20:31.360 ⇒ 00:20:34.510 Luke Daque: I guess. Well, it’s Oauth.
198 00:20:34.940 ⇒ 00:20:40.000 Luke Daque: So I guess we can use all of them. Just use all the the accounts that. And so yeah.
199 00:20:41.110 ⇒ 00:20:43.740 Luke Daque: should be fine. We can test it out.
200 00:20:44.280 ⇒ 00:20:44.945 Trevor Cohen: Okay.
201 00:20:47.730 ⇒ 00:20:54.139 Trevor Cohen: yeah, let me know I haven’t. I haven’t played around with Dbt a lot. So and I’m honestly like the permission stuff is a little confusing to me. But
202 00:20:54.577 ⇒ 00:21:05.139 Trevor Cohen: I know that, like the projects in Cloud Run, obviously has access and the way that it’s triggered again is through the Api endpoint which you guys don’t have access to, because you need, like a certain
203 00:21:06.130 ⇒ 00:21:06.640 Luke Daque: Right.
204 00:21:06.640 ⇒ 00:21:09.422 Trevor Cohen: Permission to to hit that Api endpoint.
205 00:21:10.060 ⇒ 00:21:20.189 Trevor Cohen: so I wouldn’t want to give you guys access to to run everything. But maybe, like this is, it would be a little complicated, annoying. But if there’s like we could
206 00:21:20.950 ⇒ 00:21:24.879 Trevor Cohen: make it so that I like, I limit your access to particular data sets.
207 00:21:25.440 ⇒ 00:21:31.469 Trevor Cohen: But yeah, just let me let me know exactly what you need when you like. When you get to that point, and I’ll we can figure it out.
208 00:21:32.420 ⇒ 00:21:34.594 Luke Daque: Sounds good. And
209 00:21:35.870 ⇒ 00:21:41.089 Luke Daque: so we’re not supposed to create any new data sets right for the Dbt models. We just.
210 00:21:41.940 ⇒ 00:21:46.529 Luke Daque: we just created models in the synthetic data, I guess, for now.
211 00:21:46.530 ⇒ 00:21:47.630 Trevor Cohen: Yeah, that’d be great.
212 00:21:48.030 ⇒ 00:21:49.140 Luke Daque: Okay. Cool.
213 00:21:50.940 ⇒ 00:21:52.340 Amber Lin: Sounds good. Okay.
214 00:21:52.910 ⇒ 00:21:55.460 Amber Lin: So Bigquery is working. I’ll tell Matthew I was just.
215 00:21:55.460 ⇒ 00:21:55.940 Trevor Cohen: That’s right.
216 00:21:55.940 ⇒ 00:22:19.389 Amber Lin: Back an email. So that’s working we’re gonna work on the joins is what I hear. And then Matthew responded, that he’s gonna help us figure out how what type of analytics we need. And probably I’ll schedule a meeting between between he and Utah this Wednesday. So we’ll have something for that, and I guess we can move forward from there.
217 00:22:20.230 ⇒ 00:22:21.610 Annie Yu: Awesome. That sounds great.
218 00:22:21.610 ⇒ 00:22:29.909 Annie Yu: A small thing. Look, I’m not sure if you saw my comment to your pr, I emerged that Pr, which
219 00:22:30.150 ⇒ 00:22:39.290 Annie Yu: I think it’s great. I think one thing that I noticed was that 2 of the data sets are not aligned with the kind of the column naming convention.
220 00:22:39.590 ⇒ 00:22:40.579 Annie Yu: but I think that.
221 00:22:40.580 ⇒ 00:22:43.619 Amber Lin: Do we need? Sorry. Do we need Trevor to be here for this?
222 00:22:44.126 ⇒ 00:22:45.139 Amber Lin: Probably not.
223 00:22:45.140 ⇒ 00:22:47.990 Amber Lin: Okay, Trevor, you’re free to go. Don’t wanna take up more of your time.
224 00:22:47.990 ⇒ 00:22:49.839 Trevor Cohen: Cool guys. Alright. I’ll talk to you later.
225 00:22:49.840 ⇒ 00:22:52.560 Luke Daque: Alrighty, bye-bye, thanks.
226 00:22:53.150 ⇒ 00:23:09.547 Annie Yu: Yeah, look, so I yeah. I left some comments which I don’t think we need to fix anything from the Github side. So, but the naming convention there for that list. Events and list call records. I think we can probably manually
227 00:23:10.380 ⇒ 00:23:11.409 Annie Yu: edit the column.
228 00:23:12.173 ⇒ 00:23:13.700 Luke Daque: File, yeah.
229 00:23:13.700 ⇒ 00:23:23.379 Annie Yu: Yeah, yeah, either in Csv or bigquery. And I can help you with that. But let me know, like, what’s the best way? Do I like, download the Csv. And then
230 00:23:24.460 ⇒ 00:23:33.689 Annie Yu: change them and then upload them again. Or should I try to do that in bigquery once you load them? I don’t know what works the best.
231 00:23:35.240 ⇒ 00:23:39.259 Luke Daque: I guess I can do that because it’s only just these 2.
232 00:23:39.380 ⇒ 00:23:41.709 Annie Yu: Yeah, those 2 and the others are.
233 00:23:41.710 ⇒ 00:23:42.130 Luke Daque: Anyway.
234 00:23:42.130 ⇒ 00:23:43.359 Annie Yu: All perfect. Yeah.
235 00:23:43.360 ⇒ 00:23:48.079 Luke Daque: And basically, we just want this to be where was that?
236 00:23:48.320 ⇒ 00:23:56.000 Luke Daque: Just like we just need this to be lower case. And then snake case. Basically, right? So I underscore call
237 00:23:56.210 ⇒ 00:23:58.661 Luke Daque: underscore you id something like that.
238 00:23:59.070 ⇒ 00:23:59.420 Annie Yu: Best, call.
239 00:23:59.420 ⇒ 00:24:00.279 Luke Daque: I’m not sure. Yeah.
240 00:24:00.280 ⇒ 00:24:11.369 Annie Yu: Yeah, but I think they want the daytime to be one word. So daytime and time zone to be one word. So don’t break out date and time, and don’t break out time and zone.
241 00:24:12.520 ⇒ 00:24:14.790 Annie Yu: Yeah. So this will be created.
242 00:24:14.790 ⇒ 00:24:15.120 Luke Daque: Great.
243 00:24:15.449 ⇒ 00:24:16.109 Annie Yu: The score.
244 00:24:16.110 ⇒ 00:24:26.040 Luke Daque: Or daytime. Or yeah, maybe if you want to, you can download the Csv file from the Github repo and just send it to me like with the update
245 00:24:26.447 ⇒ 00:24:29.709 Luke Daque: without putting them up. As a P
246 00:24:29.710 ⇒ 00:24:39.239 Luke Daque: without, yeah, yeah, or yeah, or, yeah, we can. Also, you can also do that like, create the Pr so that we I can up. We have the updated file here
247 00:24:39.560 ⇒ 00:24:41.219 Luke Daque: in your time.
248 00:24:41.870 ⇒ 00:24:48.300 Annie Yu: Yeah, sure, I think I’ll do both because I’ll send send them your way so you can start, and then I’ll.
249 00:24:48.300 ⇒ 00:24:49.080 Luke Daque: Yeah.
250 00:24:49.080 ⇒ 00:24:51.180 Annie Yu: You prioritize like updating this.
251 00:24:51.290 ⇒ 00:24:53.380 Annie Yu: But okay, I’ll do that.
252 00:24:54.040 ⇒ 00:24:55.799 Luke Daque: Sounds good. Yeah, we can do that.
253 00:24:56.740 ⇒ 00:25:02.070 Luke Daque: So yeah, I think we have everything needed now to proceed.
254 00:25:03.210 ⇒ 00:25:09.273 Amber Lin: Are we doing anything in Dbt. Or is that I? I totally did not get that part.
255 00:25:09.630 ⇒ 00:25:12.899 Luke Daque: That’s actually a good question, because we can do.
256 00:25:13.410 ⇒ 00:25:23.879 Luke Daque: There’s 2 ways we can do the joints. We can either do the joints directly in bigquery, for now, as a view or something like that, or we can just we can use Dbt to do it that way we can.
257 00:25:24.040 ⇒ 00:25:31.510 Luke Daque: It’s like, I think 2 birds in one stone like we can. We were able to test dbt, and then do the joins as well. So so.
258 00:25:31.510 ⇒ 00:25:34.430 Amber Lin: Sure do. We have access to Dvt and everything.
259 00:25:35.360 ⇒ 00:25:37.130 Luke Daque: We’ll have to test it out.
260 00:25:37.130 ⇒ 00:25:37.930 Amber Lin: Okay. Okay.
261 00:25:38.210 ⇒ 00:25:39.169 Luke Daque: Yeah, cause, like.
262 00:25:39.170 ⇒ 00:25:45.449 Amber Lin: I mean, ultimately we’ll we’ll know what sequel codes we need. I guess if we get
263 00:25:45.640 ⇒ 00:25:51.230 Amber Lin: dbt, it just saves us a step of eventually uploading everything to Dbt. As well. Right.
264 00:25:53.140 ⇒ 00:25:55.039 Luke Daque: I. So I hope so. Yeah.
265 00:25:55.040 ⇒ 00:25:55.770 Amber Lin: Okay. Okay.
266 00:25:55.770 ⇒ 00:25:59.789 Luke Daque: As as long as, like the actual data is the same.
267 00:26:00.290 ⇒ 00:26:06.690 Amber Lin: Format like, for example, these unnested fields. We are like storing them as Jason Fields here.
268 00:26:07.350 ⇒ 00:26:13.610 Amber Lin: but we don’t really know what the actual data would be. Maybe they’re already nested here unnested, or something.
269 00:26:13.610 ⇒ 00:26:14.290 Luke Daque: Oh, I didn’t
270 00:26:14.290 ⇒ 00:26:22.540 Luke Daque: to change some some of the sequel queries, but or for this we’ll have to write SQL. Queries that will
271 00:26:22.950 ⇒ 00:26:24.410 Luke Daque: unnecess these.
272 00:26:25.760 ⇒ 00:26:47.099 Amber Lin: I see. How long do you guys think like a initial step would take like, are are there stages to figuring out like different phases to figuring out the joins, because ultimately would love to show something earlier. I know this will take a while, but like, is there each phases to getting this done, and how long would that take.
273 00:26:51.530 ⇒ 00:26:54.350 Luke Daque: Do we have tickets for that.
274 00:26:54.755 ⇒ 00:26:59.620 Amber Lin: Nope, it is not. Yeah. We don’t have specific tickets for that.
275 00:27:00.550 ⇒ 00:27:05.610 Amber Lin: We have, like figuring out the joins, which is, wait, where is it?
276 00:27:07.640 ⇒ 00:27:10.910 Amber Lin: Define your own logic am in progress.
277 00:27:12.410 ⇒ 00:27:16.619 Amber Lin: I think I don’t know if that one’s done like bigquery schema dimensions.
278 00:27:18.960 ⇒ 00:27:23.090 Amber Lin: because now we have bigquery. So I’m gonna mute. Move. That done.
279 00:27:31.950 ⇒ 00:27:37.620 Amber Lin: yeah, I moved it. Is this done like, I don’t know if this is the right ticket.
280 00:27:43.010 ⇒ 00:27:45.729 Luke Daque: So I guess this is Dbt. Related.
281 00:27:45.730 ⇒ 00:27:47.129 Amber Lin: Oh, okay, I see.
282 00:27:47.130 ⇒ 00:27:51.840 Luke Daque: So. And this would be just the join logic.
283 00:27:53.340 ⇒ 00:27:56.719 Luke Daque: Yeah, I think this is fine splitting these into 2.
284 00:27:57.170 ⇒ 00:27:57.870 Luke Daque: And
285 00:27:59.590 ⇒ 00:28:07.620 Luke Daque: yeah, this will have to include, like test out that maybe we can have a sub issue for this, which is test out the dpt connection
286 00:28:07.960 ⇒ 00:28:10.930 Luke Daque: like. But this doesn’t have to be blocker.
287 00:28:10.930 ⇒ 00:28:12.140 Amber Lin: I see, I see.
288 00:28:12.330 ⇒ 00:28:14.919 Luke Daque: Okay, we cannot connect to Dbt.
289 00:28:15.913 ⇒ 00:28:16.410 Amber Lin: Hmm.
290 00:28:16.410 ⇒ 00:28:21.660 Luke Daque: Yeah, we’ll just. We’ll just continue with the join logic using views for now, just so we can.
291 00:28:21.660 ⇒ 00:28:22.799 Luke Daque: Okay, keep on.
292 00:28:22.800 ⇒ 00:28:23.310 Luke Daque: That’s good.
293 00:28:23.310 ⇒ 00:28:33.879 Amber Lin: Sounds good. Let’s leave Dbt until later. Let’s get some. Join logic to them. First.st let them check if it makes sense, and then dbt, I bet we.
294 00:28:33.880 ⇒ 00:28:34.420 Luke Daque: Okay.
295 00:28:34.420 ⇒ 00:28:37.089 Amber Lin: Figure out like parallel in parallel.
296 00:28:37.780 ⇒ 00:28:38.160 Luke Daque: That’s good.
297 00:28:38.160 ⇒ 00:28:39.209 Amber Lin: Let’s do 2.
298 00:28:39.210 ⇒ 00:28:39.770 Luke Daque: Make, sense.
299 00:28:39.770 ⇒ 00:28:46.580 Amber Lin: Okay, I’ll move the test. Dbt, connection to and to do this cycle and.
300 00:28:46.790 ⇒ 00:28:56.350 Luke Daque: Yeah, yeah, I guess this one. You can moved it down the the synthetic data set.
301 00:28:57.090 ⇒ 00:28:58.360 Amber Lin: Okay.
302 00:28:58.989 ⇒ 00:29:03.210 Amber Lin: Dbt, repo. We have access to it right already.
303 00:29:05.630 ⇒ 00:29:06.789 Luke Daque: Yes, we do.
304 00:29:06.790 ⇒ 00:29:14.429 Amber Lin: Sounds good join logic and progress. 4 dimensions. Dbt, connection. Okay?
305 00:29:15.410 ⇒ 00:29:16.340 Amber Lin: All right.
306 00:29:17.097 ⇒ 00:29:21.480 Amber Lin: Anything about the victory staging models is that something that we’re gonna do.
307 00:29:21.480 ⇒ 00:29:23.379 Luke Daque: This is Dvt. Related for this one.
308 00:29:23.380 ⇒ 00:29:28.320 Amber Lin: Oh, gosh, okay. DVD, okay, this is blocked.
309 00:29:34.080 ⇒ 00:29:35.880 Amber Lin: okay, sounds good.
310 00:29:37.870 ⇒ 00:29:38.560 Luke Daque: Cool.
311 00:29:38.840 ⇒ 00:29:39.810 Amber Lin: Alrighty.
312 00:29:41.660 ⇒ 00:29:43.740 Amber Lin: Thank you. All. Talk to you soon.
313 00:29:45.170 ⇒ 00:29:46.720 Luke Daque: See you. Bye-bye.
314 00:29:46.980 ⇒ 00:29:49.630 Amber Lin: Okay, bye.
315 00:29:51.310 ⇒ 00:29:54.080 Amber Lin: Oh, Annie, are you talking? You’re muted.
316 00:29:54.940 ⇒ 00:29:59.965 Annie Yu: Oh, yeah, no, I’m I’m just asking Amber if you have like a 15 min day.
317 00:30:00.280 ⇒ 00:30:01.250 Amber Lin: Dashboard! Yes.
318 00:30:01.250 ⇒ 00:30:01.800 Annie Yu: Yeah, yeah.
319 00:30:01.800 ⇒ 00:30:04.219 Amber Lin: Okay, I’ll call you. I’ll call you in a bit, I think.
320 00:30:04.740 ⇒ 00:30:06.690 Amber Lin: but they might not show up.
321 00:30:06.690 ⇒ 00:30:07.095 Annie Yu: Okay.
322 00:30:07.500 ⇒ 00:30:09.340 Amber Lin: Alrighty! Bye-bye.