Meeting Title: Eden __ Brainforge - Python script review with Bo Date: 2025-01-09 Meeting participants: Luke Daque, Nicolas Sucari, Bo Yoon
WEBVTT
1 00:00:14.460 ⇒ 00:00:15.350 Nicolas Sucari: Hello!
2 00:00:18.130 ⇒ 00:00:19.030 Luke Daque: And you go.
3 00:00:23.460 ⇒ 00:00:28.569 Nicolas Sucari: You tired already this time today, I think. Yes, right. It’s too late for you.
4 00:00:28.570 ⇒ 00:00:36.549 Luke Daque: Yeah, a bit, especially considering I was like migrating a lot of yeah queries, it was pretty
5 00:00:38.110 ⇒ 00:00:42.000 Luke Daque: yeah, pretty tough, really.
6 00:00:42.530 ⇒ 00:00:43.650 Nicolas Sucari: Really long day.
7 00:00:44.050 ⇒ 00:00:44.980 Luke Daque: Yeah.
8 00:00:48.410 ⇒ 00:00:53.310 Nicolas Sucari: The idea here is both to just share a quick walkthrough of the code.
9 00:00:53.530 ⇒ 00:00:56.369 Nicolas Sucari: He he shared the code. You have you seen it?
10 00:00:56.370 ⇒ 00:00:59.700 Luke Daque: Yeah, yeah. But I haven’t really like taken time.
11 00:01:00.370 ⇒ 00:01:02.079 Luke Daque: Understand it. But I I just.
12 00:01:02.080 ⇒ 00:01:02.610 Nicolas Sucari: Thank you.
13 00:01:03.020 ⇒ 00:01:04.440 Luke Daque: Took a quick look.
14 00:01:07.090 ⇒ 00:01:08.760 Nicolas Sucari: Ideally with that.
15 00:01:08.780 ⇒ 00:01:09.810 Luke Daque: Far.
16 00:01:09.810 ⇒ 00:01:10.400 Nicolas Sucari: Yeah.
17 00:01:11.350 ⇒ 00:01:12.430 Luke Daque: Yeah. Yeah. Go ahead.
18 00:01:13.940 ⇒ 00:01:16.410 Nicolas Sucari: Ideally with that, we’ll just try to
19 00:01:16.610 ⇒ 00:01:22.800 Nicolas Sucari: replicate. Maybe it’s just like asking Chatgpt to translate that code into
20 00:01:23.741 ⇒ 00:01:27.619 Nicolas Sucari: Dbt file, or something like a SQL. File right.
21 00:01:27.620 ⇒ 00:01:34.940 Luke Daque: Yeah, but it looks like it might be a little more complicated like the other.
22 00:01:39.520 ⇒ 00:01:41.970 Luke Daque: So Hello!
23 00:01:42.420 ⇒ 00:01:43.679 Luke Daque: Can you hear me now?
24 00:01:44.190 ⇒ 00:01:46.540 Nicolas Sucari: I can hear you now. Yeah, I think I lost you for a while.
25 00:01:46.540 ⇒ 00:01:50.919 Luke Daque: Yeah, yeah, sorry. I think it’s my Internet connection. But
26 00:01:51.350 ⇒ 00:01:56.499 Luke Daque: yeah, there’s 2 files there. The main by, and the function.
27 00:01:56.670 ⇒ 00:01:59.210 Luke Daque: The main one might be pretty
28 00:01:59.340 ⇒ 00:02:02.594 Luke Daque: easy to do in in Dbt, but the other one
29 00:02:03.130 ⇒ 00:02:07.270 Luke Daque: like it, has like a lot of probably.
30 00:02:07.270 ⇒ 00:02:07.820 Bo Yoon: Hey, guys.
31 00:02:08.580 ⇒ 00:02:10.040 Luke Daque: Oh, Hi Hi moon!
32 00:02:11.170 ⇒ 00:02:11.490 Nicolas Sucari: Table.
33 00:02:12.120 ⇒ 00:02:13.140 Bo Yoon: Good afternoon.
34 00:02:13.170 ⇒ 00:02:13.980 Bo Yoon: How’s it going.
35 00:02:14.800 ⇒ 00:02:16.569 Luke Daque: Yeah. Doing. Well, how are you?
36 00:02:17.290 ⇒ 00:02:21.540 Bo Yoon: Good, good do you guys get a chance to look at the code that I sent you.
37 00:02:23.200 ⇒ 00:02:24.460 Nicolas Sucari: We were just talking about that.
38 00:02:25.810 ⇒ 00:02:26.170 Bo Yoon: Okay.
39 00:02:26.170 ⇒ 00:02:27.450 Luke Daque: Took a look at it. Now.
40 00:02:29.480 ⇒ 00:02:37.990 Bo Yoon: Yeah, I don’t. I don’t think it’s really complicated. It’s just we just gotta design a way to build like a pipeline connected with the.
41 00:02:38.280 ⇒ 00:02:40.290 Bo Yoon: It’s a SQL. Server that we have.
42 00:02:41.300 ⇒ 00:02:44.209 Bo Yoon: so I don’t know what. What do you guys think.
43 00:02:47.680 ⇒ 00:02:57.489 Luke Daque: yeah. Another way. We can probably do this if we can’t convert this into sequel, like, like, 1st thing that we will want to try is to see if we can
44 00:02:57.910 ⇒ 00:03:07.240 Luke Daque: use sequel for this transformation. But if it’s a little bit too complicated for sequel, then maybe we’ll just leave it as a python script. And maybe
45 00:03:07.740 ⇒ 00:03:14.529 Luke Daque: yeah, maybe maybe we can even run the script in a cloud function, a Google cloud function
46 00:03:14.700 ⇒ 00:03:20.530 Luke Daque: and just schedule it like maybe once a day, like we like you mentioned yesterday, and then it would
47 00:03:20.850 ⇒ 00:03:23.630 Luke Daque: still output a table in bigquery
48 00:03:23.750 ⇒ 00:03:25.470 Luke Daque: that way. You don’t have to like
49 00:03:25.640 ⇒ 00:03:28.260 Luke Daque: download Csv files or stuff like that.
50 00:03:29.040 ⇒ 00:03:30.610 Bo Yoon: Hmm, yeah, that’s.
51 00:03:30.610 ⇒ 00:03:41.972 Nicolas Sucari: Yeah, the goal, the the goal here is not to do any manual work once this is done. So look if like, if that’s a possibility. And we are not. You know, we don’t have like
52 00:03:42.450 ⇒ 00:03:51.480 Nicolas Sucari: a way of translating this python script into Dbt, okay, I mean, we should look into that Google script way.
53 00:03:53.800 ⇒ 00:04:03.249 Bo Yoon: Yeah. So I I think the the data processing script that I sent you. That’s I think that’s doable in sequel. I don’t. I don’t see why not?
54 00:04:04.170 ⇒ 00:04:09.549 Bo Yoon: The the only problem for that would be that. We’ll have to do that by product.
55 00:04:10.310 ⇒ 00:04:11.230 Bo Yoon: So.
56 00:04:11.780 ⇒ 00:04:13.600 Luke Daque: That’s 1 of the problems.
57 00:04:16.430 ⇒ 00:04:17.100 Bo Yoon: Yeah,
58 00:04:18.903 ⇒ 00:04:22.939 Luke Daque: So like we’re expecting to have one table per product.
59 00:04:25.780 ⇒ 00:04:27.520 Bo Yoon: Yeah, that that’s what I’m thinking. But
60 00:04:28.060 ⇒ 00:04:35.110 Bo Yoon: there’s a good way we can do it. not sure.
61 00:04:36.970 ⇒ 00:04:45.579 Luke Daque: Yeah, okay, no, no worries. Well, I’ll take a stab at this. I have to. Take some time to see. I’ll look into
62 00:04:46.060 ⇒ 00:04:46.800 Luke Daque: the script.
63 00:04:46.800 ⇒ 00:04:47.150 Bo Yoon: Yeah.
64 00:04:47.150 ⇒ 00:04:51.490 Luke Daque: And see if we can convert this to SQL, and then, yeah, so.
65 00:04:51.490 ⇒ 00:04:51.890 Bo Yoon: And.
66 00:04:52.366 ⇒ 00:04:56.789 Luke Daque: Yeah, if I have any like, we have any further questions regarding like, how we can.
67 00:04:58.511 ⇒ 00:05:03.939 Luke Daque: Materialize this by product and stuff like that. Then yeah, we’ll go from there.
68 00:05:04.830 ⇒ 00:05:08.330 Bo Yoon: Yeah, sure. Yeah, just just let me know if you need any help.
69 00:05:08.630 ⇒ 00:05:09.365 Bo Yoon: But
70 00:05:10.600 ⇒ 00:05:21.350 Bo Yoon: we’re on the same page that the that the main python code that I send you. That’s that’s not doable in Dvt, right? That’s just. It’s just gotta be in python right?
71 00:05:24.070 ⇒ 00:05:24.430 Nicolas Sucari: Nor.
72 00:05:24.430 ⇒ 00:05:25.110 Luke Daque: Same thing, again.
73 00:05:25.110 ⇒ 00:05:33.690 Nicolas Sucari: I I think no, I think that one is something that we can do in Python. The the difficult one is the one of all of the functions, I think.
74 00:05:34.140 ⇒ 00:05:34.750 Nicolas Sucari: Alright. Look!
75 00:05:34.750 ⇒ 00:05:43.770 Bo Yoon: Yeah, I I’m not sure if there’s a way we can convert the functions in sequel, because I’m using all these libraries from python.
76 00:05:44.000 ⇒ 00:05:45.369 Bo Yoon: Never heard that.
77 00:05:45.370 ⇒ 00:05:45.909 Luke Daque: There is.
78 00:05:45.950 ⇒ 00:05:46.470 Bo Yoon: That’s.
79 00:05:46.470 ⇒ 00:05:55.049 Luke Daque: We’re doing like cater plots. And yeah, this is basically creating the the graph right? Like, the
80 00:05:55.050 ⇒ 00:05:56.770 Luke Daque: yeah, creating the graph.
81 00:05:57.797 ⇒ 00:06:06.570 Bo Yoon: Using an exponential fit to basically predict the future as well. That’s yeah. It’s kinda hmm.
82 00:06:09.220 ⇒ 00:06:20.550 Nicolas Sucari: Is this just like all of these python is for that by visualizations that you’re doing or like, have you considered using mixed panel, or anything else.
83 00:06:21.340 ⇒ 00:06:29.615 Bo Yoon: Mix panel. I I have no experience with mixed panel, but when I when I had a conversation with Robert, he told me that
84 00:06:31.330 ⇒ 00:06:34.930 Bo Yoon: it might be doable, but it’s it’ll be kind of hard to do it.
85 00:06:35.570 ⇒ 00:06:36.060 Nicolas Sucari: Okay.
86 00:06:36.060 ⇒ 00:06:37.160 Bo Yoon: And mix panel.
87 00:06:38.100 ⇒ 00:06:41.090 Bo Yoon: Yeah? And and the the other problem is that
88 00:06:41.460 ⇒ 00:06:48.730 Bo Yoon: this is not it for this app? I’m gonna add a lot of more more features in it. It’s just this is just the beginning of it.
89 00:06:49.350 ⇒ 00:06:55.220 Bo Yoon: So I I’m pretty sure that it’s gonna it’s gonna get a lot more complicated
90 00:06:55.810 ⇒ 00:06:57.439 Bo Yoon: than what we have right now.
91 00:06:58.920 ⇒ 00:07:02.359 Bo Yoon: So a custom app.
92 00:07:02.660 ⇒ 00:07:08.410 Bo Yoon: so building a custom app deploying it on Google Cloud, that I think that’s the best solution that we can
93 00:07:08.820 ⇒ 00:07:10.910 Bo Yoon: we can have at the moment.
94 00:07:13.550 ⇒ 00:07:15.915 Bo Yoon: unless you guys have another idea.
95 00:07:17.440 ⇒ 00:07:20.540 Nicolas Sucari: No, I think we. We just need to look deeper into
96 00:07:21.400 ⇒ 00:07:23.180 Nicolas Sucari: these files and see what we can do. There.
97 00:07:24.530 ⇒ 00:07:27.050 Bo Yoon: Yeah, sure, sure.
98 00:07:31.110 ⇒ 00:07:39.600 Bo Yoon: yeah. And and also for now I’m, I built other models as well, but I didn’t have time to convert that in
99 00:07:39.860 ⇒ 00:07:45.249 Bo Yoon: like in an in an app. Yet. So I’m presenting that in an excel format.
100 00:07:46.300 ⇒ 00:07:49.699 Bo Yoon: So I’m gonna share that with you as well, but
101 00:07:50.340 ⇒ 00:07:57.559 Bo Yoon: our end goal will be transforming that excel excel, file, format, file into the cloud as well.
102 00:08:00.090 ⇒ 00:08:00.780 Luke Daque: Okay.
103 00:08:01.770 ⇒ 00:08:08.933 Bo Yoon: Yeah, sure, yeah. Do you guys have any other questions for that?
104 00:08:11.400 ⇒ 00:08:15.110 Nicolas Sucari: I don’t think so. Not not me, Luke.
105 00:08:15.110 ⇒ 00:08:20.759 Luke Daque: Yeah, I I think not at the moment, but once I take a look at this more.
106 00:08:20.990 ⇒ 00:08:26.299 Luke Daque: Once I get more time to take a look at this, and maybe I’ll have some other questions. I’ll just probably
107 00:08:27.315 ⇒ 00:08:28.070 Luke Daque: contact.
108 00:08:28.070 ⇒ 00:08:31.659 Bo Yoon: Yeah. Just send me a slack message. Yeah, sounds great.
109 00:08:32.070 ⇒ 00:08:32.450 Nicolas Sucari: Perfect.
110 00:08:32.450 ⇒ 00:08:35.649 Bo Yoon: Yeah. Alright. Then, yeah, thank you so much for your time.
111 00:08:36.480 ⇒ 00:08:38.160 Nicolas Sucari: Thank you both as well. Thank you, Luke.
112 00:08:38.169 ⇒ 00:08:38.539 Bo Yoon: Yeah.
113 00:08:38.539 ⇒ 00:08:39.919 Nicolas Sucari: Talk later. Bye-bye.
114 00:08:40.929 ⇒ 00:08:41.609 Bo Yoon: See you guys.