Meeting Title: Brainforge ETL Pipeline Debugging Sync Date: 2025-07-10 Meeting participants: Awaish Kumar, Vishnu
WEBVTT
1 00:01:02.350 ⇒ 00:01:04.239 Vishnu: Oh, hi avish! Good morning!
2 00:01:04.650 ⇒ 00:01:05.379 Awaish Kumar: Hello!
3 00:01:07.261 ⇒ 00:01:12.490 Awaish Kumar: Good morning. Can you please share your updates?
4 00:01:13.110 ⇒ 00:01:22.980 Vishnu: Yes, I wish I’m just sharing my screen so you can. I I ran it. So I basically started working on this late at night yesterday, and I ran into certain issues. I need your help here.
5 00:01:24.640 ⇒ 00:01:27.189 Vishnu: Oh, I’ll just share my screen.
6 00:01:31.050 ⇒ 00:01:37.510 Vishnu: So okay, so this is the Etl pipeline.
7 00:01:39.370 ⇒ 00:01:40.035 Vishnu: And
8 00:01:42.505 ⇒ 00:01:48.884 Vishnu: these are the Api keys I’ve given. These are the Snowflake credentials I’ve given. And then
9 00:01:50.850 ⇒ 00:01:57.239 Vishnu: when I run, oh, that still does.
10 00:02:00.510 ⇒ 00:02:03.529 Vishnu: Yeah, but you don’t have to run the local machine.
11 00:02:04.600 ⇒ 00:02:07.490 Awaish Kumar: Just just run your script as a.
12 00:02:08.120 ⇒ 00:02:10.840 Awaish Kumar: And then a local as a local file
13 00:02:11.080 ⇒ 00:02:19.279 Awaish Kumar: and then push it right? So you don’t need to like, go open a local texture.
14 00:02:20.720 ⇒ 00:02:23.339 Awaish Kumar: Okay, Web. But don’t worry about.
15 00:02:23.340 ⇒ 00:02:31.164 Vishnu: It was so. I was running into an error like the error said, unable to like 2 factor authentication with
16 00:02:31.760 ⇒ 00:02:32.250 Vishnu: snow.
17 00:02:32.250 ⇒ 00:02:34.350 Awaish Kumar: Yeah. And I was not able to.
18 00:02:35.080 ⇒ 00:02:36.590 Awaish Kumar: Okay. So
19 00:02:37.150 ⇒ 00:02:55.098 Awaish Kumar: so wait. Like, that’s an issue. We actually have started having 2 factor authentication for all the users. So you are not able to connect with your regular user through programmatically. So I I don’t know if I think, have you tried a pat like
20 00:02:56.421 ⇒ 00:02:58.978 Awaish Kumar: personal access tokens in Snowflake.
21 00:02:59.650 ⇒ 00:03:02.499 Awaish Kumar: like. But I’m I’m creating a service
22 00:03:03.491 ⇒ 00:03:13.119 Awaish Kumar: user account which we are going to use for snowflake but before I create
23 00:03:13.951 ⇒ 00:03:25.849 Awaish Kumar: so when I will create that, I will let you know. But in the meantime, if you want to try creating personal access token like, you can go and slow flick in your user.
24 00:03:26.480 ⇒ 00:03:32.240 Awaish Kumar: And you can create your own personal access token in the users profile, I think.
25 00:03:32.710 ⇒ 00:03:41.660 Awaish Kumar: And yeah, like this one, no active tokens
26 00:03:43.690 ⇒ 00:03:46.480 Awaish Kumar: can you create one? Is there any way to.
27 00:03:47.900 ⇒ 00:03:48.750 Vishnu: Oh, I.
28 00:03:48.750 ⇒ 00:03:49.269 Awaish Kumar: Yeah. Jen.
29 00:03:49.270 ⇒ 00:03:50.340 Vishnu: Generate new.
30 00:03:53.980 ⇒ 00:03:55.900 Awaish Kumar: You can see.
31 00:03:59.630 ⇒ 00:04:00.310 Vishnu: Also.
32 00:04:00.862 ⇒ 00:04:09.090 Vishnu: So yeah, what role do I select? And because I was trying to create database, I was not able to create database here. So.
33 00:04:09.920 ⇒ 00:04:15.120 Awaish Kumar: You. Why would you like to create a new database just.
34 00:04:15.120 ⇒ 00:04:15.880 Vishnu: I thought.
35 00:04:15.880 ⇒ 00:04:17.519 Awaish Kumar: Just use the one we have.
36 00:04:17.790 ⇒ 00:04:24.270 Awaish Kumar: No, no, we already. We don’t need to create another database like we already have these in the raw.
37 00:04:27.550 ⇒ 00:04:35.969 Awaish Kumar: raw, there is already raw, raw data. And then if for for your script you can store in, maybe
38 00:04:36.080 ⇒ 00:04:36.935 Awaish Kumar: on
39 00:04:41.280 ⇒ 00:04:46.220 Awaish Kumar: in the like broad march. So
40 00:04:46.600 ⇒ 00:04:51.580 Awaish Kumar: yeah, like, you can store in prodmarts, for example, database.
41 00:04:54.160 ⇒ 00:05:05.069 Awaish Kumar: So I know what you’re doing. You’re just moving data. Yeah, you can keep in the here, like in the raw database, create a new data schema or data set called clockify.
42 00:05:05.600 ⇒ 00:05:07.619 Awaish Kumar: And inside of that create a table.
43 00:05:08.310 ⇒ 00:05:09.380 Awaish Kumar: Understand?
44 00:05:10.340 ⇒ 00:05:17.190 Vishnu: Okay? So under raw, I have to create a new schema called Clockify, and.
45 00:05:17.190 ⇒ 00:05:30.169 Awaish Kumar: For a source. Claude, like you can see, like we have a schema for different source, one for Github, one for linear post hoc, quickbooks slack. You can create one for clockify as well.
46 00:05:30.690 ⇒ 00:05:31.840 Vishnu: Okay. Okay.
47 00:05:32.130 ⇒ 00:05:35.180 Awaish Kumar: And then I know that you will have multiple tables.
48 00:05:36.450 ⇒ 00:05:38.180 Vishnu: Inside that I put multiple tabs.
49 00:05:38.180 ⇒ 00:05:40.690 Awaish Kumar: Like if you if you click on Github.
50 00:05:40.970 ⇒ 00:05:48.079 Awaish Kumar: this is the data set. If you click on that, get on the Github Schema, you can see multiple tables under that.
51 00:05:48.660 ⇒ 00:05:50.160 Awaish Kumar: Okay, if you
52 00:05:51.930 ⇒ 00:06:02.479 Awaish Kumar: so you can create multiple labels. Yeah, like that. Like, it depends if you need multiple labels or not. Or if you need just one. But that’s that’s like how we are going to structure here.
53 00:06:02.620 ⇒ 00:06:03.420 Awaish Kumar: Data.
54 00:06:03.420 ⇒ 00:06:09.728 Vishnu: Okay, okay, so I just want to make sure that. Okay, I have permissions for that.
55 00:06:10.100 ⇒ 00:06:19.330 Awaish Kumar: So just try with a personal access token. If it works, it works. If if it does not, just let me know I have to create a service
56 00:06:19.650 ⇒ 00:06:22.770 Awaish Kumar: account, and I will let you know Wednesday, and I will let you know.
57 00:06:24.500 ⇒ 00:06:27.809 Vishnu: Okay, personal access token. Okay, I’ll do that.
58 00:06:27.810 ⇒ 00:06:33.260 Awaish Kumar: Try with. Try with Pat, and also let me know if it works for you. If it works
59 00:06:33.698 ⇒ 00:06:50.121 Awaish Kumar: then it’s okay. If if it does not, I will create a service account, and we we will work with that one. And secondly, and if, like, I know, you are blocked, if you personal access token will not work, you will be blocked on this task by me.
60 00:06:50.450 ⇒ 00:06:51.180 Vishnu: Okay.
61 00:06:51.180 ⇒ 00:06:56.079 Awaish Kumar: So just leave it here and start working on other task which I
62 00:06:56.580 ⇒ 00:06:58.420 Awaish Kumar: I given to you, like, you know, yesterday.
63 00:06:58.420 ⇒ 00:06:59.750 Vishnu: Okay, so yeah.
64 00:06:59.750 ⇒ 00:07:01.410 Awaish Kumar: That’s what he’s done right now.
65 00:07:01.410 ⇒ 00:07:02.070 Vishnu: Task.
66 00:07:02.650 ⇒ 00:07:05.389 Vishnu: So I started like the repo and.
67 00:07:05.390 ⇒ 00:07:12.539 Awaish Kumar: This one i i sent you on slack the one I sent you on slack. That’s the I told you. That’s the highest priority.
68 00:07:13.600 ⇒ 00:07:14.700 Vishnu: This one right.
69 00:07:14.700 ⇒ 00:07:15.630 Awaish Kumar: Yes.
70 00:07:17.142 ⇒ 00:07:20.969 Vishnu: Can once. Can you just brief me about it? Please.
71 00:07:21.450 ⇒ 00:07:23.829 Awaish Kumar: So like this, like this is a
72 00:07:24.080 ⇒ 00:07:31.380 Awaish Kumar: Api, like like the clock. If I, this is another tool called operating, the name of the tool is operating.
73 00:07:31.510 ⇒ 00:07:49.730 Awaish Kumar: and I have provided you the docs on top. There’s a link to its documentation like you can explore the like Api endpoints for operating right? So operating is a tool which basically where you can
74 00:07:50.770 ⇒ 00:07:56.840 Awaish Kumar: like, provide details for your employees and their hours and manage like
75 00:07:57.090 ⇒ 00:07:59.609 Awaish Kumar: part time, full time employees. So it’s them
76 00:07:59.890 ⇒ 00:08:05.140 Awaish Kumar: for the employee management kind of tool for operations like so.
77 00:08:05.140 ⇒ 00:08:05.790 Vishnu: Okay.
78 00:08:06.641 ⇒ 00:08:35.480 Awaish Kumar: Here we we have like for each week for each client. What people are working, or how many hours and things like that. So we basically use that tool for that for planning of the efforts. And so documentation on the top is just about operating what it is. Then there is Api key. So it’s basically you. You provide this key to use it. Then. I just mentioned that you have to take care of like. When you make some requests.
79 00:08:35.669 ⇒ 00:09:01.900 Awaish Kumar: request can fail. There might be pagination. There may be rate limiting on the Api endpoint, so just take care of that like adding retries, or try except blocks and things like that. So that’s just a suggestion for implementation. And in the comment, you can see I have listed the endpoints you need to hit, so you don’t need to hit all the endpoints. Just hit the ones I share here. So we have 1, 2, 3, 4, 5, 6, 7.
80 00:09:03.080 ⇒ 00:09:03.740 Vishnu: Okay.
81 00:09:03.740 ⇒ 00:09:10.000 Awaish Kumar: What you are basically going to do is create a spreadsheet, create a spreadsheet
82 00:09:11.180 ⇒ 00:09:14.513 Awaish Kumar: and with your brainforge account,
83 00:09:15.760 ⇒ 00:09:18.039 Awaish Kumar: And give it a
84 00:09:18.500 ⇒ 00:09:44.659 Awaish Kumar: and maybe like for the Google we will have. We have, I think, a service account, so I will share you share the keys with you, so you can. Using that service account. You are going to access this sheet. So inside of that sheet so response of each of these endpoint is going to be a tab in the spreadsheet. So in the spreadsheet you can have multiple like 7 different worksheets.
85 00:09:46.190 ⇒ 00:09:46.890 Vishnu: Okay.
86 00:09:46.890 ⇒ 00:09:51.550 Awaish Kumar: And where each worksheet will have response from each of the endpoint. So the
87 00:09:51.650 ⇒ 00:09:56.529 Awaish Kumar: the response from work effort list, the 1st endpoint goes to one
88 00:09:57.010 ⇒ 00:09:59.910 Awaish Kumar: worksheet in the in that spreadsheet
89 00:10:01.032 ⇒ 00:10:20.897 Awaish Kumar: the time entry list is going to be another tab, or whatever you call worksheet or tab, or whatever in the spreadsheet. So there’s single spreadsheet with multiple tabs, and each tab will have data for single endpoint, and we will have 7 sheets that way or or tabs. And
90 00:10:21.760 ⇒ 00:10:23.840 Awaish Kumar: basically, that’s it. That’s the task.
91 00:10:24.320 ⇒ 00:10:29.159 Awaish Kumar: So yeah, you need, like, if 1st of all, to hit that
92 00:10:29.320 ⇒ 00:10:35.317 Awaish Kumar: to get the data from Api, you need Api key, which I’ve already shared with you. Secondly, you are going to need
93 00:10:36.317 ⇒ 00:10:44.580 Awaish Kumar: the Service County to store the data in Google sheets. So I I will just share it with you as well. And that’s all you need.
94 00:10:46.920 ⇒ 00:11:02.100 Awaish Kumar: And you can try. Basically, when you click on this endpoints. You can see like they have very nice documentation. You can see how to make a call to this Api like in the python using python. So you can, you can basically just copy.
95 00:11:02.810 ⇒ 00:11:11.680 Awaish Kumar: you have to just click on like the curl command on top. This is the bit simple response. Right? So if you scroll up a little bit
96 00:11:12.892 ⇒ 00:11:33.430 Awaish Kumar: you can. See, like you can provide the token real token here from the Api key. I have told you. Just provide it here. And you can see the actual responses. And when you click on python, you can basically see how to make that request in python, and then just move it to the dexter so you
97 00:11:33.670 ⇒ 00:11:43.039 Awaish Kumar: you can get help like there are a lot of scripts in the Dexter pipeline repository like the one clockified or snowflake, or things like that. So you can just
98 00:11:43.250 ⇒ 00:11:44.369 Awaish Kumar: see that
99 00:11:44.600 ⇒ 00:11:53.919 Awaish Kumar: how we are basically building a pipeline in Dexter, because we just need few decorators for that which are useful for building, a
100 00:11:55.250 ⇒ 00:12:05.409 Awaish Kumar: for for to like to let the dexter know that it’s 1 of the pipeline. I did not just. I order any python script, but that’s all you just like when you see an
101 00:12:05.530 ⇒ 00:12:12.391 Awaish Kumar: another script. Let’s let’s see how we are calling everything, and you can do that. And
102 00:12:13.520 ⇒ 00:12:15.480 Awaish Kumar: and for the Api
103 00:12:16.070 ⇒ 00:12:23.620 Awaish Kumar: code you can see in the documentation for Api keys in in one pass and the service count.
104 00:12:23.790 ⇒ 00:12:28.290 Awaish Kumar: I will see where it is like. Maybe it is also in the one pass. But yeah, I have to
105 00:12:28.580 ⇒ 00:12:30.389 Awaish Kumar: look for it, and that’s all.
106 00:12:30.390 ⇒ 00:12:37.460 Vishnu: I have 1 1 doubt you said to use service account here. Right? So I’m I’m
107 00:12:37.460 ⇒ 00:12:38.140 Vishnu: yeah. Thank you.
108 00:12:38.140 ⇒ 00:12:41.449 Vishnu: Like service account for Google sheets.
109 00:12:42.190 ⇒ 00:12:53.880 Awaish Kumar: Yeah, you you create your. So how are like, how are you going to create a access? Your Google sheets with python? Right? So it’s not not a
110 00:12:54.010 ⇒ 00:13:10.089 Awaish Kumar: public Google spreadsheet. It will be a private one, and you need something to access it. So what you are going to do is you share your spreadsheet with the service account in the service account. When I share you service account key in that you will find an email.
111 00:13:10.610 ⇒ 00:13:15.379 Awaish Kumar: So share this spreadsheet with that email address. So.
112 00:13:15.380 ⇒ 00:13:15.790 Vishnu: Okay.
113 00:13:15.790 ⇒ 00:13:19.900 Awaish Kumar: By doing that. The service account, then, can access your spreadsheet.
114 00:13:20.160 ⇒ 00:13:38.860 Awaish Kumar: and in the clockified same same pipeline clock like like. We are exactly doing the same thing. We already have a script. I told you that clockified to Google Sheet is is there already, and you can just see that like how they are uploading the data
115 00:13:39.020 ⇒ 00:13:42.189 Awaish Kumar: and which key there, basically, I don’t have to give you the key.
116 00:13:42.340 ⇒ 00:13:47.819 Awaish Kumar: because you can also see that, like whatever key they are using. Just use that one.
117 00:13:49.370 ⇒ 00:13:50.030 Vishnu: Okay.
118 00:13:50.260 ⇒ 00:14:01.710 Awaish Kumar: Okay, so just see in the script what environment variable they are using which key they are using just just reference that and and utilize the same environment variable.
119 00:14:04.750 ⇒ 00:14:08.749 Awaish Kumar: If you go scroll, scroll down, or we can just check it. Now.
120 00:14:09.760 ⇒ 00:14:10.639 Vishnu: Yes. Okay.
121 00:14:10.640 ⇒ 00:14:12.279 Awaish Kumar: Scroll down a little bit.
122 00:14:13.840 ⇒ 00:14:16.649 Awaish Kumar: Yeah, it’s it’s just getting the data right?
123 00:14:16.990 ⇒ 00:14:20.610 Awaish Kumar: So just scroll scroll down on the loading part of it.
124 00:14:22.440 ⇒ 00:14:23.970 Awaish Kumar: Yeah, this one.
125 00:14:24.465 ⇒ 00:14:35.519 Awaish Kumar: So, yeah, these are the account. So it’s it’s called brain for Gcp service account. So just use that service account. And the the values. You can see if you log into Dexter Cloud.
126 00:14:36.080 ⇒ 00:14:36.550 Vishnu: I’m sure.
127 00:14:36.550 ⇒ 00:14:42.830 Awaish Kumar: Do that. And then if you go to environment variable, you can exactly see this environment variable. And then you copy
128 00:14:43.810 ⇒ 00:14:47.360 Awaish Kumar: basically. So if you want to test locally, you can copy that.
129 00:14:48.090 ⇒ 00:14:53.980 Awaish Kumar: Yeah, this one copy that and put it in your environment. In your local environment.
130 00:14:55.410 ⇒ 00:14:55.840 Vishnu: Okay.
131 00:14:55.840 ⇒ 00:14:56.390 Awaish Kumar: Yep.
132 00:14:59.270 ⇒ 00:15:07.180 Vishnu: Alright. So okay, I I’ll refer this, and I will make the changes. And I’ll do that. So for now I’m just gonna try.
133 00:15:07.180 ⇒ 00:15:08.299 Awaish Kumar: Basically this is gonna.
134 00:15:08.300 ⇒ 00:15:09.129 Vishnu: Access token.
135 00:15:09.130 ⇒ 00:15:12.259 Awaish Kumar: This script is just using one endpoint.
136 00:15:12.370 ⇒ 00:15:15.374 Awaish Kumar: and when our script is little bit more
137 00:15:17.120 ⇒ 00:15:20.030 Awaish Kumar: yeah, it will be be little bit more using all the time
138 00:15:20.030 ⇒ 00:15:38.660 Awaish Kumar: to handle all these 7 endpoints. So we just have to iterate over it, or somehow like, and maybe for each of them like. Just try as as like, try to organize your code as best as you can, and then I can review it, and we can see.
139 00:15:39.450 ⇒ 00:15:40.510 Vishnu: Okay, okay.
140 00:15:40.930 ⇒ 00:15:55.930 Vishnu: okay, so with respect to this, this code so these were some tests, I’m gonna like, delete these. And I’m gonna upload this like, I’ll I’ll create the Pr. Once like I’ll try the personal.
141 00:15:55.930 ⇒ 00:16:02.529 Awaish Kumar: Yeah, for now try personal access token. If it works, you can push it otherwise. Just you can
142 00:16:02.660 ⇒ 00:16:07.429 Awaish Kumar: push your code on the Github, but just leave it there
143 00:16:07.560 ⇒ 00:16:22.430 Awaish Kumar: like once. Once I create a service account. When we are done with that I will let you know you have make your changes and then create a Pr like you can push it, but it will be in your one of your branches. We don’t need to create a Pr. Until it’s done, because.
144 00:16:24.226 ⇒ 00:16:24.820 Vishnu: Okay. Yeah.
145 00:16:24.820 ⇒ 00:16:53.720 Awaish Kumar: Yeah. So just just don’t. Don’t spend much time on this. Just try, Pat. Maybe spend an hour or so like maybe 30 min. Just try with Pat. Just 1st of all, just try establish a connection. If if you can establish a connection or not using Pat programmatically, if if you can. Then you can go ahead if you can’t just stop and start working on another
146 00:16:54.030 ⇒ 00:16:55.020 Awaish Kumar: like the ticket.
147 00:16:55.020 ⇒ 00:16:56.960 Vishnu: Yeah. Okay, yes, I’ll do that.
148 00:16:59.710 ⇒ 00:17:00.300 Vishnu: Oh.
149 00:17:00.300 ⇒ 00:17:00.660 Awaish Kumar: Okay.
150 00:17:00.660 ⇒ 00:17:05.800 Vishnu: Alright, avish! So I’ll I’ll get this done by today evening, and I’ll try.
151 00:17:05.800 ⇒ 00:17:06.300 Awaish Kumar: To complete.
152 00:17:06.300 ⇒ 00:17:07.889 Vishnu: The other tasks also here.
153 00:17:09.040 ⇒ 00:17:10.130 Awaish Kumar: Yeah, so yesterday.
154 00:17:10.609 ⇒ 00:17:11.199 Vishnu: These are the.
155 00:17:11.200 ⇒ 00:17:16.310 Awaish Kumar: Like, yeah, like the like. The top one is the highest priority. Just.
156 00:17:16.319 ⇒ 00:17:16.819 Vishnu: Is, the.
157 00:17:16.819 ⇒ 00:17:21.369 Awaish Kumar: Try to finish up today, and we can see the other ones. After that.
158 00:17:21.970 ⇒ 00:17:29.070 Vishnu: Okay. Okay, fine. Yeah, yeah. That will be done today. Okay, thanks. Avish. Bye-bye.
159 00:17:29.070 ⇒ 00:17:30.109 Awaish Kumar: Thank you. Bye.