Meeting Title: Brainforge Final Interview Date: 2026-03-12 Meeting participants: Gilbert Adjei, Awaish Kumar, Demilade Agboola, Uttam Kumaran
WEBVTT
1 00:01:44.870 ⇒ 00:01:45.650 Awaish Kumar: Hi.
2 00:01:49.540 ⇒ 00:01:50.400 Gilbert Adjei: Hi.
3 00:01:50.630 ⇒ 00:01:51.590 Gilbert Adjei: Nice.
4 00:01:52.070 ⇒ 00:01:54.260 Awaish Kumar: Hi, how to pronounce your name?
5 00:01:54.760 ⇒ 00:01:55.690 Gilbert Adjei: Gobbitz.
6 00:01:56.420 ⇒ 00:01:57.390 Awaish Kumar: Gilbert?
7 00:01:57.550 ⇒ 00:01:58.250 Gilbert Adjei: Yes.
8 00:01:59.030 ⇒ 00:01:59.800 Awaish Kumar: Okay.
9 00:02:05.530 ⇒ 00:02:08.970 Awaish Kumar: Yeah, let’s wait for… Fathers.
10 00:02:09.590 ⇒ 00:02:10.330 Gilbert Adjei: Awesome.
11 00:02:30.690 ⇒ 00:02:33.510 Awaish Kumar: Oh, what is… Where are you located?
12 00:02:35.810 ⇒ 00:02:37.050 Gilbert Adjei: I’m currently in Ghana.
13 00:02:38.920 ⇒ 00:02:39.770 Awaish Kumar: Okay.
14 00:02:40.050 ⇒ 00:02:40.660 Gilbert Adjei: Yeah.
15 00:03:25.270 ⇒ 00:03:26.140 Uttam Kumaran: Hello.
16 00:03:28.630 ⇒ 00:03:30.449 Gilbert Adjei: Hi. Hi, everyone.
17 00:03:31.230 ⇒ 00:03:32.140 Uttam Kumaran: Hello.
18 00:03:35.640 ⇒ 00:03:36.600 Gilbert Adjei: Great.
19 00:03:37.550 ⇒ 00:03:42.150 Gilbert Adjei: Alright, so, thank you all so much for…
20 00:03:42.960 ⇒ 00:03:45.900 Gilbert Adjei: Joining in. I’m trying to turn on my camera.
21 00:03:46.620 ⇒ 00:03:47.409 Uttam Kumaran: All good.
22 00:03:56.490 ⇒ 00:03:57.310 Gilbert Adjei: Alright.
23 00:03:57.550 ⇒ 00:04:00.789 Gilbert Adjei: Anyway… You know, sticking out.
24 00:04:01.230 ⇒ 00:04:07.120 Gilbert Adjei: a minute. Anyway, so, thank you so much for making it,
25 00:04:07.260 ⇒ 00:04:12.459 Gilbert Adjei: for this walkthrough. So, I’m also going to share my screen.
26 00:04:12.690 ⇒ 00:04:16.240 Gilbert Adjei: And then please let me know if you’re able to see my screen as well.
27 00:04:16.970 ⇒ 00:04:17.490 Uttam Kumaran: Okay.
28 00:04:17.990 ⇒ 00:04:18.649 Gilbert Adjei: Okay.
29 00:04:23.630 ⇒ 00:04:25.130 Uttam Kumaran: Yes, we can see it.
30 00:04:25.630 ⇒ 00:04:34.850 Gilbert Adjei: Awesome, great. So, this is going to be a walkthrough of an end-to-end demonstration of…
31 00:04:35.260 ⇒ 00:04:39.399 Gilbert Adjei: The take-home assignments, and then how it’s worked.
32 00:04:39.600 ⇒ 00:04:43.009 Gilbert Adjei: From… right from data ingestion app onto…
33 00:04:43.110 ⇒ 00:04:49.159 Gilbert Adjei: their CI process, and then their decisions that governed their choices of
34 00:04:49.380 ⇒ 00:04:51.399 Gilbert Adjei: Pools and things of that sort.
35 00:04:51.660 ⇒ 00:04:58.420 Gilbert Adjei: And… We see the way forward. So, thus… Solution is…
36 00:04:59.030 ⇒ 00:05:02.829 Gilbert Adjei: production-style, implementation, that’s in just…
37 00:05:03.060 ⇒ 00:05:10.809 Gilbert Adjei: the data that was shared, which is the JSNL exports, and then it exports them into,
38 00:05:11.080 ⇒ 00:05:12.990 Gilbert Adjei: saves them into Postgres.
39 00:05:13.110 ⇒ 00:05:23.409 Gilbert Adjei: and transformed them with dbt, and then the whole solution is also… governed clearly with CI automations.
40 00:05:23.670 ⇒ 00:05:30.839 Gilbert Adjei: And… Things of that sort. So, the data ingestion was done by using AirByte.
41 00:05:31.020 ⇒ 00:05:37.110 Gilbert Adjei: Again, transformations, DBT, and then the, RBSC,
42 00:05:37.530 ⇒ 00:05:43.890 Gilbert Adjei: we’ll look at it, and then also we’ll finally look at the GitHub Actions.
43 00:05:44.070 ⇒ 00:05:46.050 Gilbert Adjei: And within the…
44 00:05:47.130 ⇒ 00:05:54.570 Gilbert Adjei: README had also included how to set these things up, and the whole system is being Dockerized.
45 00:05:54.690 ⇒ 00:05:57.330 Gilbert Adjei: So… so that it can be.
46 00:05:57.480 ⇒ 00:06:05.969 Gilbert Adjei: reproducible in various environments. Now, talking about the AirBytes, which is a data ingestion tool.
47 00:06:06.110 ⇒ 00:06:13.490 Gilbert Adjei: what I did was to… use ABCTL, which essentially Crete.
48 00:06:14.060 ⇒ 00:06:22.819 Gilbert Adjei: a Kubernetes in Docker, right? So, it uses kind to create a Kubernetes cluster in Docker.
49 00:06:23.090 ⇒ 00:06:28.549 Gilbert Adjei: And then… ensures that all the different parts within, AirBytes
50 00:06:28.740 ⇒ 00:06:34.670 Gilbert Adjei: has been installed and then set up correctly. So, referencing the
51 00:06:35.260 ⇒ 00:06:44.570 Gilbert Adjei: documentation, it gives… outlines how to do that. So, I set it up this way, and again, the prerequisites for
52 00:06:44.840 ⇒ 00:06:46.360 Gilbert Adjei: does projectors.
53 00:06:46.530 ⇒ 00:06:53.039 Gilbert Adjei: for you to have Docker environment running, for you to have Python, and then the rest are all within there.
54 00:06:53.290 ⇒ 00:06:57.369 Gilbert Adjei: read me. So, as you can see.
55 00:06:57.720 ⇒ 00:07:06.180 Gilbert Adjei: This is running on port 8000, it has… They’re different files.
56 00:07:06.310 ⇒ 00:07:12.320 Gilbert Adjei: being set up as connections. I’ll go into the code base,
57 00:07:12.970 ⇒ 00:07:26.509 Gilbert Adjei: briefly, but to show… demonstrate how it works, we have the sources, which have these files, and then the destination, which is Postgres, which is also running in Docker.
58 00:07:26.680 ⇒ 00:07:36.319 Gilbert Adjei: Right, so the… Once… the connection has been established between the files and then the Postgres database, which is
59 00:07:36.560 ⇒ 00:07:41.989 Gilbert Adjei: running in the Docker Compose file, so this is the, whole…
60 00:07:42.620 ⇒ 00:07:47.120 Gilbert Adjei: setup of the project. So, we have the Dockerm.
61 00:07:47.250 ⇒ 00:07:56.730 Gilbert Adjei: compose file, which has the Postgres running, with, pgadmin. PGAdmin, in this case, is
62 00:07:57.140 ⇒ 00:08:01.779 Gilbert Adjei: for you to graphically query your database, I…
63 00:08:01.880 ⇒ 00:08:11.500 Gilbert Adjei: It’s also… I spend this app, and then, it runs on ports 50-50. So, this is how the various services,
64 00:08:12.160 ⇒ 00:08:16.070 Gilbert Adjei: interact in the Docker Compose file. Now.
65 00:08:16.720 ⇒ 00:08:25.299 Gilbert Adjei: going back to AirByte, so I’ve shown the destination and then the sources. So, once… You want to…
66 00:08:25.630 ⇒ 00:08:30.040 Gilbert Adjei: move data from the source to the database, so…
67 00:08:30.160 ⇒ 00:08:38.440 Gilbert Adjei: Over here, I just clicked on one source, and this process is repeatable, right? It’s… you can always reproduce this,
68 00:08:38.850 ⇒ 00:08:47.279 Gilbert Adjei: in the environment, so I just click… you can click on sync now, and then you can see that the data will move from the source, which are the
69 00:08:47.400 ⇒ 00:08:53.129 Gilbert Adjei: files that were shared into the Postgres… Postgres database.
70 00:08:53.230 ⇒ 00:08:56.020 Gilbert Adjei: Now to… Validate that.
71 00:08:56.550 ⇒ 00:09:01.560 Gilbert Adjei: thus… Has indeed gone into the database.
72 00:09:01.780 ⇒ 00:09:05.099 Gilbert Adjei: The… once the database is initialized.
73 00:09:05.620 ⇒ 00:09:10.369 Gilbert Adjei: there are certain scripts that are run, right? So, these scripts
74 00:09:11.120 ⇒ 00:09:17.720 Gilbert Adjei: give the various initiatives the schemas, and then the rules. So the four…
75 00:09:18.070 ⇒ 00:09:22.820 Gilbert Adjei: different rules that exist. The developer, air bytes.
76 00:09:22.960 ⇒ 00:09:28.580 Gilbert Adjei: DDT, and then the DI. And then for each of them, they have what they do.
77 00:09:28.710 ⇒ 00:09:33.560 Gilbert Adjei: Right? So, as defined within the, README. So…
78 00:09:33.700 ⇒ 00:09:41.899 Gilbert Adjei: AirByte, for raw air bytes, it’s responsible for creating or ingesting the data.
79 00:09:42.310 ⇒ 00:09:50.770 Gilbert Adjei: into the Postgres database, right? So, it creates that, and then the schema raw.
80 00:09:51.370 ⇒ 00:10:01.510 Gilbert Adjei: over here, as we can see, creates… once it’s successful… it’s a successful push, it moves the data from Airbyte and then creates these three.
81 00:10:01.610 ⇒ 00:10:04.699 Gilbert Adjei: tables stored under the raw schema.
82 00:10:04.820 ⇒ 00:10:11.879 Gilbert Adjei: So… This is how, the… raw schema.
83 00:10:12.920 ⇒ 00:10:14.660 Gilbert Adjei: using the AirBytes.
84 00:10:15.250 ⇒ 00:10:17.150 Gilbert Adjei: User. Create.
85 00:10:17.740 ⇒ 00:10:20.720 Gilbert Adjei: The various tables within the schema.
86 00:10:20.840 ⇒ 00:10:22.950 Gilbert Adjei: Now, when that happens.
87 00:10:23.190 ⇒ 00:10:32.940 Gilbert Adjei: the data, we can validate that it’s all here, right? So, we can query it, as I mentioned earlier, on… to graphically query your data. I explained that the Docker
88 00:10:33.160 ⇒ 00:10:39.650 Gilbert Adjei: container of PGAdmin so that you can run various commands, like.
89 00:10:39.840 ⇒ 00:10:44.680 Gilbert Adjei: If you want to take a quick peek of the data, so let’s just say… I want to…
90 00:10:44.680 ⇒ 00:10:45.480 Uttam Kumaran: Yeah, perfect.
91 00:10:45.480 ⇒ 00:10:52.010 Gilbert Adjei: Yeah, so… It creates… the raw data.
92 00:10:52.290 ⇒ 00:10:55.230 Gilbert Adjei: And then, airbtes.
93 00:10:55.600 ⇒ 00:10:58.999 Gilbert Adjei: Automatically and add this, useful…
94 00:10:59.380 ⇒ 00:11:10.970 Gilbert Adjei: field to it, right, so that you can, if you want to perform any other things on it. But this raw data in itself comes, according to the metadata.
95 00:11:11.120 ⇒ 00:11:13.180 Gilbert Adjei: Primary keys, and things of that sort.
96 00:11:13.360 ⇒ 00:11:14.250 Gilbert Adjei: wherever I’m.
97 00:11:14.800 ⇒ 00:11:21.300 Gilbert Adjei: So, once that has been ingested, the pipeline uses DDT.
98 00:11:21.680 ⇒ 00:11:33.850 Gilbert Adjei: To perform various transformations on the data. So, the DBT, we have the staging, which… Essentially,
99 00:11:34.340 ⇒ 00:11:40.540 Gilbert Adjei: the raw data, It performs, some casting on it, and then,
100 00:11:41.670 ⇒ 00:11:52.349 Gilbert Adjei: ensures that renaming and things of that sort, initially, the data came within, uppercases, so the raw… the staging environment takes the raw
101 00:11:52.730 ⇒ 00:12:00.260 Gilbert Adjei: Data from the raw schema, and then also, performs these light transformations on them.
102 00:12:00.550 ⇒ 00:12:08.349 Gilbert Adjei: And within DBT, I added, various tests to it, so that the various models are
103 00:12:08.790 ⇒ 00:12:18.179 Gilbert Adjei: properly tested, and then you don’t have surprises when we move into production. So, how we run that, I have defined them within
104 00:12:18.320 ⇒ 00:12:19.290 Gilbert Adjei: Damn.
105 00:12:19.730 ⇒ 00:12:24.509 Gilbert Adjei: read me. However, I can do a quick show of how. So…
106 00:12:24.760 ⇒ 00:12:34.890 Gilbert Adjei: to run the staging targets, do dbt run dash dash target, and then the staging. And this will run all the models.
107 00:12:35.000 ⇒ 00:12:38.000 Gilbert Adjei: We do… Damn.
108 00:12:38.690 ⇒ 00:12:53.170 Gilbert Adjei: folder, and it’s good to mention that I also created a macro. So, macro is more like a function, and then it helps you to reuse code, so that I don’t have to be repeating myself
109 00:12:53.230 ⇒ 00:13:03.730 Gilbert Adjei: within the dbt environment. Now, when that is done, So, the…
110 00:13:05.540 ⇒ 00:13:09.350 Gilbert Adjei: staging that we have a dbt user, which…
111 00:13:09.480 ⇒ 00:13:13.740 Gilbert Adjei: It has the ability to create tables, it has the ability to
112 00:13:13.820 ⇒ 00:13:24.249 Gilbert Adjei: also modify, the staging environments, and then also, yes. But however, the data that is within the RAW, that is ingested within RAW,
113 00:13:24.280 ⇒ 00:13:39.729 Gilbert Adjei: the raw schema. DBT… a dbt user cannot make modifications to that. It is immutable, because we want dbt to focus on the transformations, and then we want the data that comes from the various sources to also remain the same.
114 00:13:39.820 ⇒ 00:13:46.160 Gilbert Adjei: So, that’s… S, with regards to the DBT
115 00:13:46.270 ⇒ 00:13:54.490 Gilbert Adjei: But it’s also, since the data came in with JSON fields and all that, I created an intermediate,
116 00:13:55.080 ⇒ 00:13:57.350 Gilbert Adjei: a stage which… Export.
117 00:13:57.350 ⇒ 00:14:01.070 Uttam Kumaran: Yeah, I guess my question is, talk us through, like,
118 00:14:01.270 ⇒ 00:14:06.309 Uttam Kumaran: Why… why each of the different, you know, stages and models?
119 00:14:07.910 ⇒ 00:14:16.889 Gilbert Adjei: Okay, sure. So, I… added the different, stages, so I have the raw.
120 00:14:17.010 ⇒ 00:14:22.969 Gilbert Adjei: And then the staging intermediate and the mat. The raw, in itself takes in the…
121 00:14:23.440 ⇒ 00:14:29.049 Gilbert Adjei: data from the source, right from the source, and then ingest into the raw schema. They’re staging,
122 00:14:29.250 ⇒ 00:14:34.219 Gilbert Adjei: Performs light transformations, like add… renaming the fields, and the
123 00:14:39.050 ⇒ 00:14:40.980 Gilbert Adjei: The intermediate stage.
124 00:14:41.410 ⇒ 00:14:53.690 Gilbert Adjei: handles the JSON side of the data, right? So the data came in in the form of nested JSONs and all that, so that transformations happen within the intermediate stage. And then the math.
125 00:14:54.590 ⇒ 00:15:01.960 Gilbert Adjei: performs the business side… business case. So, if there are metrics that need to be generated, the maths
126 00:15:02.400 ⇒ 00:15:15.049 Gilbert Adjei: handles the analytical, business cases. So, that was why I, separated the folder structure in this way. So…
127 00:15:16.430 ⇒ 00:15:27.479 Awaish Kumar: Yeah, my follow-up question is, like, why we kept the JSONB, like, JSON columns as JSONB in the staging, instead of splitting it into separate columns.
128 00:15:30.450 ⇒ 00:15:34.399 Gilbert Adjei: So… In the staging,
129 00:15:35.070 ⇒ 00:15:42.809 Gilbert Adjei: I… I wanted to maintain, things as they were, because in the… within the staging, I just wanted to…
130 00:15:42.940 ⇒ 00:15:50.800 Gilbert Adjei: Do minimal transformations like casting, and then also renaming of fields of that sort.
131 00:15:51.150 ⇒ 00:15:55.340 Gilbert Adjei: And within the intermediates, stage, our radar.
132 00:15:55.930 ⇒ 00:16:02.540 Gilbert Adjei: work on the JSON field, so that was why I kept the staging field, the staging stage in that
133 00:16:02.750 ⇒ 00:16:03.500 Gilbert Adjei: We.
134 00:16:09.360 ⇒ 00:16:15.720 Gilbert Adjei: Okay, so moving on, the tests… Run.
135 00:16:16.460 ⇒ 00:16:24.339 Gilbert Adjei: when you do a GBT test, and then, staging, and then it will run there.
136 00:16:24.560 ⇒ 00:16:32.109 Gilbert Adjei: YAML files, so ensuring that the customer ID for it to be a primary case should not be null, should also be unique.
137 00:16:32.210 ⇒ 00:16:35.340 Gilbert Adjei: And then, glass?
138 00:16:37.040 ⇒ 00:16:42.209 Awaish Kumar: So, like, while you were writing these tests, did you come across any…
139 00:16:42.500 ⇒ 00:16:49.099 Awaish Kumar: Business-related tests, which, like, instead of just the standard ones, not all unique.
140 00:16:50.470 ⇒ 00:16:53.869 Gilbert Adjei: Okay, so, within the…
141 00:16:55.080 ⇒ 00:17:01.540 Gilbert Adjei: business side of things, I wanted to create it… I wanted it to be minimal, right? So I…
142 00:17:02.360 ⇒ 00:17:07.310 Gilbert Adjei: Ensuring that the grain is one rule per order.
143 00:17:07.510 ⇒ 00:17:15.170 Gilbert Adjei: I created that for the standard ones, yes. And then I also… Okay.
144 00:17:15.170 ⇒ 00:17:17.990 Awaish Kumar: You can name, for example, if you have anything in mind.
145 00:17:18.119 ⇒ 00:17:19.990 Awaish Kumar: Like, what else we can create?
146 00:17:21.520 ⇒ 00:17:31.000 Gilbert Adjei: Yes, so, for example, we can, for example, if there are… we want to see fulfillment status, we want to see, whether
147 00:17:31.620 ⇒ 00:17:35.550 Gilbert Adjei: It meets a particular threshold, we can add that to
148 00:17:35.610 ⇒ 00:17:53.620 Gilbert Adjei: the test as well, right? So, you can always include a test based on the business case, and then let’s say a total amount… I’m expecting the amount not to be greater than, say, $100,000. I can also add that expression
149 00:17:53.840 ⇒ 00:17:56.149 Gilbert Adjei: As a form of a test in here.
150 00:17:58.610 ⇒ 00:17:59.300 Awaish Kumar: Okay.
151 00:18:00.160 ⇒ 00:18:16.960 Gilbert Adjei: Yes, and then, aside these ones to… if the data were to be, coming from a live source, you would want to test for freshness of the data, right, so that the data wouldn’t be stale and all that. However, this particular data is…
152 00:18:17.100 ⇒ 00:18:20.800 Gilbert Adjei: Just… they are just files, and so,
153 00:18:21.320 ⇒ 00:18:32.720 Gilbert Adjei: Yeah, but if they are live data, you want to test for data freshness and ensure that your data isn’t steal. Yes, so those are some of the things.
154 00:18:32.720 ⇒ 00:18:38.789 Awaish Kumar: Like, all of these models are materialized as views. Is there any reason for that?
155 00:18:39.800 ⇒ 00:18:45.420 Gilbert Adjei: Yes, so for standardization, of layers,
156 00:18:45.630 ⇒ 00:18:48.519 Gilbert Adjei: I would prefer that they are…
157 00:18:48.690 ⇒ 00:18:55.060 Gilbert Adjei: Saved, or they are materialized as views, and then for… Layers that are…
158 00:18:55.560 ⇒ 00:19:00.740 Gilbert Adjei: often hit by downstream tools, like the DI tools.
159 00:19:01.140 ⇒ 00:19:08.870 Gilbert Adjei: things of that sort, that should be materialized as tables. So, in situations where performance matters,
160 00:19:09.060 ⇒ 00:19:15.319 Gilbert Adjei: You would want to save those mats and things of that sort as tables, and then for the
161 00:19:15.850 ⇒ 00:19:21.509 Gilbert Adjei: The staging part, we want to save it as views.
162 00:19:22.720 ⇒ 00:19:26.530 Awaish Kumar: No, no, I mean the Mars table is also materialized as a view.
163 00:19:26.770 ⇒ 00:19:29.859 Awaish Kumar: What do you think? Like, should it be a view or a table?
164 00:19:31.450 ⇒ 00:19:45.119 Gilbert Adjei: Yes, so what I’m saying is that, in this case, it’s on a case-by-case basis, scenarios where we have BI tools interacting more with the model.
165 00:19:45.120 ⇒ 00:19:52.489 Gilbert Adjei: I would want it to be materialized as a table, especially when performance really matters.
166 00:19:57.060 ⇒ 00:19:57.780 Awaish Kumar: Okay.
167 00:19:58.830 ⇒ 00:20:04.520 Awaish Kumar: If you go back to that mart order summary, model.
168 00:20:05.200 ⇒ 00:20:12.140 Awaish Kumar: Yeah, so in that… I can see it’s only joining, orders.
169 00:20:12.290 ⇒ 00:20:13.989 Awaish Kumar: With all the line items.
170 00:20:14.220 ⇒ 00:20:17.429 Awaish Kumar: But it is missing information regarding, like,
171 00:20:17.910 ⇒ 00:20:22.780 Awaish Kumar: The… the customer… customer information, or the product’s information.
172 00:20:27.100 ⇒ 00:20:36.289 Gilbert Adjei: Yes. So, this is a metric that I defined. So, I think, within the take-home, it was more like you can
173 00:20:36.430 ⇒ 00:20:41.940 Gilbert Adjei: You can have metrics, you can define your metrics, and then, it’s…
174 00:20:42.050 ⇒ 00:20:48.509 Gilbert Adjei: having, it going into a mat. So that’s a metric that I opted to define.
175 00:20:53.170 ⇒ 00:21:06.700 Gilbert Adjei: But however, yes, if there are metrics that would need to, combine all the different tables, it’s something that can be joined in over here, since we already have
176 00:21:06.840 ⇒ 00:21:11.490 Gilbert Adjei: this necessity, we can… Easily bake it in.
177 00:21:16.210 ⇒ 00:21:16.840 Gilbert Adjei: Yes.
178 00:21:17.410 ⇒ 00:21:22.959 Awaish Kumar: Yeah, are there any other questions regarding dbt? Otherwise, I want to ask something about databases.
179 00:21:27.270 ⇒ 00:21:28.150 Awaish Kumar: hear me?
180 00:21:30.090 ⇒ 00:21:35.299 Demilade Agboola: So, my question with DBT would be…
181 00:21:36.460 ⇒ 00:21:39.310 Demilade Agboola: Well, not dbt per se, but how would you…
182 00:21:39.550 ⇒ 00:21:48.990 Demilade Agboola: control… can you… how would you control access to specific maths, for instance? So if you were trying to do some data governance and ensure that
183 00:21:49.120 ⇒ 00:22:00.540 Demilade Agboola: Only, like, finance. So right now, it’s only, like, matter to other summary, but if we had finance, we had marketing data, how would you ensure that the people that saw it were the only, appropriate people?
184 00:22:02.450 ⇒ 00:22:10.600 Gilbert Adjei: Sure. So, you can use tax to help you with that, and then also, within them.
185 00:22:10.990 ⇒ 00:22:17.329 Gilbert Adjei: are BSC, so if… You have departments, and then you grant
186 00:22:17.500 ⇒ 00:22:19.679 Gilbert Adjei: A user is from a particular
187 00:22:20.080 ⇒ 00:22:26.410 Gilbert Adjei: department like finance, all users inheriting from the finance
188 00:22:26.480 ⇒ 00:22:46.289 Gilbert Adjei: role should be able to see things that are pertaining or related to finance, and then I can do a similar thing for those in operations, right? So, all users within operations team and are inheriting roles from operations should be able to see things pertaining to operations only.
189 00:22:49.320 ⇒ 00:22:50.610 Demilade Agboola: Okay, thank you.
190 00:22:51.140 ⇒ 00:22:55.060 Awaish Kumar: Yeah, can we see the roles RBSE file?
191 00:22:55.440 ⇒ 00:22:56.200 Gilbert Adjei: Yes.
192 00:22:57.290 ⇒ 00:23:04.800 Awaish Kumar: Yeah, I… yeah, like… If, like, can you open some… somewhere where it’s creating the roles?
193 00:23:05.790 ⇒ 00:23:06.859 Gilbert Adjei: Yes. Oh, yeah.
194 00:23:07.210 ⇒ 00:23:08.740 Awaish Kumar: So, like, what is this known…
195 00:23:08.740 ⇒ 00:23:11.109 Gilbert Adjei: How are the rules? Okay, the rows, yes.
196 00:23:11.880 ⇒ 00:23:15.839 Awaish Kumar: So what is this, like, keyword no login means here?
197 00:23:17.680 ⇒ 00:23:25.209 Gilbert Adjei: Okay, so what that means in Postgres is that this particular one is A room, and not…
198 00:23:25.640 ⇒ 00:23:36.660 Gilbert Adjei: a user, right? So, in other flavors, you can do something like create user, and then the name of the user. However, in Postgres, the syntax is
199 00:23:36.770 ⇒ 00:23:39.270 Gilbert Adjei: Side to that, if you want it to be more like…
200 00:23:39.500 ⇒ 00:23:46.970 Gilbert Adjei: A role, and how it differs from a user, you use the keyword, no login.
201 00:23:48.610 ⇒ 00:23:49.250 Gilbert Adjei: Yes.
202 00:23:49.930 ⇒ 00:23:50.670 Awaish Kumar: Okay.
203 00:23:51.780 ⇒ 00:24:05.620 Awaish Kumar: And, how do you define these variables in your initialization of the project? The ones that are being used at the bottom, like dev user password and the air byte user password?
204 00:24:06.850 ⇒ 00:24:15.520 Gilbert Adjei: Yes, so, those are being passed in from So, if I go to…
205 00:24:22.330 ⇒ 00:24:23.150 Gilbert Adjei: Yes.
206 00:24:23.310 ⇒ 00:24:28.960 Gilbert Adjei: So… I already have, how this is being run.
207 00:24:29.310 ⇒ 00:24:37.409 Gilbert Adjei: And then, so, once this is being run, I have… So, on my…
208 00:24:38.530 ⇒ 00:24:43.210 Gilbert Adjei: I’ll show, how things are done, on the,
209 00:24:43.910 ⇒ 00:24:51.800 Gilbert Adjei: when it’s in deployment, but on my local, you can set these variables in here, right, so that it can point the…
210 00:24:52.360 ⇒ 00:24:57.360 Gilbert Adjei: it can extract the, the credentials. However, on the…
211 00:24:59.370 ⇒ 00:25:13.880 Gilbert Adjei: in production, how I do that is I use They’re secret, so… Let me come here… And then… So…
212 00:25:14.310 ⇒ 00:25:16.059 Gilbert Adjei: Go to Settings…
213 00:25:20.410 ⇒ 00:25:28.390 Gilbert Adjei: And then, over here… So… Secrets and variables.
214 00:25:30.260 ⇒ 00:25:31.310 Gilbert Adjei: Auctions…
215 00:25:32.740 ⇒ 00:25:43.260 Gilbert Adjei: And then over here, so this is how, in production, I set my secret, right, so that, it’s being ingested in there, and then I don’t need to,
216 00:25:44.500 ⇒ 00:25:49.060 Gilbert Adjei: or push my secrets to… Okay. Yes.
217 00:25:49.420 ⇒ 00:25:52.060 Awaish Kumar: Okay, I don’t see these, these,
218 00:25:52.630 ⇒ 00:25:56.130 Awaish Kumar: like, workflow files in the… in the repo.
219 00:25:57.310 ⇒ 00:26:01.490 Gilbert Adjei: Yes. So, when I submitted it,
220 00:26:01.650 ⇒ 00:26:04.980 Gilbert Adjei: In… as part of my… the…
221 00:26:06.230 ⇒ 00:26:11.370 Gilbert Adjei: email response, one of the things that I realized was that So…
222 00:26:12.470 ⇒ 00:26:20.340 Gilbert Adjei: This one is bring forward, assessments, right? Yeah. Yeah. So, one thing I realized was that when I…
223 00:26:20.570 ⇒ 00:26:21.770 Gilbert Adjei: push dust.
224 00:26:21.940 ⇒ 00:26:28.299 Gilbert Adjei: And then I want… I wanted to create a PR out of it. It’s… showed me…
225 00:26:28.600 ⇒ 00:26:31.770 Gilbert Adjei: So, let me even show you how it’s looking like.
226 00:26:32.040 ⇒ 00:26:35.820 Gilbert Adjei: they told me that.
227 00:26:36.230 ⇒ 00:26:39.150 Gilbert Adjei: you cannot create a PR.
228 00:26:40.220 ⇒ 00:26:41.470 Gilbert Adjei: Out of this.
229 00:26:42.260 ⇒ 00:26:45.549 Gilbert Adjei: And then I added it as part of the email.
230 00:26:51.530 ⇒ 00:26:55.310 Awaish Kumar: Okay, but you can show me the actions you are running right now.
231 00:26:56.060 ⇒ 00:26:57.140 Awaish Kumar: Yes.
232 00:26:58.530 ⇒ 00:27:07.699 Gilbert Adjei: Yes, so this particular one, this particular one was done in a private repo, right?
233 00:27:07.900 ⇒ 00:27:24.910 Gilbert Adjei: this one, when I pushed it on… in this particular repo, I was only able to create a branch. I wasn’t able to create a PR, it just told me that it wasn’t, so I created a private repo, and then just to demonstrate the end-to-end, solution, and then
234 00:27:25.180 ⇒ 00:27:32.239 Gilbert Adjei: where this particular one, I’m able to do a PR, and then to show How this whole process works.
235 00:27:33.840 ⇒ 00:27:36.859 Awaish Kumar: Yeah, can you walk me through the… the YML file?
236 00:27:37.000 ⇒ 00:27:38.410 Awaish Kumar: For those actions.
237 00:27:39.560 ⇒ 00:27:40.230 Gilbert Adjei: Yes.
238 00:27:41.510 ⇒ 00:27:42.550 Gilbert Adjei: So…
239 00:27:46.800 ⇒ 00:27:49.380 Gilbert Adjei: So we have that thus far.
240 00:27:49.860 ⇒ 00:27:51.480 Gilbert Adjei: fraud.
241 00:27:51.880 ⇒ 00:27:57.400 Gilbert Adjei: I created two files, one for prod and then one for staging, so…
242 00:27:57.610 ⇒ 00:28:03.650 Gilbert Adjei: as I had shown earlier, the… credentials.
243 00:28:04.020 ⇒ 00:28:06.989 Gilbert Adjei: I had shown how they are being stored in.
244 00:28:08.620 ⇒ 00:28:18.219 Gilbert Adjei: gates, and then you can access them using the secret dots, and then the name of the secrets. So, how Prod works is.
245 00:28:18.610 ⇒ 00:28:28.660 Gilbert Adjei: I have scheduled this to run every 6 hours, right? So, it uses the Ubuntu, base…
246 00:28:29.680 ⇒ 00:28:31.509 Gilbert Adjei: OS, and then…
247 00:28:31.740 ⇒ 00:28:46.940 Gilbert Adjei: I have put this into various stages, steps. So, after it spins up Ubuntu, it installs Python, and then it installs all the necessary dependencies for this to run, and then
248 00:28:47.230 ⇒ 00:28:57.109 Gilbert Adjei: As I mentioned, it extracts the secrets from the repo secret, so that it can use it to spin up.
249 00:28:57.390 ⇒ 00:28:58.420 Gilbert Adjei: Dopper.
250 00:28:59.090 ⇒ 00:29:09.139 Gilbert Adjei: image, right? It pulls a Docker container, Postgres Docker container, and then sets the credentials using that with, the one from
251 00:29:09.370 ⇒ 00:29:10.230 Gilbert Adjei: Git.
252 00:29:10.560 ⇒ 00:29:16.970 Gilbert Adjei: And once that is done, We give it some minutes to fully initialize, and then…
253 00:29:17.810 ⇒ 00:29:22.930 Gilbert Adjei: Create the necessary schema that is needed, and then also says.
254 00:29:23.430 ⇒ 00:29:29.750 Gilbert Adjei: I was creating the workflow in, gitHub, at…
255 00:29:29.960 ⇒ 00:29:35.600 Gilbert Adjei: we needed to ingest the raw data, right? So we needed to make it Similar to how…
256 00:29:35.750 ⇒ 00:29:38.529 Gilbert Adjei: AirByte was running locally.
257 00:29:38.700 ⇒ 00:29:41.700 Gilbert Adjei: And within this take-home we…
258 00:29:42.150 ⇒ 00:29:50.249 Gilbert Adjei: how to set air bytes locally, and then also store ingest the data. So, I wrote a Python script that
259 00:29:50.590 ⇒ 00:29:57.610 Gilbert Adjei: Picks the data, and then ingests it into this ephemeral database that has been created.
260 00:29:57.760 ⇒ 00:30:03.050 Gilbert Adjei: And then, since dbt had already been installed, it’s…
261 00:30:03.380 ⇒ 00:30:08.810 Gilbert Adjei: Initializes the working directory, which is the same as the one that has been,
262 00:30:08.970 ⇒ 00:30:21.789 Gilbert Adjei: on the project, and then installs all the dependencies that are needed, and then sets the various targets, and then the various environment variables. And when that is done, it runs the
263 00:30:22.400 ⇒ 00:30:26.410 Gilbert Adjei: Prod, and then target, and then also test it.
264 00:30:26.830 ⇒ 00:30:33.459 Gilbert Adjei: And in order to see it, whether it’s working or not, you go to Workflow over here, so once
265 00:30:33.570 ⇒ 00:30:34.520 Gilbert Adjei: Any…
266 00:30:34.680 ⇒ 00:30:43.650 Gilbert Adjei: PR is pushed, or… yes, you make… so I intentionally made it to fail, and then also made it to pass, so that we see
267 00:30:43.830 ⇒ 00:30:45.909 Gilbert Adjei: How things work.
268 00:30:46.120 ⇒ 00:30:50.330 Gilbert Adjei: So… To see it in action.
269 00:30:52.340 ⇒ 00:31:00.650 Gilbert Adjei: Again, once you push, it will automatically trigger it, and then when you go to usage, you see Ow.
270 00:31:01.470 ⇒ 00:31:10.389 Gilbert Adjei: it’s what all the different steps, right? So, setting up the job, the dependencies, the Postgres container.
271 00:31:10.570 ⇒ 00:31:14.740 Gilbert Adjei: the schema, dependencies, and all that, so when…
272 00:31:14.740 ⇒ 00:31:16.810 Demilade Agboola: Can you click into run dbt?
273 00:31:17.480 ⇒ 00:31:18.130 Gilbert Adjei: Yes.
274 00:31:18.900 ⇒ 00:31:19.820 Gilbert Adjei: So…
275 00:31:23.150 ⇒ 00:31:25.659 Demilade Agboola: So, it’s transferred and targets prod.
276 00:31:26.060 ⇒ 00:31:38.939 Gilbert Adjei: then the secrets have been hashed out, and then, so that we don’t have any form of leakages, and then the system is secure, from any form of attacks. And then when all this is done.
277 00:31:39.460 ⇒ 00:31:49.119 Gilbert Adjei: indicates that it has passed, and then the various models that need to be ran have been run, and yeah. So that’s how it’s…
278 00:31:49.350 ⇒ 00:31:53.350 Gilbert Adjei: runs, and it also executes the tests as well, so…
279 00:31:55.410 ⇒ 00:31:57.979 Gilbert Adjei: That’s how things run in production.
280 00:31:58.130 ⇒ 00:32:00.420 Demilade Agboola: Also, if we’re going to set up,
281 00:32:01.510 ⇒ 00:32:04.140 Demilade Agboola: Alerting. How would you set up alerting?
282 00:32:05.650 ⇒ 00:32:09.029 Gilbert Adjei: Yes, in fact, it’s… it’s one of the things that,
283 00:32:09.840 ⇒ 00:32:12.280 Gilbert Adjei: Maybe as a next step,
284 00:32:12.420 ⇒ 00:32:16.460 Gilbert Adjei: would be great to do. So, typically, what’s…
285 00:32:16.610 ⇒ 00:32:21.980 Gilbert Adjei: I do when it comes to alertness, If we are using…
286 00:32:22.480 ⇒ 00:32:27.340 Gilbert Adjei: teams. I integrate Teams with Damn.
287 00:32:27.640 ⇒ 00:32:34.559 Gilbert Adjei: The version control, so that if it fails, automatically it will send in
288 00:32:34.820 ⇒ 00:32:43.650 Gilbert Adjei: a Teams alert, so if it’s a slap to integrate… you create that integration with the version control, and then when it fails, you can proactively
289 00:32:43.780 ⇒ 00:32:51.110 Gilbert Adjei: click on it, and I see where the failures are happening, and then you can, see… fix them.
290 00:32:51.300 ⇒ 00:32:53.620 Gilbert Adjei: So, that’s how I would do it.
291 00:33:00.440 ⇒ 00:33:04.180 Awaish Kumar: Yeah, so why… I see, like, we use AirByte
292 00:33:04.560 ⇒ 00:33:09.700 Awaish Kumar: Because we want to ingest using your byte instead of writing custom scripts.
293 00:33:12.900 ⇒ 00:33:13.740 Gilbert Adjei: Yes.
294 00:33:13.970 ⇒ 00:33:16.689 Awaish Kumar: But in the action, I can still see…
295 00:33:17.230 ⇒ 00:33:20.749 Awaish Kumar: the step to load the data. What is that?
296 00:33:22.270 ⇒ 00:33:24.950 Gilbert Adjei: Yes, so the… this…
297 00:33:26.420 ⇒ 00:33:38.179 Gilbert Adjei: this whole process is… so, if it’s being dockerized, it needs the raw data to work on it, the dbt to work on it. Now.
298 00:33:38.450 ⇒ 00:33:41.880 Gilbert Adjei: for Git Action to work on it, it means that
299 00:33:42.260 ⇒ 00:33:46.590 Gilbert Adjei: This… this environment needs access to the data.
300 00:33:47.860 ⇒ 00:33:52.610 Gilbert Adjei: since the Postgres has not been deployed to and life.
301 00:33:52.740 ⇒ 00:33:54.720 Gilbert Adjei: fever.
302 00:33:55.020 ⇒ 00:34:05.859 Gilbert Adjei: we cannot assess the data from there, right? So, and still, we need the data somehow to be in Postgres before dbt can run
303 00:34:05.980 ⇒ 00:34:08.570 Gilbert Adjei: On them. So that’s why…
304 00:34:08.800 ⇒ 00:34:12.460 Gilbert Adjei: we spin up the Postgres database, and then
305 00:34:12.790 ⇒ 00:34:20.519 Gilbert Adjei: we needed to ingest the data into Postgres before we ran the, their DBT on it.
306 00:34:20.710 ⇒ 00:34:24.529 Gilbert Adjei: Because Postgres hasn’t been deployed to a lifesaver.
307 00:34:26.639 ⇒ 00:34:27.269 Gilbert Adjei: Yes.
308 00:34:32.219 ⇒ 00:34:32.919 Awaish Kumar: Okay.
309 00:34:36.420 ⇒ 00:34:41.799 Uttam Kumaran: I guess my, my question is gonna be, like, let’s say you notice that one of the models
310 00:34:42.159 ⇒ 00:34:46.980 Uttam Kumaran: Goes from taking, like, 10-15 seconds to, like, 5-6 minutes.
311 00:34:47.170 ⇒ 00:34:54.680 Uttam Kumaran: Like, walk me through some of, like, your investigation that you would… you… you commonly think you would… you would do to sort of see what’s going on.
312 00:34:56.409 ⇒ 00:35:03.219 Gilbert Adjei: Yes, so my first thing that I’ll do is to look at the model that is failing, and then also look at
313 00:35:03.459 ⇒ 00:35:08.489 Gilbert Adjei: the queries that are running it, right? So if there are…
314 00:35:09.039 ⇒ 00:35:12.999 Gilbert Adjei: A lot of unnecessary joints, and then…
315 00:35:13.699 ⇒ 00:35:22.329 Gilbert Adjei: it means that there are some inefficiencies in there, which needs to be worked on, and I would also look at whether the data
316 00:35:22.619 ⇒ 00:35:25.959 Gilbert Adjei: The model… the query itself.
317 00:35:26.119 ⇒ 00:35:35.099 Gilbert Adjei: Once it has run, the materialized data has some form of indexes on it, or it doesn’t, or…
318 00:35:35.559 ⇒ 00:35:41.059 Gilbert Adjei: If it doesn’t, then it means that probably things are not being done well. I’ll look at the upstream
319 00:35:41.939 ⇒ 00:35:47.079 Gilbert Adjei: Models that are being run, and see if there are some
320 00:35:47.379 ⇒ 00:35:55.749 Gilbert Adjei: new changes that are coming in, and that is causing the downstream model to delay unnecessarily. So, for example, if I’m expecting
321 00:35:55.959 ⇒ 00:36:04.819 Gilbert Adjei: thousand records from an upstream model, and then it can type in, say, 1 million, or…
322 00:36:04.999 ⇒ 00:36:07.679 Gilbert Adjei: 10 million, I know that’s maybe something…
323 00:36:07.819 ⇒ 00:36:23.539 Gilbert Adjei: may have changed upstream, so I’ll also investigate further to see whether things have changed upstream, and then… but most importantly, it’s about the efficiency of the queries that have been written within that particular… that builds that particular model.
324 00:36:25.040 ⇒ 00:36:25.580 Uttam Kumaran: Okay.
325 00:36:25.860 ⇒ 00:36:35.519 Awaish Kumar: Okay, let’s build on top of that, since you mentioned you could add in taxes, so let’s take an example, like, we have mart order summary table.
326 00:36:35.800 ⇒ 00:36:43.880 Awaish Kumar: And… It grows to, like, 500 million plus votes, and our queries really becomes very slow.
327 00:36:44.180 ⇒ 00:36:51.579 Awaish Kumar: And then we have a downstream dashboard, which is, like, which uses filters on order date, customer ID,
328 00:36:51.800 ⇒ 00:36:53.870 Awaish Kumar: And financial status.
329 00:36:54.090 ⇒ 00:37:01.679 Awaish Kumar: So, like, what exactly, what kind of… what indexes you would consider first on that table, and why?
330 00:37:05.810 ⇒ 00:37:16.400 Gilbert Adjei: So, if I understand you, the indexes that can be done on the maths data, right?
331 00:37:18.940 ⇒ 00:37:22.379 Gilbert Adjei: Like, we have a smart order summary.
332 00:37:22.550 ⇒ 00:37:25.720 Awaish Kumar: And that is being used by a BI tool.
333 00:37:25.980 ⇒ 00:37:31.919 Awaish Kumar: Which basically uses filters on order date, customer ID, And financial status.
334 00:37:32.790 ⇒ 00:37:39.150 Awaish Kumar: And and the query is really slow, so it takes a lot of time to load on the BI tool.
335 00:37:39.350 ⇒ 00:37:41.720 Awaish Kumar: So what indexes would you consider?
336 00:37:42.050 ⇒ 00:37:45.849 Awaish Kumar: First, and… and why would you consider that?
337 00:37:47.420 ⇒ 00:37:48.110 Gilbert Adjei: Okay.
338 00:37:48.290 ⇒ 00:37:53.960 Gilbert Adjei: So, first of all, I’ll consider, the primary
339 00:37:54.670 ⇒ 00:37:59.939 Gilbert Adjei: key index, and then also the… so it ensures that the role
340 00:38:00.180 ⇒ 00:38:04.790 Gilbert Adjei: It’s unique, and then makes the lookup, very fast.
341 00:38:05.080 ⇒ 00:38:14.120 Gilbert Adjei: And then I also look at, the foreign key index, right? So if there are fact tables, that often references, say, dimension tables.
342 00:38:14.230 ⇒ 00:38:19.550 Gilbert Adjei: These forms of indexing can really speed up the joints.
343 00:38:20.080 ⇒ 00:38:23.799 Gilbert Adjei: And then also, since there may be some dates time
344 00:38:24.110 ⇒ 00:38:38.030 Gilbert Adjei: within the data, I would also look at… it’s very common within analytics apps, in fact, since maybe you may be filtering by date ranges and things like that, I also create indexes on that.
345 00:38:38.140 ⇒ 00:38:39.040 Gilbert Adjei: I swear.
346 00:38:43.030 ⇒ 00:38:49.539 Awaish Kumar: Okay, and then how would you validate index usefulness and… And avoid over-indexing.
347 00:38:51.480 ⇒ 00:38:52.770 Gilbert Adjei: How will you validate?
348 00:38:53.740 ⇒ 00:38:54.310 Awaish Kumar: Yeah.
349 00:38:55.020 ⇒ 00:38:57.990 Gilbert Adjei: How… no, I’m asking for you to repeat.
350 00:38:58.290 ⇒ 00:39:01.779 Uttam Kumaran: Oh yeah, how would, like, how would you validate that the indexes worked?
351 00:39:01.940 ⇒ 00:39:04.289 Uttam Kumaran: And that you didn’t, like, over-index.
352 00:39:04.550 ⇒ 00:39:07.309 Uttam Kumaran: Like, add too many, or like… yeah.
353 00:39:09.700 ⇒ 00:39:14.519 Gilbert Adjei: Yeah, so normally, when you add index,
354 00:39:14.820 ⇒ 00:39:17.730 Gilbert Adjei: You would want to run a query, like.
355 00:39:17.950 ⇒ 00:39:27.859 Gilbert Adjei: explain, analyze, before and after, right? So if it’s taking a long time, let’s say, you do explain, analyze, and you do select star from
356 00:39:28.460 ⇒ 00:39:33.849 Gilbert Adjei: something, a table, say customized with customer ID, 100, and I see…
357 00:39:34.050 ⇒ 00:39:52.819 Gilbert Adjei: how, it’s run, and then you compare the time execution, right? So if the first one takes, like, one minute after applying the indexing, you ran the same, and then you see that the time has dropped significantly, and then, you see that, yes, the indexing worked.
358 00:39:53.890 ⇒ 00:39:59.379 Gilbert Adjei: Yeah, and also you can use the database in their statistics as well to check
359 00:39:59.770 ⇒ 00:40:03.670 Gilbert Adjei: But normally, I would use the explain, analyze, too.
360 00:40:04.100 ⇒ 00:40:07.470 Gilbert Adjei: Confirm that’s the time before, and then the time after.
361 00:40:07.950 ⇒ 00:40:09.649 Gilbert Adjei: It has a lot of differences.
362 00:40:11.950 ⇒ 00:40:14.380 Demilade Agboola: Have you ever walked me with red chicks?
363 00:40:15.960 ⇒ 00:40:18.450 Gilbert Adjei: Yes, I have.
364 00:40:18.700 ⇒ 00:40:20.800 Gilbert Adjei: My bad hours a long time.
365 00:40:21.430 ⇒ 00:40:22.070 Demilade Agboola: Okay.
366 00:40:22.190 ⇒ 00:40:29.140 Demilade Agboola: Yeah. Alright, so let me now ask you that question, because I just wanted to… to see, like, your understanding of source keys, but that’s fine.
367 00:40:32.250 ⇒ 00:40:45.479 Awaish Kumar: Yeah, since this model now have grown to 500 million plus rows, it is possible that if I create this table, it is going to take, like, maybe 10 minutes to run on a DVD.
368 00:40:45.700 ⇒ 00:40:50.230 Awaish Kumar: So what, like, DVT materialization strategy
369 00:40:50.350 ⇒ 00:40:52.960 Awaish Kumar: You will change to get to that.
370 00:40:56.340 ⇒ 00:40:58.150 Gilbert Adjei: Can you please take that again?
371 00:41:00.420 ⇒ 00:41:02.940 Awaish Kumar: So, for example, we have a table.
372 00:41:03.270 ⇒ 00:41:11.319 Awaish Kumar: Because of… now it… there’s a lot of data, it takes, like, a lot of time to run or execute it.
373 00:41:11.430 ⇒ 00:41:16.379 Awaish Kumar: Like, maybe, for example, you can say, it takes 10 minutes to execute one model.
374 00:41:16.810 ⇒ 00:41:22.520 Awaish Kumar: So what can I change in my, DBT materialization strategy?
375 00:41:22.810 ⇒ 00:41:24.629 Awaish Kumar: So I can make it faster.
376 00:41:27.700 ⇒ 00:41:28.490 Gilbert Adjei: Yeah.
377 00:41:28.950 ⇒ 00:41:32.020 Gilbert Adjei: So… I think?
378 00:41:34.210 ⇒ 00:41:40.040 Gilbert Adjei: I think that if it’s slowing down, and then we want to make it
379 00:41:40.370 ⇒ 00:41:46.020 Gilbert Adjei: very fast. In DBT, it comes… M…
380 00:41:47.290 ⇒ 00:41:54.140 Gilbert Adjei: You can, first of all, make it incremental, on top of my head, since…
381 00:41:54.780 ⇒ 00:42:07.059 Gilbert Adjei: Yeah, you can make it incremental, and then when the model is very large, and then if there are new or chain growths that need to be processed, using incremental can really
382 00:42:07.250 ⇒ 00:42:08.290 Gilbert Adjei: Help you.
383 00:42:08.430 ⇒ 00:42:13.939 Gilbert Adjei: with… Things that are very large, like, like, like this.
384 00:42:19.500 ⇒ 00:42:24.619 Demilade Agboola: And if you’re going to do incremental, how would you build it? Would you, like…
385 00:42:25.090 ⇒ 00:42:29.560 Demilade Agboola: How do you ensure that you’re incrementing the data properly?
386 00:42:30.140 ⇒ 00:42:39.310 Gilbert Adjei: Sure. So, normally, I’ll add… so, something like this, I’ll… if you are still seeing my screen, I’ll just do something like this.
387 00:42:39.520 ⇒ 00:42:47.879 Gilbert Adjei: Config, and then… Define its, like, Materialized view. Materialized?
388 00:42:48.260 ⇒ 00:42:54.509 Gilbert Adjei: incremental, and then the unique key, I’ll just use something like the order ID. So, something like this.
389 00:42:55.330 ⇒ 00:43:00.100 Demilade Agboola: So, quick question, just to follow up to that. If it doesn’t have a unique key, what would you use to…
390 00:43:02.320 ⇒ 00:43:10.970 Gilbert Adjei: BLP doesn’t have a unique key, I’ll… Combine, say, two… fields.
391 00:43:11.220 ⇒ 00:43:19.109 Gilbert Adjei: And, make it as a unique two or more fields, and then… Which uniquely identify the…
392 00:43:19.500 ⇒ 00:43:24.080 Gilbert Adjei: The rule, and then use it as… That’s a key.
393 00:43:36.320 ⇒ 00:43:43.129 Uttam Kumaran: Cool, I know we’re just coming up on time. I think this is great. I mean, I think you have a pretty good understanding of, sort of, the…
394 00:43:43.490 ⇒ 00:43:49.049 Uttam Kumaran: you know, how to set up each of the components locally. I think running through dbt was great.
395 00:43:49.270 ⇒ 00:43:51.130 Uttam Kumaran: I’m wondering, like.
396 00:43:51.290 ⇒ 00:44:03.169 Uttam Kumaran: I mean, I want to leave enough time for your question, but I’m kind of interested in your feedback on the exercise and, like, what part was interesting, or if any part was tough, just, like, your reflection.
397 00:44:05.070 ⇒ 00:44:12.240 Gilbert Adjei: Yes, so… I think with the exercise, it was great, because it was more practical.
398 00:44:13.540 ⇒ 00:44:17.459 Gilbert Adjei: Because it’s something that you would see normally on a…
399 00:44:17.950 ⇒ 00:44:20.580 Gilbert Adjei: If you are working with clients and things of that sort.
400 00:44:20.800 ⇒ 00:44:29.000 Gilbert Adjei: One of the things that, interestingly, was a challenge for me when I realized was that… what I saw was that
401 00:44:29.870 ⇒ 00:44:34.109 Gilbert Adjei: my SSH key had, for some reason, been corrupted, so…
402 00:44:34.470 ⇒ 00:44:41.809 Gilbert Adjei: I had worked on it, and then when I was pushing to gate, I realized that something had been corrupted, so I had to use
403 00:44:42.030 ⇒ 00:44:48.860 Gilbert Adjei: quickly use AI tool to speed up things, right? So, I didn’t need to memorize
404 00:44:49.140 ⇒ 00:45:06.469 Gilbert Adjei: get commands to set up SSH keys and things like that, which I felt that was very helpful to me. And leveraging these AI tools, can help you move fast, and you can focus more on the broad objective, of saying.
405 00:45:07.550 ⇒ 00:45:13.870 Gilbert Adjei: helping clients, or making revenues go up, and things like that. And then,
406 00:45:14.070 ⇒ 00:45:20.040 Gilbert Adjei: As opposed to memorizing things and then making yourself slow. So…
407 00:45:20.220 ⇒ 00:45:26.520 Gilbert Adjei: That was my challenge, and then that was how I was able to overcome it with these AI tools.
408 00:45:29.090 ⇒ 00:45:29.650 Uttam Kumaran: Cool.
409 00:45:30.550 ⇒ 00:45:33.490 Uttam Kumaran: What questions, do you have for us, Gilbert?
410 00:45:35.080 ⇒ 00:45:36.559 Gilbert Adjei: Yes, so on.
411 00:45:36.560 ⇒ 00:45:39.710 Uttam Kumaran: We also didn’t do a round of introductions, but,
412 00:45:39.990 ⇒ 00:45:43.989 Uttam Kumaran: That’s fine either way. Yeah, but tell me what questions you have for us.
413 00:45:45.270 ⇒ 00:45:52.170 Gilbert Adjei: Yes, so I’ve been reading and I’ve been following, Brain Forge’s, progress.
414 00:45:52.720 ⇒ 00:45:55.190 Gilbert Adjei: us. A detailed consultant.
415 00:45:55.680 ⇒ 00:46:02.740 Gilbert Adjei: company, and you’ve been helping different clients and all that, so… How have you…
416 00:46:03.290 ⇒ 00:46:08.700 Gilbert Adjei: seeing them embrace the usage of AI, and then how do you also
417 00:46:08.970 ⇒ 00:46:12.230 Gilbert Adjei: Use that internally as well, too.
418 00:46:12.580 ⇒ 00:46:14.930 Gilbert Adjei: Maybe speed up work and inspire that.
419 00:46:16.840 ⇒ 00:46:19.830 Uttam Kumaran: Yeah. Does anyone else? Does anyone want to take that?
420 00:46:22.630 ⇒ 00:46:23.850 Demilade Agboola: I mean, I can take it.
421 00:46:24.020 ⇒ 00:46:24.670 Uttam Kumaran: Yeah.
422 00:46:25.160 ⇒ 00:46:40.909 Demilade Agboola: So internally, to answer that, it’s… we highly, encourage the use of AI. Not just highly, we drive and try to make everyone on the team, both engineering and, like, sales and every, like, literally everyone on the team use AI.
423 00:46:41.020 ⇒ 00:46:48.659 Demilade Agboola: We set them up in cursor, we have ChatGPT accounts, And the idea is…
424 00:46:49.070 ⇒ 00:46:51.240 Demilade Agboola: We want people to focus on
425 00:46:51.350 ⇒ 00:47:01.369 Demilade Agboola: the core aspect of their jobs, rather than the nitty-gritty details. So if you can get AI to get you to do that faster, that is great.
426 00:47:01.580 ⇒ 00:47:07.930 Demilade Agboola: And also, because of that, we then try and use that to… Empower our,
427 00:47:08.180 ⇒ 00:47:16.600 Demilade Agboola: clients as well. So, in terms of being able to, use AI-powered tools, set them up with.
428 00:47:17.140 ⇒ 00:47:34.749 Demilade Agboola: like, for instance, we like to use a tool called Omni for, like, their BI, because Omni integrates AI very well, and so now the people, the stakeholders can go in there, ask the questions they need to ask, without always coming back to us for, like, small things, like, oh, how many orders do we have in 2025?
429 00:47:34.750 ⇒ 00:47:41.670 Demilade Agboola: Like, that’s not something we want to spend our hours doing for them, when you can just use AI to find that for you.
430 00:47:41.790 ⇒ 00:47:46.450 Demilade Agboola: So yeah, to answer your question, yes, we… we use and we leverage AI highly.
431 00:47:47.980 ⇒ 00:47:50.020 Gilbert Adjei: That’s impressive, that’s impressive.
432 00:47:50.020 ⇒ 00:47:52.170 Demilade Agboola: And it’s good to know that,
433 00:47:52.170 ⇒ 00:47:58.720 Gilbert Adjei: Their clients are also open about it, and… Yeah, in fact, this… whole…
434 00:47:58.880 ⇒ 00:48:08.150 Gilbert Adjei: landscape is mind-boggling. You can be very efficient. However, you should also know what you are doing, and make sure that.
435 00:48:08.150 ⇒ 00:48:11.630 Uttam Kumaran: Exactly, like, that second piece is what matters more than anything.
436 00:48:11.800 ⇒ 00:48:15.660 Uttam Kumaran: It’s cause, like, we all did it manually for so long, you know?
437 00:48:17.270 ⇒ 00:48:22.909 Gilbert Adjei: Yeah, yeah. That’s great, that’s great. I think you’ve answered my question.
438 00:48:25.070 ⇒ 00:48:34.280 Uttam Kumaran: Yeah, I think maybe my question is, like, tell me about, like, what you’re looking forward to, like, next in terms of next job, or, like.
439 00:48:34.310 ⇒ 00:48:46.869 Uttam Kumaran: next role in data, maybe something that you didn’t previously get a chance to do, or, like, a way that you’re, like, thinking about, like, I would love to grow in this new domain, or new part of the stack.
440 00:48:47.300 ⇒ 00:48:48.890 Uttam Kumaran: Yeah.
441 00:48:51.300 ⇒ 00:48:57.580 Gilbert Adjei: Yeah, so… what I would like to, one more, S…
442 00:48:58.410 ⇒ 00:49:06.580 Gilbert Adjei: And which, in fact, has resonated with what, Demi shared, as to leverage
443 00:49:07.010 ⇒ 00:49:19.610 Gilbert Adjei: these tools, that will make you more efficient. In the past, I was using… or a lot of engineers were using Stack Overflow, and currently, I don’t remember the last time I went to Stack Overflow for.
444 00:49:19.610 ⇒ 00:49:22.129 Uttam Kumaran: Me neither. Yeah, maybe it’s, like, 3 years, 2 years.
445 00:49:22.720 ⇒ 00:49:23.420 Gilbert Adjei: Yes.
446 00:49:23.420 ⇒ 00:49:24.100 Uttam Kumaran: Yeah.
447 00:49:24.670 ⇒ 00:49:32.089 Gilbert Adjei: I want to leverage the use of these tools, however, I need to understand things, how they work.
448 00:49:32.230 ⇒ 00:49:36.509 Gilbert Adjei: Holistically, and then how they can speed up work.
449 00:49:36.570 ⇒ 00:49:56.519 Gilbert Adjei: For not only internal teams also, but for clients, because once you are more efficient with your processes, if you are closing in on one client, maybe you can double that, you can work on three client projects and things of that sort, right? And it can still be very efficient. So,
450 00:49:56.660 ⇒ 00:50:01.999 Gilbert Adjei: that’s what I’m really looking out for, using these tools, and then also
451 00:50:02.160 ⇒ 00:50:06.819 Gilbert Adjei: And most importantly, being able to Help grow.
452 00:50:06.970 ⇒ 00:50:08.240 Gilbert Adjei: our businesses.
453 00:50:08.560 ⇒ 00:50:13.030 Gilbert Adjei: revenue line, so that’s what I’m really looking out for in my next role.
454 00:50:14.980 ⇒ 00:50:15.550 Uttam Kumaran: Great.
455 00:50:18.750 ⇒ 00:50:20.960 Uttam Kumaran: Cool, guys. Any other questions?
456 00:50:21.220 ⇒ 00:50:22.730 Awaish Kumar: Nope, nope, I’m nice.
457 00:50:23.930 ⇒ 00:50:25.890 Demilade Agboola: Nothing for me, as well.
458 00:50:26.660 ⇒ 00:50:27.390 Uttam Kumaran: Okay.
459 00:50:27.510 ⇒ 00:50:35.549 Uttam Kumaran: Perfect. Thank you so much, Gilbert. I appreciate you taking all the time to work on the exercise as well, and I’m kind of going into detail today, so it’s really helpful.
460 00:50:38.330 ⇒ 00:50:39.590 Awaish Kumar: Okay, thank you.
461 00:50:39.590 ⇒ 00:50:40.680 Demilade Agboola: Thank you, everyone. Okay.
462 00:50:40.680 ⇒ 00:50:42.360 Uttam Kumaran: Perfect. Thank you so much.