Meeting Title: Brainforge Interview with Gideon Date: 2026-04-17 Meeting participants: Awaish Kumar, Gideon Fernandez
WEBVTT
1 00:00:27.120 ⇒ 00:00:27.970 Awaish Kumar: Hi.
2 00:00:28.940 ⇒ 00:00:29.930 Gideon Fernandez: Hello!
3 00:00:30.280 ⇒ 00:00:31.200 Awaish Kumar: How are you doing?
4 00:00:31.580 ⇒ 00:00:32.939 Gideon Fernandez: Good, how are you doing?
5 00:00:33.510 ⇒ 00:00:36.730 Awaish Kumar: Good as well. How can I pronounce your name?
6 00:00:37.170 ⇒ 00:00:38.150 Gideon Fernandez: Gideon.
7 00:00:38.510 ⇒ 00:00:39.779 Awaish Kumar: Gideon, okay.
8 00:00:39.780 ⇒ 00:00:40.740 Gideon Fernandez: And you?
9 00:00:41.050 ⇒ 00:00:42.070 Awaish Kumar: Aishkuman.
10 00:00:42.380 ⇒ 00:00:43.210 Gideon Fernandez: Avesh.
11 00:00:44.320 ⇒ 00:00:47.390 Awaish Kumar: Yeah, thank you for joining today, Gideon.
12 00:00:48.670 ⇒ 00:00:51.069 Awaish Kumar: So, where are you located?
13 00:00:51.880 ⇒ 00:00:55.990 Gideon Fernandez: Right now, I’m located in Spain, but I’m based in, Florida.
14 00:00:57.110 ⇒ 00:00:57.820 Awaish Kumar: Okay.
15 00:00:57.990 ⇒ 00:00:59.279 Gideon Fernandez: That’s where home is.
16 00:00:59.380 ⇒ 00:01:06.440 Gideon Fernandez: But I’m currently, living here with, my wife is here from Spain, so… Just out here now.
17 00:01:06.930 ⇒ 00:01:08.990 Awaish Kumar: So, you are just visiting, or…
18 00:01:09.490 ⇒ 00:01:11.369 Awaish Kumar: Settle down there.
19 00:01:11.730 ⇒ 00:01:19.050 Gideon Fernandez: I’m a bit settled, but, yeah, everyth- all my client work, projects, home…
20 00:01:19.410 ⇒ 00:01:25.360 Gideon Fernandez: finances, everything, is based in… in Florida, and I’m originally from D.C.
21 00:01:25.990 ⇒ 00:01:26.590 Awaish Kumar: Okay.
22 00:01:26.950 ⇒ 00:01:36.859 Awaish Kumar: Okay, so, yeah, nice to meet you, and thank you for taking the time today for this interview. It’s just a 30-minute session where we are going to
23 00:01:37.200 ⇒ 00:01:38.479 Awaish Kumar: Talk about it.
24 00:01:38.600 ⇒ 00:01:43.840 Awaish Kumar: Briefly about your background and what your… Cool.
25 00:01:44.660 ⇒ 00:01:48.419 Awaish Kumar: Data experience looks like, and what kind of projects you’ve worked on.
26 00:01:48.910 ⇒ 00:01:52.159 Awaish Kumar: And I’m here to answer any questions you might have for me.
27 00:01:53.170 ⇒ 00:01:54.030 Gideon Fernandez: Sure.
28 00:01:54.200 ⇒ 00:01:54.820 Awaish Kumar: Okay.
29 00:01:54.820 ⇒ 00:01:55.699 Gideon Fernandez: Good plan.
30 00:01:56.190 ⇒ 00:01:58.320 Awaish Kumar: Let’s get started with your introduction.
31 00:01:59.800 ⇒ 00:02:08.430 Gideon Fernandez: Thank you! So, I’m Gideon Fernandez, and I run my own data and AI consulting company.
32 00:02:09.020 ⇒ 00:02:17.099 Gideon Fernandez: And I am the only solo person that I, am working on client projects, and so…
33 00:02:17.430 ⇒ 00:02:20.259 Gideon Fernandez: A lot of the clients that I work with
34 00:02:20.390 ⇒ 00:02:25.200 Gideon Fernandez: Have a solution… a problem that has a solution that’s around…
35 00:02:25.310 ⇒ 00:02:33.659 Gideon Fernandez: analytic engineering solutions, so things like dbt, Fivetran, Snowflake implementing that.
36 00:02:33.840 ⇒ 00:02:40.670 Gideon Fernandez: For them is what I do. Building out the analytic models is my strong suit.
37 00:02:41.020 ⇒ 00:02:48.380 Gideon Fernandez: In fact, I also teach DBT and Snowflake courses at Siemens in Munich, Germany.
38 00:02:48.730 ⇒ 00:03:01.250 Gideon Fernandez: And currently working with a client right now based in DC, where I’m also implementing a Fivetran dbt and Snowflake solution for them, and then also building out their dashboards.
39 00:03:02.030 ⇒ 00:03:02.410 Awaish Kumar: Okay.
40 00:03:03.320 ⇒ 00:03:12.360 Gideon Fernandez: And that’s… a lot of the work that I do is implementing those kind of solutions, a lot of analytic models in dbt, implementing that from scratch.
41 00:03:12.780 ⇒ 00:03:16.709 Gideon Fernandez: And then, as far as dashboarding.
42 00:03:17.030 ⇒ 00:03:23.960 Gideon Fernandez: and enabling them for AI. My clients that I work with, they like to use some natural language prompting within it.
43 00:03:24.130 ⇒ 00:03:29.260 Gideon Fernandez: I’m also a partner with Omni, for dashboards.
44 00:03:30.080 ⇒ 00:03:38.940 Gideon Fernandez: So, been doing this, since 2017, so about, what is that, 9 years now?
45 00:03:39.200 ⇒ 00:03:47.070 Gideon Fernandez: I also worked at a startup, in 2021 for 3 years. They’re based out of, Austin, Texas.
46 00:03:47.410 ⇒ 00:03:52.410 Gideon Fernandez: And I was also an analytic engineer, a staff engineer.
47 00:03:52.560 ⇒ 00:03:56.450 Gideon Fernandez: For them, where I was working with their marketing.
48 00:03:56.660 ⇒ 00:04:02.100 Gideon Fernandez: And customer success divisions building out their, customer retention models.
49 00:04:02.400 ⇒ 00:04:06.120 Gideon Fernandez: And then drawing out the customer journey for them.
50 00:04:06.440 ⇒ 00:04:15.790 Gideon Fernandez: And then, I also started Accenture 2009, so I was doing a lot of consulting systems integration work, at Accenture for…
51 00:04:16.040 ⇒ 00:04:19.559 Gideon Fernandez: At the very start of my career for 6 years.
52 00:04:20.940 ⇒ 00:04:22.560 Awaish Kumar: Okay, great.
53 00:04:22.690 ⇒ 00:04:28.629 Awaish Kumar: Yeah, like, can we walk through a project that you have worked on end-to-end?
54 00:04:28.780 ⇒ 00:04:34.010 Awaish Kumar: although I have already listened to one of your examples in Loom.
55 00:04:34.130 ⇒ 00:04:38.580 Awaish Kumar: But you can give any other example, or… Right.
56 00:04:38.840 ⇒ 00:04:51.090 Awaish Kumar: of the project where you worked end-to-end, and what was the… like, what I want to hear is more about what was the data sources, how it was ingested, what was the tool stick, and then,
57 00:04:51.550 ⇒ 00:04:54.670 Awaish Kumar: Basically, what was the outcome, and yeah, that’s all.
58 00:04:55.040 ⇒ 00:05:01.889 Gideon Fernandez: Sure. So, I’ll tell you about an end-to-end project that I worked at Sunstone Credit, based out of New York City.
59 00:05:02.100 ⇒ 00:05:06.339 Gideon Fernandez: They are a solar development financing platform.
60 00:05:07.050 ⇒ 00:05:11.340 Gideon Fernandez: And so… I was the sole engineer.
61 00:05:11.630 ⇒ 00:05:16.020 Gideon Fernandez: On that project, where they didn’t have any analytic solutions at all.
62 00:05:16.480 ⇒ 00:05:20.930 Gideon Fernandez: And so the solution was dbt, Fivetran, and Snowflake.
63 00:05:21.080 ⇒ 00:05:26.100 Gideon Fernandez: their data sources was, one, Salesforce.
64 00:05:26.640 ⇒ 00:05:32.389 Gideon Fernandez: to their Postgres application, so data coming from a Postgres database.
65 00:05:32.530 ⇒ 00:05:42.580 Gideon Fernandez: And then 3 data coming from their vendor banking partners through an AWS SFTP server.
66 00:05:43.110 ⇒ 00:06:00.650 Gideon Fernandez: So those were 3 different sources that I was pulling in from, depending on the source. For the one in the SFTP server, I built a, AWS Lambda function that, pulled that data into the warehouse.
67 00:06:00.720 ⇒ 00:06:08.920 Gideon Fernandez: And then I used Fivetran to connect Postgres to Snowflake and Salesforce to Snowflake. So I was managing those two.
68 00:06:09.150 ⇒ 00:06:14.779 Gideon Fernandez: And then I worked with their strategy team to build out the analytic models.
69 00:06:15.390 ⇒ 00:06:22.569 Gideon Fernandez: in DBT, to serve their marketing, finance, And business operation teams.
70 00:06:22.730 ⇒ 00:06:25.860 Gideon Fernandez: Their analytic models for them to self-serve.
71 00:06:26.200 ⇒ 00:06:45.190 Gideon Fernandez: And they were using coefficient, connected to their, Enterprise Google Sheets to be able to do their analytics. So, on that side, they actually handled, the visualization, for this particular project. But the reason why I believe it’s a…
72 00:06:45.350 ⇒ 00:07:00.600 Gideon Fernandez: A… a project that I’d like to highlight is because they came back to me with a case study that my data platform has enabled them to save close to $180,000 a year, that they were, previously spending in manual reports.
73 00:07:02.090 ⇒ 00:07:02.770 Awaish Kumar: Okay.
74 00:07:04.770 ⇒ 00:07:09.050 Awaish Kumar: Sounds great. So while you are… you are working on the…
75 00:07:10.220 ⇒ 00:07:13.619 Awaish Kumar: those projects, how did you ensure the…
76 00:07:15.080 ⇒ 00:07:19.090 Awaish Kumar: Pipeline stability, and the data quality.
77 00:07:20.190 ⇒ 00:07:34.630 Gideon Fernandez: So, as far as data stability, I ensured that there was, alerts through our Slack messaging system, that alerted us for failed jobs. In addition to that, I implemented,
78 00:07:34.750 ⇒ 00:07:54.440 Gideon Fernandez: generic testing plus, unique testing on critical tables that alerted us on, any critical models that needed, extra supervision and support on. So, one of them was their loan application data and knowing those changes.
79 00:07:54.710 ⇒ 00:08:00.010 Gideon Fernandez: Who made the changes by a human user, so I built a snapshot table.
80 00:08:00.510 ⇒ 00:08:04.560 Gideon Fernandez: That monitored, a critical, source table.
81 00:08:04.820 ⇒ 00:08:21.400 Gideon Fernandez: And then if changes were made, it made a summary on when that previous… what that previous value was, the new value, and then who made the, entry change into the Salesforce application. And so that was an, that was conducted, nightly.
82 00:08:21.660 ⇒ 00:08:27.610 Gideon Fernandez: And then at the same time, what I like to do is, I like to…
83 00:08:27.790 ⇒ 00:08:36.209 Gideon Fernandez: run some kind of, elementary models, so I install elementary, where I ensure that I can see top-level KPIs of…
84 00:08:36.350 ⇒ 00:08:45.010 Gideon Fernandez: Of all the data models, the quality of how that’s being successfully built, and put it out onto a Streamlit application dashboard for them.
85 00:08:47.740 ⇒ 00:08:48.380 Awaish Kumar: Okay.
86 00:08:50.700 ⇒ 00:08:59.200 Awaish Kumar: Okay, so… Yeah, let’s talk a little bit about dbt.
87 00:08:59.510 ⇒ 00:09:09.130 Awaish Kumar: Like, for example, if we have a table which is taking… is…
88 00:09:09.260 ⇒ 00:09:13.170 Awaish Kumar: Very slow, like, in terms of If we carry that table.
89 00:09:13.370 ⇒ 00:09:16.490 Awaish Kumar: The care performance is very low.
90 00:09:16.870 ⇒ 00:09:21.730 Awaish Kumar: How would you go about… optimizing that.
91 00:09:23.390 ⇒ 00:09:28.160 Gideon Fernandez: I would look at what kind of table is it first. How is it being materialized?
92 00:09:28.570 ⇒ 00:09:33.470 Gideon Fernandez: So, my first thought is to understand what this table does.
93 00:09:33.620 ⇒ 00:09:34.700 Gideon Fernandez: is about.
94 00:09:35.730 ⇒ 00:09:41.609 Gideon Fernandez: And why is… I would assume that the model is,
95 00:09:42.090 ⇒ 00:09:46.540 Gideon Fernandez: is of a large volume, so let’s just take, for example, Y is…
96 00:09:46.860 ⇒ 00:09:51.470 Gideon Fernandez: For example, the click events, table taking so long.
97 00:09:51.820 ⇒ 00:09:58.160 Gideon Fernandez: And if I find that the model is being materialized incorrectly, or not…
98 00:09:58.220 ⇒ 00:10:10.910 Gideon Fernandez: not incorrect, but materialized in a way that’s being reprocessed every time, that would be one of the first things that I would look at to see if it’s being materialized as not just as a table or a view, for example.
99 00:10:10.950 ⇒ 00:10:20.349 Gideon Fernandez: But rather as an incremental model where we only need to process new records that come in, rather than processing historical and old records instead.
100 00:10:24.200 ⇒ 00:10:28.050 Awaish Kumar: Okay, apart from, like, this is one solution.
101 00:10:29.450 ⇒ 00:10:33.170 Awaish Kumar: Obviously, using, materialization.
102 00:10:33.380 ⇒ 00:10:37.310 Awaish Kumar: But this is… this solves the problem of
103 00:10:37.680 ⇒ 00:10:39.969 Awaish Kumar: Updating the table, like, the load…
104 00:10:40.390 ⇒ 00:10:44.360 Awaish Kumar: Load… loading part, where you have to either,
105 00:10:46.160 ⇒ 00:10:53.530 Awaish Kumar: like, create the table entirely, or add new rows, append new rows. This is more about right part of a table.
106 00:10:53.890 ⇒ 00:10:59.069 Awaish Kumar: What I… What I’m talking about is more read, queries.
107 00:10:59.720 ⇒ 00:11:08.170 Awaish Kumar: the table is already loaded. Now, if I’m curing it, it is taking longer than, expected time later.
108 00:11:08.170 ⇒ 00:11:20.369 Gideon Fernandez: Okay. So now it’s… it’s… so you’re saying that the table is already there, and now the, how would you optimize, like, the query performance so that you can actually see it? So…
109 00:11:20.830 ⇒ 00:11:24.789 Gideon Fernandez: One thing that I will also look at is if the table is being partitioned correctly.
110 00:11:24.900 ⇒ 00:11:40.519 Gideon Fernandez: if there’s any partitions happening, so that, it is, categorized or partitioned in certain, elements to see if that is, something that is slowing down, the query performance. So, I would look at that.
111 00:11:41.020 ⇒ 00:11:59.139 Gideon Fernandez: any… also, I would also look for if there’s some kind of Cartesian join happening, with a query, if we’re looking at long-running queries, are there ways for us to, see if there’s any type of table explosion that’s happening that’s causing it?
112 00:11:59.660 ⇒ 00:12:04.650 Gideon Fernandez: But yeah, I would start to look at what is exactly happening with those queries.
113 00:12:05.700 ⇒ 00:12:18.979 Awaish Kumar: So, what… what else… yeah, okay, let’s start… let’s… let’s talk about partitioning. So, since you mentioned it, but, you mentioned that you are working mostly with, Snowflake, so…
114 00:12:19.330 ⇒ 00:12:22.720 Awaish Kumar: How partitioning works in Snowflake.
115 00:12:24.190 ⇒ 00:12:28.259 Gideon Fernandez: Partitioning in Snowflake, so if I understand correctly.
116 00:12:28.650 ⇒ 00:12:33.919 Gideon Fernandez: It looks at, the… I guess the…
117 00:12:34.920 ⇒ 00:12:47.349 Gideon Fernandez: I would have to double-check, but if I recall, I saw this about a few months ago, is that it’s partitioned by, like, the dates and the date volumes, so it categorizes if there’s,
118 00:12:47.790 ⇒ 00:12:49.940 Gideon Fernandez: Some kind of…
119 00:12:50.530 ⇒ 00:13:00.510 Gideon Fernandez: date that can be partitioned by, so that you can, look at the most recent data, but I… don’t quote me on that, I… I don’t know off the top of my head.
120 00:13:01.860 ⇒ 00:13:02.590 Awaish Kumar: Okay.
121 00:13:05.250 ⇒ 00:13:10.090 Awaish Kumar: And, what about partitioning in Postgres?
122 00:13:14.330 ⇒ 00:13:17.470 Gideon Fernandez: Is there a ques… so what’s the question? What do you mean about that?
123 00:13:17.640 ⇒ 00:13:22.200 Awaish Kumar: Like… So, the partitioning is a single technique.
124 00:13:22.340 ⇒ 00:13:31.019 Awaish Kumar: But… The way it is you can apply is different based on different databases or data warehouses.
125 00:13:31.800 ⇒ 00:13:46.990 Awaish Kumar: I just heard two different names of the databases in your experience, or what you have worked on, like in your introduction, that you have worked with Postglass, and you also worked with Snowflake, so I just want to understand how partitioning works.
126 00:13:47.830 ⇒ 00:13:53.820 Awaish Kumar: So we talked about Snowflake, now I was asking more about how that work in Postgres.
127 00:13:54.040 ⇒ 00:14:01.709 Gideon Fernandez: So, the Postgres… when I mentioned Postgres, that was a data source that was managed by the client application.
128 00:14:01.710 ⇒ 00:14:05.209 Awaish Kumar: Okay, so… Actually worked, like, several…
129 00:14:05.350 ⇒ 00:14:09.270 Awaish Kumar: worked on setting up the database. It’s more just reading the data.
130 00:14:09.920 ⇒ 00:14:10.720 Gideon Fernandez: Correct.
131 00:14:10.930 ⇒ 00:14:11.600 Awaish Kumar: Okay.
132 00:14:12.300 ⇒ 00:14:16.220 Awaish Kumar: So, do you have any ex… any experience with any of the databases?
133 00:14:16.480 ⇒ 00:14:20.419 Awaish Kumar: Like, setting up, actually, the databases, schemas, tables.
134 00:14:20.980 ⇒ 00:14:22.759 Gideon Fernandez: I have in Snowflake.
135 00:14:24.010 ⇒ 00:14:31.529 Awaish Kumar: I mean more, like, transactional systems, like Postgres, SQL Server, My signal, anything?
136 00:14:32.670 ⇒ 00:14:37.230 Gideon Fernandez: So from an application point of view, can you clarify your question?
137 00:14:37.530 ⇒ 00:14:43.079 Awaish Kumar: Like any experience, either building Warehouses, or building an app.
138 00:14:43.600 ⇒ 00:14:48.439 Awaish Kumar: If… have you ever used… The data… transactional databases.
139 00:14:49.300 ⇒ 00:15:05.690 Gideon Fernandez: Yeah, so, building an analytics data warehouse, not necessarily an application database, but an analytics warehouse where it did receive transactional data, right? So, I’ve set up, schemas, I set up databases for that,
140 00:15:06.040 ⇒ 00:15:08.439 Awaish Kumar: Like, which database that you used?
141 00:15:08.860 ⇒ 00:15:09.830 Gideon Fernandez: Snowflake.
142 00:15:10.390 ⇒ 00:15:16.010 Awaish Kumar: Oh, okay, yeah, Mike, okay, never mind. My question was more…
143 00:15:16.570 ⇒ 00:15:21.049 Awaish Kumar: Have you ever set up the database, like, a transactional database?
144 00:15:22.670 ⇒ 00:15:29.809 Awaish Kumar: For example, sometimes, you can create those models in Postgres as well.
145 00:15:30.090 ⇒ 00:15:32.739 Awaish Kumar: When the data volume is very small.
146 00:15:32.960 ⇒ 00:15:41.339 Awaish Kumar: So my question was more about, like, in your role experience, have you ever used Postgres or SQL Server
147 00:15:41.460 ⇒ 00:15:46.710 Awaish Kumar: or MySQL in any of your projects for any kind of work.
148 00:15:47.730 ⇒ 00:15:50.590 Gideon Fernandez: I worked… so, when I was at Zen Business.
149 00:15:51.200 ⇒ 00:16:00.819 Gideon Fernandez: I worked on the… they were originally on the Postgres database before they migrated to Snowflake, so I would say within the first
150 00:16:00.980 ⇒ 00:16:11.379 Gideon Fernandez: 6 to 8 months, I was part of that effort, of that transition, and within… when I first started, all of that transactional data was in Postgres.
151 00:16:11.610 ⇒ 00:16:18.989 Gideon Fernandez: But only from, like, a analytics engineering where building models on top of it.
152 00:16:19.390 ⇒ 00:16:24.120 Gideon Fernandez: So that was in Postgres, before we converted over to Snowflake?
153 00:16:25.030 ⇒ 00:16:25.690 Awaish Kumar: Okay.
154 00:16:25.840 ⇒ 00:16:28.970 Awaish Kumar: So that was, like, also more about,
155 00:16:29.180 ⇒ 00:16:31.930 Awaish Kumar: Migration, and then building the models.
156 00:16:31.930 ⇒ 00:16:32.910 Gideon Fernandez: Yeah.
157 00:16:32.910 ⇒ 00:16:34.900 Awaish Kumar: About setting up itself, yeah.
158 00:16:39.570 ⇒ 00:16:43.720 Awaish Kumar: Okay, then I think I have a few more questions regarding…
159 00:16:43.840 ⇒ 00:16:48.800 Awaish Kumar: dbt, since we talked about materialization techniques,
160 00:16:49.860 ⇒ 00:16:54.730 Awaish Kumar: So, like, let’s take an example, like, for example.
161 00:16:55.970 ⇒ 00:17:04.290 Awaish Kumar: I’m working with dbt, but my data warehouse is a Postgres database, not actual warehouse.
162 00:17:04.470 ⇒ 00:17:05.180 Awaish Kumar: Come on.
163 00:17:05.500 ⇒ 00:17:06.400 Awaish Kumar: So…
164 00:17:06.930 ⇒ 00:17:14.800 Awaish Kumar: and I have a table, which is really big, and I want to insert more rows, is already an incremental model
165 00:17:14.950 ⇒ 00:17:17.630 Awaish Kumar: But still, it takes time to load new data.
166 00:17:17.810 ⇒ 00:17:18.490 Awaish Kumar: Oh.
167 00:17:19.270 ⇒ 00:17:22.780 Awaish Kumar: So, and it has indexes on some of the columns.
168 00:17:23.619 ⇒ 00:17:25.369 Awaish Kumar: So what I want to do is…
169 00:17:25.530 ⇒ 00:17:30.340 Awaish Kumar: Before, like, this model executes.
170 00:17:30.560 ⇒ 00:17:39.090 Awaish Kumar: the DVT actually runs this model. I want that DVT drops the indexes, For all the columns
171 00:17:39.510 ⇒ 00:17:41.340 Awaish Kumar: And then build the table.
172 00:17:41.500 ⇒ 00:17:43.479 Awaish Kumar: And then recreate the indexes.
173 00:17:43.830 ⇒ 00:17:47.770 Awaish Kumar: How can we achieve that, using DVD?
174 00:17:50.410 ⇒ 00:17:56.099 Gideon Fernandez: I wasn’t following that entire scenario. Maybe, could you restate that again?
175 00:17:57.120 ⇒ 00:18:01.560 Awaish Kumar: Yeah, so, so you’re familiar with indexing, right?
176 00:18:01.560 ⇒ 00:18:02.290 Gideon Fernandez: Yes.
177 00:18:02.490 ⇒ 00:18:03.050 Awaish Kumar: Okay.
178 00:18:03.560 ⇒ 00:18:09.799 Awaish Kumar: So, when a… A table has indexes, the load operation becomes slow.
179 00:18:11.300 ⇒ 00:18:14.200 Awaish Kumar: If the data insertion is… is huge.
180 00:18:14.930 ⇒ 00:18:22.680 Awaish Kumar: Right? So, my… the scenario is, for me, that I’m getting a lot of new
181 00:18:22.830 ⇒ 00:18:29.789 Awaish Kumar: Data to append into my existing table, and it is just slower to execute.
182 00:18:30.120 ⇒ 00:18:34.150 Awaish Kumar: So, one technique we use normally is to drop the indexes.
183 00:18:34.530 ⇒ 00:18:40.649 Awaish Kumar: and load the data, and then reapply the indexes. That way, it makes it faster.
184 00:18:40.820 ⇒ 00:18:48.369 Awaish Kumar: Basically, to load the data. And then apply… applying indexes is also faster than having to…
185 00:18:48.940 ⇒ 00:18:56.560 Awaish Kumar: insert rows when the indexes are applied. So normally, this technique is used to optimize your load dying.
186 00:18:57.190 ⇒ 00:19:02.099 Awaish Kumar: So… I want to do that without going outside of dbt.
187 00:19:02.320 ⇒ 00:19:07.590 Awaish Kumar: So, using dbt, how can I achieve that? Then I drop the indexes from a table.
188 00:19:08.280 ⇒ 00:19:12.309 Awaish Kumar: And dbt builds the model, and then it reapplies the indexes.
189 00:19:14.020 ⇒ 00:19:21.769 Gideon Fernandez: I haven’t encountered that scenario before, but I would look into that. Maybe, if I could draw a guess, it would be, like, some kind of full refresh.
190 00:19:22.700 ⇒ 00:19:32.829 Awaish Kumar: Yeah, fully refresh is… is a… just creates a table, right? From… Like, from the start.
191 00:19:33.250 ⇒ 00:19:37.319 Awaish Kumar: Right? It is not incremental anymore, if you apply it.
192 00:19:38.540 ⇒ 00:19:41.170 Gideon Fernandez: Yeah, I haven’t run into that scenario before, so I don’t know.
193 00:19:41.660 ⇒ 00:19:45.950 Awaish Kumar: Okay, so it’s… yeah, okay. I… that scenario was just to…
194 00:19:47.230 ⇒ 00:19:54.060 Awaish Kumar: let you understand what the situ… what I’m actually asking? But the question is more like, if we have a table and we want to apply
195 00:19:54.200 ⇒ 00:20:02.590 Awaish Kumar: some grants at the end of the model. If a table is created, and then I want to… so normally, for example,
196 00:20:03.280 ⇒ 00:20:06.499 Awaish Kumar: I want to apply any grants after that for any user.
197 00:20:08.090 ⇒ 00:20:12.399 Awaish Kumar: Alright, is there any feature in dbt that can help us do that?
198 00:20:14.020 ⇒ 00:20:15.930 Gideon Fernandez: I haven’t run into that scenario, so I don’t know.
199 00:20:19.090 ⇒ 00:20:22.400 Awaish Kumar: Okay, yeah, no worries. Moving on, I…
200 00:20:23.010 ⇒ 00:20:26.789 Awaish Kumar: Yeah, we still have some time left, so I might ask a few more.
201 00:20:26.910 ⇒ 00:20:33.869 Awaish Kumar: things regarding DVD, like, are you familiar with,
202 00:20:34.580 ⇒ 00:20:37.979 Awaish Kumar: macOS, and why we use it.
203 00:20:38.890 ⇒ 00:20:43.790 Gideon Fernandez: Yes. I am familiar with macros and Jinja macros,
204 00:20:43.930 ⇒ 00:20:48.729 Gideon Fernandez: Anytime there’s a function or some part of the query that needs to be reused.
205 00:20:49.150 ⇒ 00:20:59.060 Gideon Fernandez: That is something that I will… generally will build some kind of macro for, and then call that in a… using some… some Jinja variables.
206 00:21:00.470 ⇒ 00:21:05.170 Awaish Kumar: Are you familiar with any of the built-in macros.
207 00:21:06.070 ⇒ 00:21:07.530 Awaish Kumar: Like, yeah.
208 00:21:07.530 ⇒ 00:21:17.049 Gideon Fernandez: Yeah, the, like, so one macro that I’ll use is, like, the dbt utils. There’s a union function that’s been very helpful as a macro.
209 00:21:17.420 ⇒ 00:21:19.889 Gideon Fernandez: Rather than having to union
210 00:21:20.210 ⇒ 00:21:27.929 Gideon Fernandez: so many tables at the same time, I’ll… I’ll use the dbt utils package, and then there’s a…
211 00:21:28.460 ⇒ 00:21:32.930 Gideon Fernandez: I forget the name, but there’s a union function in there, a macro that I’ll use.
212 00:21:34.040 ⇒ 00:21:38.100 Awaish Kumar: Okay, yeah, like, the simplest one is also… is incremental.
213 00:21:38.400 ⇒ 00:21:39.990 Awaish Kumar: That we normally use.
214 00:21:40.530 ⇒ 00:21:41.490 Gideon Fernandez: Yes.
215 00:21:43.180 ⇒ 00:21:48.760 Awaish Kumar: And then, what’s the role of seeds?
216 00:21:50.310 ⇒ 00:21:51.259 Gideon Fernandez: I’m sorry?
217 00:21:51.760 ⇒ 00:21:56.450 Awaish Kumar: Yeah, what’s the, like, the role of seeds in DBD?
218 00:21:56.450 ⇒ 00:21:57.050 Gideon Fernandez: Whoa!
219 00:21:57.350 ⇒ 00:21:59.890 Gideon Fernandez: dbt Seeds? Yeah, that’s,
220 00:22:00.620 ⇒ 00:22:08.279 Gideon Fernandez: It’s… it’s more of, like, a reference or a lookup table, so if you have some kind of CFC… CSV spreadsheet,
221 00:22:08.800 ⇒ 00:22:15.480 Gideon Fernandez: The role for that is, to have some kind of reference, or to help you eliminate
222 00:22:15.650 ⇒ 00:22:21.699 Gideon Fernandez: writing so many case statements is where I found it the most valuable, so having to…
223 00:22:21.960 ⇒ 00:22:28.959 Gideon Fernandez: Build some kind of seed model, and then reference that into your downstream models has been, very helpful as well.
224 00:22:29.320 ⇒ 00:22:33.120 Awaish Kumar: Okay, I think I’m good with my,
225 00:22:33.270 ⇒ 00:22:37.710 Awaish Kumar: on the questions. We have still a few minutes left.
226 00:22:37.870 ⇒ 00:22:41.079 Awaish Kumar: So, I will leave this time for you to ask any questions.
227 00:22:42.000 ⇒ 00:22:46.369 Gideon Fernandez: Can you talk a bit more about Brainforge and the kind of projects that are available?
228 00:22:49.060 ⇒ 00:22:53.930 Awaish Kumar: Yeah, BrainForge is… Data NDI Consultancy Service.
229 00:22:54.500 ⇒ 00:23:01.490 Awaish Kumar: We provide, these services, basically, to the clients, and the kind of
230 00:23:02.070 ⇒ 00:23:04.539 Awaish Kumar: Services we provide are, for example.
231 00:23:05.210 ⇒ 00:23:14.699 Awaish Kumar: I will talk about the data part. So, on the data side, we might have, like, building the data foundations, setting up the data infrastructure, and
232 00:23:15.180 ⇒ 00:23:17.019 Awaish Kumar: Then, building the data mods.
233 00:23:18.010 ⇒ 00:23:19.510 Awaish Kumar: Creating dashboards.
234 00:23:20.900 ⇒ 00:23:23.330 Awaish Kumar: Right? So, these are the same…
235 00:23:23.510 ⇒ 00:23:27.900 Awaish Kumar: projects that you have worked on. And our tool stack is also similar.
236 00:23:29.120 ⇒ 00:23:35.299 Awaish Kumar: So… Yeah, like, we ingest data using any of the ingestion tools, similar to Fivetran.
237 00:23:35.890 ⇒ 00:23:38.700 Awaish Kumar: Then we utilize… Snowflake.
238 00:23:40.140 ⇒ 00:23:47.820 Awaish Kumar: But it’s… for some clients, it’s Snowflakes, for some it could be any other Postgres, BigQuery, It can be…
239 00:23:48.490 ⇒ 00:23:55.070 Awaish Kumar: Redshift, it can be… S3, anything, but it’s just, like, a similar… Boom.
240 00:23:55.250 ⇒ 00:24:02.400 Awaish Kumar: like the tech stake, that we have an ingestion tool, then we have a processing with dbt, then we have a data warehouse.
241 00:24:04.430 ⇒ 00:24:09.029 Awaish Kumar: And then we have BI tools, in the end of it. Yeah, so that’s the normal…
242 00:24:09.520 ⇒ 00:24:15.480 Awaish Kumar: flow. Right now, we have, like…
243 00:24:16.800 ⇒ 00:24:19.150 Awaish Kumar: Few clients that are actually using Snowflake.
244 00:24:19.280 ⇒ 00:24:24.890 Awaish Kumar: So… Yeah, so… Does that, like, answer your question, or…
245 00:24:24.890 ⇒ 00:24:28.039 Gideon Fernandez: Yeah, and can you talk about the projects there at Brainforge?
246 00:24:29.480 ⇒ 00:24:31.779 Awaish Kumar: Yeah, like…
247 00:24:32.480 ⇒ 00:24:39.049 Awaish Kumar: So we have, like, a few clients that are CPG clients, then there are some clients from the healthcare.
248 00:24:39.190 ⇒ 00:24:52.169 Awaish Kumar: For them, we… Some clients, like, with an e-commerce domain, So basically, what we… Two is we run different
249 00:24:52.500 ⇒ 00:24:55.229 Awaish Kumar: So W’s with them, and we come up with some…
250 00:24:56.130 ⇒ 00:25:08.420 Awaish Kumar: plan of a project, that could be anything, like, from setting up the data foundations. As I mentioned, so some clients come up with nothing, right? They have no data.
251 00:25:08.620 ⇒ 00:25:14.349 Awaish Kumar: set up, and then, our project is basically building the data foundations.
252 00:25:14.720 ⇒ 00:25:21.420 Awaish Kumar: That involves, like, spinning up Snowflake, Fivetran…
253 00:25:24.390 ⇒ 00:25:28.150 Awaish Kumar: dbt, or… and the BI tools, and then…
254 00:25:28.790 ⇒ 00:25:30.889 Awaish Kumar: Yeah, some people don’t need it.
255 00:25:31.230 ⇒ 00:25:38.040 Awaish Kumar: ingestion tools, we might go with orchestration tools like Dexter, write vital scripts to bring in the data.
256 00:25:39.810 ⇒ 00:25:45.479 Awaish Kumar: So, like, this is example of one of the projects where you just built the data foundations.
257 00:25:45.620 ⇒ 00:25:55.020 Awaish Kumar: Then there are examples of about, like, build a data warehouse for for a marketing…
258 00:25:55.240 ⇒ 00:26:03.320 Awaish Kumar: business domain. So, basically, given that project, we go after what are the different data sources for the marketing.
259 00:26:03.520 ⇒ 00:26:06.120 Awaish Kumar: Then, it comes with… comes the…
260 00:26:06.280 ⇒ 00:26:12.619 Awaish Kumar: ingestion power ingest using any of the ingestion tools, then data lens is one of the…
261 00:26:12.750 ⇒ 00:26:21.550 Awaish Kumar: data warehouse, and it depends on the client’s use cases. Once we talk with the client, get their feedback, and we suggest the tools based on the
262 00:26:21.780 ⇒ 00:26:35.190 Awaish Kumar: based on… based on the client needs. They are not fixed, so we don’t always say Snowflake if it’s not needed. So, then we come up with a set of tools, and then basically use those to ingest the data.
263 00:26:35.360 ⇒ 00:26:36.280 Awaish Kumar: Rusty.
264 00:26:36.680 ⇒ 00:26:46.930 Awaish Kumar: Run the transformation, and then… build the data mods. So, marketing mods, It will have, set of… models.
265 00:26:47.040 ⇒ 00:26:56.130 Awaish Kumar: that are basically derived from the requirements we gather from the client initially. So when we gather the requirements, we have, kind of.
266 00:26:56.320 ⇒ 00:27:02.300 Awaish Kumar: A documentation where we define what metrics or what KPIs we are tracking.
267 00:27:02.460 ⇒ 00:27:07.440 Awaish Kumar: And basically, based on that, we run our… Blink, click.
268 00:27:07.620 ⇒ 00:27:15.239 Awaish Kumar: from… route to, like, the bronze layer, silver layer, and the final gold layer very well.
269 00:27:15.400 ⇒ 00:27:17.869 Awaish Kumar: These metrics, and then,
270 00:27:19.200 ⇒ 00:27:30.089 Awaish Kumar: it’s connected to some of the BI tools, and we have an analyst that will then basically build the dashboards, refine, and work with the client to basically
271 00:27:30.360 ⇒ 00:27:33.219 Awaish Kumar: So they can adopt, adopt the dashboard.
272 00:27:36.330 ⇒ 00:27:43.010 Gideon Fernandez: Thank you. And can you talk about your background, and how you like, working at Brainforge?
273 00:27:44.040 ⇒ 00:27:49.330 Awaish Kumar: Yeah, my background is more about data, like, I’m kind of a first-tech data engineer.
274 00:27:51.100 ⇒ 00:27:55.670 Awaish Kumar: I specialize in, setting up the data infrastructure and
275 00:27:57.970 ⇒ 00:28:04.600 Awaish Kumar: as just a data engineer, but then I have worked with, experience being analytics engineer.
276 00:28:04.760 ⇒ 00:28:11.609 Awaish Kumar: I have experience being a data analyst, I have experience building dashboards, So, and then flu.
277 00:28:12.700 ⇒ 00:28:16.459 Awaish Kumar: I have around 10 years of experience as a data engineer.
278 00:28:16.900 ⇒ 00:28:19.390 Awaish Kumar: and have worked it in, like,
279 00:28:19.660 ⇒ 00:28:26.760 Awaish Kumar: with the startups, growth stage companies, I have worked across continents,
280 00:28:27.480 ⇒ 00:28:31.349 Awaish Kumar: working in, like, Europe, US, actually is working in Asia.
281 00:28:31.550 ⇒ 00:28:36.960 Awaish Kumar: And then… Yeah, that’s basically…
282 00:28:37.160 ⇒ 00:28:42.369 Awaish Kumar: it, and I have experience with all the tools and technologies we have discussed so far.
283 00:28:42.490 ⇒ 00:28:50.629 Awaish Kumar: And then… Being at Brainforge is, kind of… I joined this company one year ago.
284 00:28:51.170 ⇒ 00:28:54.250 Awaish Kumar: And, like, we grew from being,
285 00:28:55.310 ⇒ 00:28:59.479 Awaish Kumar: Maybe 5-6 people in the company to now 30 people in a year.
286 00:29:01.260 ⇒ 00:29:06.240 Awaish Kumar: Although we don’t depend on health counts, we don’t count that, like, we are not…
287 00:29:06.370 ⇒ 00:29:11.269 Awaish Kumar: We are not, like, a company with just want to increase their headcounts, but we are more, like.
288 00:29:11.420 ⇒ 00:29:17.359 Awaish Kumar: Want to grow so that, like, you know, more… get more clients, get more work done,
289 00:29:17.720 ⇒ 00:29:20.560 Awaish Kumar: And improve… optimize and improve our workflows.
290 00:29:20.660 ⇒ 00:29:27.089 Awaish Kumar: But as a… As a byproduct of doing that, we got more clients, and we had to
291 00:29:27.420 ⇒ 00:29:29.020 Awaish Kumar: hire more people.
292 00:29:31.180 ⇒ 00:29:36.189 Awaish Kumar: Yeah, so it has been a nice growth being here. I have been,
293 00:29:37.110 ⇒ 00:29:42.020 Awaish Kumar: And Brain Forge also, I have been doing everything from being full stack.
294 00:29:42.290 ⇒ 00:29:45.150 Awaish Kumar: And to now being more on…
295 00:29:45.440 ⇒ 00:29:48.849 Awaish Kumar: architect level, and then we have ICs that can support
296 00:29:48.960 ⇒ 00:29:52.779 Awaish Kumar: The work downstream, and then,
297 00:29:53.450 ⇒ 00:29:56.720 Awaish Kumar: Yeah, it’s a fast-paced environment.
298 00:29:58.960 ⇒ 00:30:03.539 Awaish Kumar: You might have to work, like, for more than two clients at the same time.
299 00:30:05.670 ⇒ 00:30:12.720 Awaish Kumar: With, obviously, the 40 hours cap per week, but that time is being divided into…
300 00:30:13.050 ⇒ 00:30:15.430 Awaish Kumar: Multiple lines, and then we are…
301 00:30:15.800 ⇒ 00:30:28.329 Awaish Kumar: kind of use AI, basically, in all our workflows, including the non-tech teams, like sales and marketing. They also use AI to basically optimize their workflows.
302 00:30:28.570 ⇒ 00:30:33.940 Awaish Kumar: So everybody is AI-driven, and… Yeah, that’s basically…
303 00:30:35.210 ⇒ 00:30:43.830 Gideon Fernandez: And is this, for this particular role, is there already a project lined up for this role, or is this just more of, like.
304 00:30:44.000 ⇒ 00:30:49.569 Gideon Fernandez: Being able to, get staff on a particular project that may be coming up.
305 00:30:50.600 ⇒ 00:30:57.679 Awaish Kumar: So… Basically, We are hiring right now, because…
306 00:30:58.300 ⇒ 00:31:01.369 Awaish Kumar: The projects we are working on are actually…
307 00:31:01.600 ⇒ 00:31:04.140 Awaish Kumar: Recired a lot of work, and…
308 00:31:05.540 ⇒ 00:31:09.410 Awaish Kumar: I want to support our team to basically also get some time,
309 00:31:09.580 ⇒ 00:31:13.019 Awaish Kumar: For learning and development, and for basically…
310 00:31:14.650 ⇒ 00:31:22.070 Awaish Kumar: like, delivering the work, quality work. So it’s more like… we also have ongoing projects where you will be assigned
311 00:31:22.250 ⇒ 00:31:24.070 Awaish Kumar: From on, and then,
312 00:31:24.260 ⇒ 00:31:29.979 Awaish Kumar: We have new leads in the pipeline, also, that are coming up, so it’s more… mix of both things.
313 00:31:31.660 ⇒ 00:31:34.669 Awaish Kumar: Not just one client, you might be… you might have a…
314 00:31:34.830 ⇒ 00:31:38.710 Awaish Kumar: If you are hired, you might have to work on more than 2 or 3 clients.
315 00:31:38.970 ⇒ 00:31:39.800 Awaish Kumar: Boom.
316 00:31:40.040 ⇒ 00:31:44.420 Awaish Kumar: Right now, we are also in need of the resources.
317 00:31:44.810 ⇒ 00:31:47.020 Awaish Kumar: since I… I also mentioned that.
318 00:31:47.630 ⇒ 00:31:50.320 Awaish Kumar: And it’s growing, so client base is growing as well.
319 00:31:51.790 ⇒ 00:31:56.069 Gideon Fernandez: That’s great to hear that the project and the client demand is growing over time.
320 00:31:56.730 ⇒ 00:32:04.999 Awaish Kumar: Yeah, so we have a dedicated team of, like, a sales team that That basically, like,
321 00:32:05.210 ⇒ 00:32:08.930 Awaish Kumar: Bring… follows the lead, and then brings the client in.
322 00:32:09.140 ⇒ 00:32:13.719 Awaish Kumar: So, we are basically… I’m not documents lined up.
323 00:32:15.350 ⇒ 00:32:16.299 Gideon Fernandez: Super cool.
324 00:32:16.480 ⇒ 00:32:18.169 Gideon Fernandez: Thank you for the overview.
325 00:32:19.850 ⇒ 00:32:26.650 Awaish Kumar: Okay, I think we are on time, so if you don’t have any more questions, yeah, we can, basically
326 00:32:28.560 ⇒ 00:32:30.909 Awaish Kumar: Yeah, we are basically done today.
327 00:32:33.280 ⇒ 00:32:36.999 Awaish Kumar: I think you are already brief about the next steps by Kayla.
328 00:32:37.860 ⇒ 00:32:42.889 Gideon Fernandez: Okay, yes, she gave me the plan, so I look forward to hearing back from her.
329 00:32:43.100 ⇒ 00:32:43.719 Gideon Fernandez: Thank you.
330 00:32:43.940 ⇒ 00:32:47.519 Awaish Kumar: So, yeah, she will get back to you as soon as possible. Thank you.
331 00:32:47.520 ⇒ 00:32:50.729 Gideon Fernandez: Alright, thank you. Nice talking to you. Take care, bye.
332 00:32:50.730 ⇒ 00:32:51.769 Awaish Kumar: Thank you, bye.