Meeting Title: Brainforge Interview w- Awaish Date: 2026-02-06 Meeting participants: Awaish Kumar, Bijil
WEBVTT
1 00:00:10.830 ⇒ 00:00:11.790 Awaish Kumar: To the road.
2 00:00:18.650 ⇒ 00:00:19.670 Bijil: Hello?
3 00:00:19.830 ⇒ 00:00:20.919 Bijil: How are ya?
4 00:00:21.400 ⇒ 00:00:21.970 Awaish Kumar: I agree.
5 00:00:22.470 ⇒ 00:00:23.539 Awaish Kumar: How you doing?
6 00:00:24.050 ⇒ 00:00:27.200 Bijil: Good, good, good. I say your name, Awash?
7 00:00:29.550 ⇒ 00:00:30.270 Awaish Kumar: Yep.
8 00:00:32.330 ⇒ 00:00:33.670 Awaish Kumar: Yeah.
9 00:00:33.670 ⇒ 00:00:34.520 Bijil: Estimated.
10 00:00:34.940 ⇒ 00:00:41.800 Awaish Kumar: We are actually going to… like, this…
11 00:00:42.490 ⇒ 00:00:45.800 Awaish Kumar: the introductory session, we are just going to talk about…
12 00:00:46.320 ⇒ 00:00:50.360 Awaish Kumar: You, your experiences, and what you have been doing so far.
13 00:00:50.760 ⇒ 00:00:52.020 Bijil: John Thief.
14 00:00:52.020 ⇒ 00:01:00.509 Awaish Kumar: also to share with you what we do at Brain Forge, and… If you can fit,
15 00:01:00.670 ⇒ 00:01:02.330 Awaish Kumar: Pretty good, okay?
16 00:01:03.340 ⇒ 00:01:04.160 Bijil: Good to me.
17 00:01:04.769 ⇒ 00:01:08.749 Awaish Kumar: Okay. Yeah, we can start with your brief introduction.
18 00:01:09.420 ⇒ 00:01:14.670 Bijil: Yeah, sure, sure, sure. My name is Brigil, I’m from Sydney, Australia. I’ve been living here
19 00:01:15.390 ⇒ 00:01:17.460 Bijil: Quite a long time now.
20 00:01:17.910 ⇒ 00:01:33.270 Bijil: originally a chemical engineer, and then I sort of pivoted into data machine learning about 6 years ago when I was doing my PhD, and then after the PhD, I did a brief stint as a data scientist for a little while.
21 00:01:33.530 ⇒ 00:01:48.340 Bijil: And didn’t really… well, I enjoyed it, but didn’t really find a true satisfaction in that, so I pivoted into data engineering. And, since then, I’ve been doing data engineering. Worked with a consultancy firm for about 3 years or so.
22 00:01:48.550 ⇒ 00:02:07.730 Bijil: working across a handful of enterprise and startups, building out data platforms, uplifting migration, things like that. And, here I go. Just under a year ago, I left that job and joined a startup as their first sort of data engineer, building out their data platform.
23 00:02:08.850 ⇒ 00:02:27.840 Bijil: And I recently finished that up as well. Along the way, about two and a half years ago or so, I also started my own agency as well, where I sort of advised companies on data platforms and building data products and analytics and a lot of the data engineering infrastructure. So I’ve been doing that for a while.
24 00:02:28.490 ⇒ 00:02:46.060 Bijil: So that was my reason for leaving the startup, because I wanted to focus a little bit more on, you know, more of those kind of work, and that’s where you guys… I saw you guys are hiring, and I thought I’ll come and reach out, and see if there’s an opportunity to take on some of the projects from you guys, and help you guys out as well.
25 00:02:47.000 ⇒ 00:02:53.110 Awaish Kumar: Okay, so how… Like, why you have left your last job?
26 00:02:53.870 ⇒ 00:02:59.440 Bijil: Why I loved my job? Yeah, focusing on my own business, so focusing on building on my own agency.
27 00:02:59.800 ⇒ 00:03:09.719 Bijil: trying to advise more companies, trying to take on more projects, and all of that, and I can’t really do that. There’s only so much you can juggle with a full-time job, so I had to, like.
28 00:03:09.900 ⇒ 00:03:13.239 Bijil: I had to do my own thing, and that’s one of the reasons I left that job.
29 00:03:13.420 ⇒ 00:03:18.150 Bijil: But I might still have them as one of my clients, though. They might be coming on board as…
30 00:03:18.310 ⇒ 00:03:25.469 Bijil: Might be helping, advising them, like, once a day, or once a week, or something like that, for a little while longer, till they find a replacement.
31 00:03:25.770 ⇒ 00:03:29.920 Awaish Kumar: I think I have your profile as Data Genia with me.
32 00:03:30.140 ⇒ 00:03:32.790 Awaish Kumar: So… Are you looking to cover.
33 00:03:32.790 ⇒ 00:03:33.460 Bijil: remote.
34 00:03:35.420 ⇒ 00:03:40.209 Awaish Kumar: More kind of collaboration between your company, or… how are you looking for this?
35 00:03:40.210 ⇒ 00:04:03.799 Bijil: Data engineer, but I usually work with different agencies, like yourself, where you guys probably have, like, more of a subcontractor capacity, where I come in as one of their… one of your contractors, so it doesn’t really have to be through… doesn’t have to be, like, a company collaboration or anything like that, it’s just an individual, but I’m just working for you guys, representing BrainForge.
36 00:04:03.800 ⇒ 00:04:06.910 Bijil: As your own employee, and that kind of thing, yeah.
37 00:04:07.510 ⇒ 00:04:17.660 Awaish Kumar: Yeah, so BreadForges currently works with this model, that they hire people remotely from all over the world.
38 00:04:17.890 ⇒ 00:04:27.339 Awaish Kumar: as… As consultants, but then, like, we expect everybody here to actually work at least 40 hours per week.
39 00:04:28.630 ⇒ 00:04:30.419 Awaish Kumar: So is that okay with you, or your…
40 00:04:30.420 ⇒ 00:04:35.430 Bijil: Yeah, yeah, that’s completely fine with me, because I’m perfectly fine with availability right now, yeah.
41 00:04:36.360 ⇒ 00:04:42.229 Awaish Kumar: Okay, great, moving on. So, I think you already know a little bit about Brave Forge.
42 00:04:42.500 ⇒ 00:04:45.369 Awaish Kumar: at Bradford, what we are doing is basically…
43 00:04:45.590 ⇒ 00:04:50.280 Awaish Kumar: We are data consultancy, primarily in the heart of it.
44 00:04:50.380 ⇒ 00:04:57.790 Awaish Kumar: But then we are a combination of data and AI consistency services. So we provide data services.
45 00:04:58.060 ⇒ 00:05:06.049 Awaish Kumar: plus AI services. AI could be, something we are building on top of data, which we’ve already
46 00:05:06.230 ⇒ 00:05:09.689 Awaish Kumar: Created, or it’s something like,
47 00:05:10.040 ⇒ 00:05:13.469 Awaish Kumar: Creating chatbots for the companies, or…
48 00:05:13.870 ⇒ 00:05:19.209 Awaish Kumar: Are doing some, like, working on some segmentation models.
49 00:05:21.190 ⇒ 00:05:28.229 Awaish Kumar: classifications, models, things like that. So, it could be anything, but yeah.
50 00:05:28.370 ⇒ 00:05:36.329 Awaish Kumar: We are an AI-heavy company, so, like, everything we do is kind of, we have AI.
51 00:05:36.580 ⇒ 00:05:44.330 Awaish Kumar: In our life, so… Sure. Starting from engineering, to operations, to marketing.
52 00:05:44.560 ⇒ 00:05:50.120 Awaish Kumar: Everybody here uses… AI to help their workflows.
53 00:05:50.510 ⇒ 00:05:56.920 Awaish Kumar: That’s what, basically, we do, and for each of our employees.
54 00:05:57.130 ⇒ 00:06:01.660 Awaish Kumar: At least in the delivery side, we have people working on
55 00:06:02.040 ⇒ 00:06:05.430 Awaish Kumar: Two, two, three clients simultaneously.
56 00:06:06.840 ⇒ 00:06:07.620 Awaish Kumar: Yep.
57 00:06:08.140 ⇒ 00:06:11.550 Awaish Kumar: Yeah, that’s for the Brave Forge.
58 00:06:11.870 ⇒ 00:06:17.630 Awaish Kumar: Then, yeah, I would love to know, like… Already, you briefly,
59 00:06:17.830 ⇒ 00:06:29.310 Awaish Kumar: explain what you have been working on with what companies. I would love to know your, like, experience with the recent tools in the market, which is, like, Snowflake, dbt.
60 00:06:29.470 ⇒ 00:06:40.740 Awaish Kumar: the engine tools, like FiveTrad, Polytopic, real-time streaming, batch streaming…
61 00:06:41.160 ⇒ 00:06:45.230 Awaish Kumar: Like, are your experiences across all these different things?
62 00:06:45.940 ⇒ 00:06:57.740 Bijil: Yeah, of course, no worries. So… I started all of my… a lot of my early, sort of, careers started to focus a lot on the GCP stack, so I focused a lot of on the GCP side of things, so thinking about your…
63 00:06:58.020 ⇒ 00:07:11.370 Bijil: your data form, Dataflow, BigQuery, PubSub, all of that stuff for a very long time, built a lot of platforms, and helped a lot of teams with migrating to that platform at the early days.
64 00:07:11.410 ⇒ 00:07:28.049 Bijil: That also involves things like, you know, potentially using Cloud Composer, Airflow, or DBT, that kind of stuff as well, all of that. And then, perhaps a year, year and a half into my career, I started pivoting into some of the other tools as well, like Databricks, so I did a…
65 00:07:28.210 ⇒ 00:07:37.549 Bijil: I did build one of the largest… one of the largest directs platforms in the country, for one of the energy retailers, end-to-end design and build of it.
66 00:07:37.630 ⇒ 00:07:49.650 Bijil: So that was one of the data bricks, and that was mostly all batch, perhaps a couple of streaming pipelines, but I didn’t own the streaming pipelines, so I basically set up the infrastructure for it, so all of the…
67 00:07:49.790 ⇒ 00:08:04.260 Bijil: I guess the cloud and the infrastructure behind the Databricks infrastructure was built by me. So that’s Databricks side of things. I also built an end-to-end database platform for a startup over in Canada.
68 00:08:04.530 ⇒ 00:08:19.169 Bijil: So that was basically the design, ingestion, transformation, all the way to reporting. That was purely Databricks stack, including Azure as the main cloud provider in that one, and dbt.
69 00:08:19.720 ⇒ 00:08:33.650 Bijil: Along the way, I worked at Snowflake briefly. I didn’t do a lot of Snowflake work. The only time I did Snowflake work was migrating someone from Snowflake to Databricks, so that’s what I did there for one of the clients. Then,
70 00:08:34.419 ⇒ 00:08:45.000 Bijil: dbt was probably a consistent sort of plot stack that I used across in the last 5 years. Almost every project that I worked on had some form of dbt, and I also…
71 00:08:45.370 ⇒ 00:08:58.550 Bijil: Do a lot of coaching nowadays for dbt as well, like, so I coach Siemens engineers in Germany, for their dbt upskilling and things like that, so that’s one of the things that I do on the side.
72 00:09:00.150 ⇒ 00:09:06.000 Bijil: So, DVDs, I’m familiar with it. And lately I’ve been working a fair bit on Mother Dark.
73 00:09:07.040 ⇒ 00:09:09.480 Bijil: And… what else?
74 00:09:10.380 ⇒ 00:09:21.069 Bijil: Yeah, Mother Ducker, probably a bit of AWS work as well, like, but not my strong suit when it comes to cloud. AWS is not my strong suit, but I’ve done a bit of work with AWS as well.
75 00:09:21.460 ⇒ 00:09:33.480 Bijil: My favorite would be going to GCP, and yeah, that’s mostly Instruct. In terms of the ingestion tools, I’ve done a little bit of work with Fivetran, not intensely. That’s not… hasn’t been my main focus.
76 00:09:33.570 ⇒ 00:09:42.960 Bijil: I’ve done a lot of work with Polytomic lately. One of my clients been using that a lot, so I’ve done a lot of work with Polytomic, which is interesting. A lot of DLTHub,
77 00:09:43.800 ⇒ 00:09:54.870 Bijil: ingestion-wise, yeah, that’s mostly it. And I probably use, like, native integrations with a lot of the BigQuery integration with some of the tools, or Databricks integration with some of the tools, that’s ingestion as well.
78 00:09:55.320 ⇒ 00:09:57.510 Bijil: But, yeah, learned a lot of fun.
79 00:09:58.560 ⇒ 00:10:01.630 Awaish Kumar: Okay, so, like, you can talk about one of your…
80 00:10:01.950 ⇒ 00:10:05.609 Awaish Kumar: The most complicated project which we’re proud of.
81 00:10:05.840 ⇒ 00:10:07.300 Awaish Kumar: So, what do we want to…
82 00:10:07.400 ⇒ 00:10:11.180 Awaish Kumar: Figure out is we want to… Like, see the problem?
83 00:10:11.280 ⇒ 00:10:17.560 Awaish Kumar: Then the approach you took, alternatives considered, and the… what results were achieved.
84 00:10:18.240 ⇒ 00:10:25.639 Bijil: Yeah, of course. I guess one of the… I’ll probably talk about the database migration one that kind of comes to my mind quite fresh.
85 00:10:25.650 ⇒ 00:10:39.139 Bijil: So, this was probably a year, a couple years ago now. The client approached me to migrate from Snowflake to Dartbreaks, and main concern for them was cost. They were paying a lot of money.
86 00:10:39.230 ⇒ 00:10:45.220 Bijil: And that was not just Norfleco, they were also paying a lot of money to Portable at the time, as a ELT tool they were using.
87 00:10:45.330 ⇒ 00:10:56.120 Bijil: So they were paying quite a few, quite a few, their bills were quite high for the month, and they wanted to move to Databricks. At the time, they weren’t, like.
88 00:10:56.120 ⇒ 00:11:16.590 Bijil: They made the decision themselves that this is the solution they want, so I couldn’t actually say, hey, this is not the best solution, let’s try and optimize Snowflake. They really wanted to move to Databricks. Also, the founder was quite a big fan of Databricks, so we had to do it. There’s no other way around it. I could have probably gone back and said, let’s optimize Snowflake, and let’s optimize portable, let’s reduce the cost.
89 00:11:16.590 ⇒ 00:11:22.949 Bijil: But it’s a different problem, we didn’t go there. So I opted to migrate, and I was at the beginning, at the time.
90 00:11:22.950 ⇒ 00:11:27.500 Bijil: just starting to learn Databricks, I wasn’t an expert in Databricks at all.
91 00:11:27.570 ⇒ 00:11:30.200 Bijil: I decided to take on the project,
92 00:11:30.310 ⇒ 00:11:49.220 Bijil: being confident that I can pick things up as I go. So what I did was essentially, well, consulted with one of my close, sort of, colleagues. He’s a direct solution architect, and I had him on, like, almost like a speed dial on a weekly basis, and I would like to consult with him.
93 00:11:49.390 ⇒ 00:11:59.229 Bijil: So initially, you know, I have some experience with Databricks, I did some certifications, but certifications are not going to help with the actual project that much, but with that sort of background.
94 00:11:59.230 ⇒ 00:12:18.770 Bijil: I designed the entire architecture on how the migration could look like, how different workspaces could look like, and what that means, how can we sort of minimize the cost for them in the long run, what sort of architecture can we use, what sort of workspace differentiations can we use, should we go serverless or not serverless, and things like that. So I made all the decisions and drew it all up,
95 00:12:19.200 ⇒ 00:12:27.129 Bijil: Then I sort of consulted with my sort of colleague, who’s sort of architecting the space, and sort of brainstormed where can we tweak, where can we make it better.
96 00:12:27.500 ⇒ 00:12:44.020 Bijil: That went on for the first couple of weeks of the project. This is about a six-weeks project, by the way. Just a couple of weeks, that discovery with my colleague, myself, and with the client kind of went along, where I designed the entire thing, once we have that finalized in, like, two weeks’ time.
97 00:12:44.020 ⇒ 00:12:52.280 Bijil: Then I went and implemented the entire thing. One thing I want to also add, I also removed portable from the equation and brought in TLT Hub.
98 00:12:52.280 ⇒ 00:12:59.100 Bijil: That was the other thing that we brought into it, just to reduce the cost of the portable. Portable was taking up almost 50% of the cost for them.
99 00:12:59.180 ⇒ 00:13:16.739 Bijil: So… anyway, the next 4 weeks, I ended up building the entire platform end-to-end, having no experience with Databricks, migrated them from Snowflake to Databricks, and we had a cost reduction of close to 90%, and the client was really happy about it, and more importantly.
100 00:13:16.780 ⇒ 00:13:22.679 Bijil: I ended up working with them for about the next 8 months as well, after the project, on various other initiatives, but…
101 00:13:22.680 ⇒ 00:13:23.140 Awaish Kumar: Nice.
102 00:13:23.140 ⇒ 00:13:23.800 Bijil: Probably.
103 00:13:24.150 ⇒ 00:13:30.060 Awaish Kumar: Okay, yeah, I would love to understand what actually got migrated, like… I understand.
104 00:13:30.680 ⇒ 00:13:35.890 Awaish Kumar: you change the tool from Portable to DLT Hub, Yep.
105 00:13:36.060 ⇒ 00:13:41.529 Awaish Kumar: Is that understandable? Like, you plugged in something else? But what was actual…
106 00:13:42.000 ⇒ 00:13:45.869 Awaish Kumar: a job as a DEA that…
107 00:13:46.280 ⇒ 00:13:50.969 Awaish Kumar: were considered, that was complicated, or…
108 00:13:51.160 ⇒ 00:13:52.430 Awaish Kumar: What, what was it a little bit?
109 00:13:52.430 ⇒ 00:13:59.659 Bijil: Yeah, yeah, yeah. So in this specific case, they had about 90-plus endpoints that they were pulling the data from, basically.
110 00:13:59.740 ⇒ 00:14:04.049 Bijil: That was why portable was useful, for many reasons.
111 00:14:04.060 ⇒ 00:14:23.109 Bijil: And to convert all of that into DLT hub in a manner that allows you to keep adding more to it easily was probably the most trickiest part in the whole equation. So, to give you an idea, the client is another consultancy firm. They would consult with more customers.
112 00:14:23.110 ⇒ 00:14:36.660 Bijil: And they would consult with more customers on a specific CRM, and they advise them on analytics. So that’s their business model. And my job is to build a system that allows the client to bring in more clients easily.
113 00:14:36.910 ⇒ 00:14:43.569 Bijil: So I build, essentially, the ingestion platform, what I build there for them, that goes into Databricks, they say.
114 00:14:43.680 ⇒ 00:14:44.360 Bijil: Yep.
115 00:14:44.360 ⇒ 00:14:49.219 Awaish Kumar: I get it, but I… like, when we’re talking about this migration project, where you…
116 00:14:49.330 ⇒ 00:14:55.390 Awaish Kumar: Right there, we have iPads from Snowflake to… Databricks.
117 00:14:56.850 ⇒ 00:15:04.390 Awaish Kumar: Yeah, let’s say, like, let’s talk about that, like… like…
118 00:15:05.380 ⇒ 00:15:08.539 Awaish Kumar: When you talk about the ingestion part, that’s…
119 00:15:08.750 ⇒ 00:15:10.699 Bijil: That is different, like…
120 00:15:10.700 ⇒ 00:15:13.480 Awaish Kumar: I understand you might have done as part of project.
121 00:15:14.020 ⇒ 00:15:18.890 Awaish Kumar: But still, I want to keep the focus on other stuff of this,
122 00:15:20.200 ⇒ 00:15:22.659 Awaish Kumar: of this project, apart from registration.
123 00:15:23.420 ⇒ 00:15:28.119 Bijil: Okay, sure. What exactly do you want? Transformation? The security side?
124 00:15:28.120 ⇒ 00:15:28.470 Awaish Kumar: That’s.
125 00:15:28.470 ⇒ 00:15:29.020 Bijil: production.
126 00:15:29.680 ⇒ 00:15:34.660 Awaish Kumar: Like, what transformations were done? Was there… prone.
127 00:15:35.040 ⇒ 00:15:36.480 Awaish Kumar: Like, what kind of a…
128 00:15:37.100 ⇒ 00:15:43.839 Awaish Kumar: you know, like, there will… there was maybe some kind of a SQL, which you just changed into a Python script, or…
129 00:15:44.230 ⇒ 00:15:45.559 Awaish Kumar: Yeah, yeah, yeah.
130 00:15:45.720 ⇒ 00:15:47.510 Awaish Kumar: Or whatever, like…
131 00:15:48.400 ⇒ 00:15:49.090 Bijil: Sure.
132 00:15:50.310 ⇒ 00:15:56.740 Awaish Kumar: And what it took you to do that, and then finally… like,
133 00:15:58.600 ⇒ 00:16:13.650 Awaish Kumar: Was that… was the time enough to do all that? Like, while now I’m hearing that, like, you did… you implemented 90 plus connectors in, like, 4 weeks, along with all the migration. Were you alone doing that, or…
134 00:16:14.900 ⇒ 00:16:16.010 Awaish Kumar: 100%.
135 00:16:17.050 ⇒ 00:16:24.009 Bijil: Happy to help, happy to answer. So, I did say that I worked with them for another 8 months after that as well, so sort of an extension of the project.
136 00:16:24.100 ⇒ 00:16:32.240 Bijil: But anyway, going back to the 6 weeks project, the migration was the scope, so there was more of a lift and shift. I didn’t have to do a whole lot of changes.
137 00:16:32.240 ⇒ 00:16:53.499 Bijil: So they didn’t have a whole lot of SQL already working. They had a dbt connected to their Snowflake platform, doing some transformations. We did migrate that to Databricks. Had to make some minor changes, because Databricks SQL and Snowflake SQL’s, like, few changes, but mostly similar. But that’s very minimal changes, so we didn’t really actually change the business logic. Obviously, there were some mistakes within the business logic, like, there was some…
138 00:16:53.500 ⇒ 00:17:03.099 Bijil: the way they build the models perhaps wasn’t the best way to… or perhaps the structure of the dbt project wasn’t the best, and all of that stuff. We didn’t… we didn’t touch on all… I didn’t touch any of those things.
139 00:17:03.600 ⇒ 00:17:10.679 Bijil: At the time, and that was sort of just called out as things we can address in the future, and that’s one of the reasons it got extended.
140 00:17:10.970 ⇒ 00:17:14.500 Bijil: And then we moved everything to Snowflake.
141 00:17:14.640 ⇒ 00:17:32.450 Bijil: I’m sorry, to Databricks, without any of those changes, and plugged in the Databricks to dbt connection, you know, dbt Core at the time. Oh, yeah, they were on dbt Cloud before I moved them to dbt Core. That was a little bit of cost saving, not a lot of cost saving for them, but still something for them too as well.
142 00:17:32.610 ⇒ 00:17:48.199 Bijil: That was done, too. It was all done by me, I was the only one on the team. At the beginning, maybe two weeks, I had, like I said, I had a colleague I would consult for one hour a week, just on the architecture side of things, so that’s probably the only help I had there, but everything else was pretty much done by me.
143 00:17:48.490 ⇒ 00:18:05.560 Bijil: all done by me. The client is not technical, really, so he was completely off the technical stuff, so it didn’t really help anything there. He didn’t know how to write SQL, so they helped a little bit on the SQL side, because they already had the SQL, so they helped me understand the business logic a little bit. So that’s that.
144 00:18:06.060 ⇒ 00:18:09.870 Bijil: So that’s all the transformation side, that’s the SQL migration there.
145 00:18:09.990 ⇒ 00:18:26.680 Bijil: The ingestion I already mentioned talked about… I did a whole heap of security side as well for them, which is something that was important for them. They had to make sure they had fine access control to some of the data sets and some of the catalogs and things like that. So, they built an entire Unity catalog.
146 00:18:26.680 ⇒ 00:18:30.039 Bijil: On top of the database, and design that properly as well.
147 00:18:31.470 ⇒ 00:18:47.060 Bijil: And what else have I done in that one? The reporting, probably, is probably the last bit. There was one dashboard I migrated as well. There was probably… they had, like, a few dashboards they previously had in Snowflake, but as part of the end-to-end migration, I also showed them
148 00:18:47.060 ⇒ 00:19:03.629 Bijil: how to migrate one… how to actually build one dashboard, just as a proof of concept that it can be done in Databricks Works workspace, because it’s, like, a native feature that comes out of it as well. So yeah, it was all pretty much done in four weeks, and it sounds like a lot, but it wasn’t… wasn’t that much…
149 00:19:04.060 ⇒ 00:19:05.310 Awaish Kumar: Did you write it in?
150 00:19:05.700 ⇒ 00:19:10.650 Awaish Kumar: Snowflake, data, which already was in Snowflake, right?
151 00:19:12.890 ⇒ 00:19:17.290 Bijil: We did a full migration, so essentially we basically re-pulled all the data.
152 00:19:17.560 ⇒ 00:19:18.490 Awaish Kumar: Like…
153 00:19:18.890 ⇒ 00:19:26.850 Awaish Kumar: I understand that you moved the switch to the connections, now the tool is moving, the TLT is setting data tool.
154 00:19:27.030 ⇒ 00:19:28.240 Awaish Kumar: Databricks…
155 00:19:29.370 ⇒ 00:19:38.990 Awaish Kumar: dbt also, like, made a switch that is territory called DLT, but there is a lot of historical data, which would have stayed in Snowflake.
156 00:19:39.140 ⇒ 00:19:42.139 Awaish Kumar: How would it move that to Dermix?
157 00:19:42.140 ⇒ 00:19:45.649 Bijil: Yeah, good question. So, the Snowflake…
158 00:19:45.730 ⇒ 00:19:55.290 Bijil: Portable was sending the tables directly to Snowflake. One of the decisions we made early on is we’ll just pull all the historical data from the endpoints.
159 00:19:55.290 ⇒ 00:20:06.350 Bijil: Because it is accessible, and it’s not a huge amount of data for them. When there was, seems like 90 endpoints, there’s only one or two endpoints that were heavy, so we just pulled all the historical data from the endpoint and sort of stored in our Azure.
160 00:20:06.350 ⇒ 00:20:07.339 Awaish Kumar: So, more than that.
161 00:20:08.720 ⇒ 00:20:09.340 Bijil: What’s that?
162 00:20:09.630 ⇒ 00:20:11.299 Awaish Kumar: Can we talk about volume?
163 00:20:11.560 ⇒ 00:20:12.660 Awaish Kumar: of the data?
164 00:20:15.890 ⇒ 00:20:20.359 Bijil: I think the largest table they had at the time was about…
165 00:20:22.210 ⇒ 00:20:31.609 Bijil: probably a million or so rows, not a lot. That’s the largest table they had. That was, like, their jobs or projects table. That was the largest table they had, so it wasn’t that…
166 00:20:31.800 ⇒ 00:20:33.919 Bijil: A million rows? Yeah.
167 00:20:34.140 ⇒ 00:20:36.160 Awaish Kumar: What were those connections?
168 00:20:36.860 ⇒ 00:20:38.120 Bijil: the connections?
169 00:20:38.320 ⇒ 00:20:40.390 Awaish Kumar: The integrations, can you name some…
170 00:20:40.390 ⇒ 00:20:43.409 Bijil: One of them is Service Titan, so Service Titan is a CRM.
171 00:20:43.540 ⇒ 00:20:48.169 Bijil: That they use to manage trades in U.S, I guess, mostly.
172 00:20:48.670 ⇒ 00:21:01.020 Bijil: That was one. Is there any other… that was the only one. Service Certain is the main one they use, that’s the only one they had, and they… we did later on had QuickBooks and things like that, but SurveySaturn was the one that was in the migration.
173 00:21:02.510 ⇒ 00:21:05.510 Awaish Kumar: Okay, now, you mentioned there were some lighting…
174 00:21:05.700 ⇒ 00:21:12.209 Awaish Kumar: connectors that you implemented after your first migration. So, would you like to name a few?
175 00:21:13.290 ⇒ 00:21:16.620 Bijil: The connections they had, like, the client.
176 00:21:18.560 ⇒ 00:21:24.369 Awaish Kumar: You mentioned that Some, like, almost around 90 plus connectors.
177 00:21:24.900 ⇒ 00:21:32.490 Bijil: 90 plus endpoints. So it’s all one, one, one, one, one source, 90 plus endpoints from there.
178 00:21:33.680 ⇒ 00:21:35.319 Awaish Kumar: Okay, got it.
179 00:21:35.480 ⇒ 00:21:46.840 Awaish Kumar: So… Okay, great. So now that… I think we are… Talking about, projects.
180 00:21:48.670 ⇒ 00:21:51.489 Awaish Kumar: Can you rate, like, how would you rate yourself?
181 00:21:52.960 ⇒ 00:22:00.209 Awaish Kumar: The different… Tools or the programming languages you’ve used in your projects.
182 00:22:00.350 ⇒ 00:22:01.470 Awaish Kumar: Out of that.
183 00:22:02.330 ⇒ 00:22:05.669 Bijil: Out of 10. That’s a very broad question.
184 00:22:06.900 ⇒ 00:22:12.619 Bijil: In the sense that, do I, like, pick a language and then sort of write myself, or…
185 00:22:12.620 ⇒ 00:22:15.930 Awaish Kumar: Like, I mean, Python, SQL.
186 00:22:15.930 ⇒ 00:22:16.570 Bijil: Okay.
187 00:22:16.740 ⇒ 00:22:19.750 Awaish Kumar: cloud providers, like, I understand.
188 00:22:20.070 ⇒ 00:22:25.449 Awaish Kumar: Although they all are very broad, they all contain a very broad set of
189 00:22:25.800 ⇒ 00:22:35.190 Awaish Kumar: use cases and services. You don’t have to consider those. Like, I’m more talking about the ones which are useful in the data space.
190 00:22:35.510 ⇒ 00:22:41.120 Bijil: Sounds good. Maybe I’ll stop at,
191 00:22:41.410 ⇒ 00:22:51.799 Bijil: Maybe if I’m going the wrong direction, let me know. I’ll probably start with the GCP, probably an advanced expert user, probably rate plans of, I don’t know, 8, 9 out of 10.
192 00:22:52.240 ⇒ 00:23:01.639 Bijil: Sql, probably Advanced Expert as well, same 8, 9 out of 10. Python, probably the same as well. 8, 9 out of 10. Terraform, same.
193 00:23:02.480 ⇒ 00:23:14.989 Bijil: Databricks, same. 9 out of 10. Expo, advanced expert years are there. Snowflake, probably not so much. I’d probably say 5, 6 out of 10. I can probably pick up things, I probably have still to learn there, haven’t done a lot of projects on Snowflake.
194 00:23:15.260 ⇒ 00:23:29.229 Bijil: Mother Duck, 8, 9 out of 10. Done 3 projects so far on Mother Duck, so I’m pretty okay with that. And, AWS, 4, 5 out of 10, not a big one. Azure, 5, 6 out of 10.
195 00:23:29.360 ⇒ 00:23:31.339 Bijil: Not huge on Azure either.
196 00:23:31.460 ⇒ 00:23:46.769 Bijil: But I can work with it. I’ve done a couple of projects there. What else is there in terms of tools? DBT, by 9 out of 10, 10 out of 10. Like I said, I do coaching for them, so I have to know a lot of stuff.
197 00:23:47.120 ⇒ 00:23:57.250 Bijil: Well, what else is there? Tools-wise. I had Daxter probably 9 out of 10, 10 out of 10. I’ve done, like, 5 projects, on Daxter so far.
198 00:23:57.810 ⇒ 00:24:01.390 Bijil: What else? Airflow, probably…
199 00:24:01.770 ⇒ 00:24:05.830 Bijil: 6 or 7 out of 10. I’ve done… haven’t done a lot, maybe a couple of projects.
200 00:24:06.000 ⇒ 00:24:19.289 Bijil: BI tools, what else? BI tools? We have Looker, Looker Studio, 9 out of 10, pretty much advanced expert there. I’ve done Power BI, not a big fan, but can do, probably
201 00:24:19.770 ⇒ 00:24:24.230 Bijil: 5 or 6 out of 10. It’s a BI tool at the end of the day, I can’t really, you know…
202 00:24:24.230 ⇒ 00:24:24.889 Awaish Kumar: What’s it?
203 00:24:26.150 ⇒ 00:24:29.310 Awaish Kumar: What are you looking for in your next room?
204 00:24:30.300 ⇒ 00:24:48.999 Bijil: I’m gonna work with as many companies as possible, help as many people as possible, as many companies to become more data-driven, try and help them with their data infrastructure, bring some of the knowledge that I have from previous clients, previous engagements, also probably learn something new as well. Obviously, there’s always things to learn, yeah.
205 00:24:50.410 ⇒ 00:24:51.160 Awaish Kumar: Okay.
206 00:24:54.380 ⇒ 00:24:58.299 Awaish Kumar: Okay, and like, what time zone do you prefer to work with?
207 00:24:59.760 ⇒ 00:25:00.720 Bijil: What type?
208 00:25:01.040 ⇒ 00:25:10.909 Bijil: What time zone? What time zone? I would prefer to stay in, like, closer to my time zone, as much as possible, if that’s okay.
209 00:25:10.940 ⇒ 00:25:25.189 Bijil: So ideally, if clients are okay with async, or if there’s minor overlap, so a lot of the clients that I work with are in States and UK and Europe, generally, so we have, like, some overlap here and there. Europe would be…
210 00:25:25.190 ⇒ 00:25:31.050 Bijil: you know, a little bit later overlap. U.S. would be around this time, would be the overlap, usually.
211 00:25:31.060 ⇒ 00:25:39.379 Bijil: And, yeah, so ideally, I would like to stick to, like, normal hours on my own side, it’s easier.
212 00:25:39.830 ⇒ 00:25:47.060 Bijil: But yeah, I’m sort of… I’m flexible, like, every now and then I could do, like, an odd, like, a meeting at the weird times.
213 00:25:47.060 ⇒ 00:25:54.890 Awaish Kumar: For example, You already mentioned you have a client in the States, you have a… Uk… So…
214 00:25:55.540 ⇒ 00:25:59.029 Awaish Kumar: How, like, how you are distributing your time there?
215 00:25:59.440 ⇒ 00:26:01.790 Awaish Kumar: Like, do you have a team of people, or…
216 00:26:03.560 ⇒ 00:26:21.440 Bijil: I did have a team of people at some point, not anymore, it’s just me now. So a lot of the clients now I have is more on managed services model, so I don’t really actively build for them anymore. I’m more that I have something built for them already, I’m just sort of there in case something goes wrong, sort of like a managed model.
217 00:26:21.450 ⇒ 00:26:28.239 Bijil: It’s different, has a different sort of engagement, so it doesn’t really affect me that much in terms of what goes on with them.
218 00:26:28.750 ⇒ 00:26:32.539 Awaish Kumar: Okay, and yeah, we would also prefer
219 00:26:32.650 ⇒ 00:26:39.790 Awaish Kumar: Someone which can at least, overlap, like, 4 hours.
220 00:26:41.410 ⇒ 00:26:43.199 Bijil: What time zone, is it?
221 00:26:43.620 ⇒ 00:26:50.860 Awaish Kumar: So, we have… most of our clients are in Eastern, Is it, Central Time?
222 00:26:51.210 ⇒ 00:26:58.839 Awaish Kumar: So… Overlap 4 hours with either of them would be… Like, okay, but yeah.
223 00:26:59.770 ⇒ 00:27:02.479 Bijil: Yeah, yeah, I think 4 hours East Eastern.
224 00:27:02.770 ⇒ 00:27:05.480 Awaish Kumar: Might be a little bit hard for me, because…
225 00:27:05.510 ⇒ 00:27:23.559 Bijil: 5AM would be the earliest I have to wake… earliest I have to wake up. I’m usually up by about… I’m usually starting my work around 6 AM in the morning. That’s still pretty early for ET, until about 9, so that’s 3 hours. That’s probably the most I can do, 3 hours, because anything more is just…
226 00:27:23.560 ⇒ 00:27:26.929 Bijil: I’ll just… I’m just burning myself out. I don’t want to do it.
227 00:27:27.860 ⇒ 00:27:30.370 Awaish Kumar: Okay, yeah, that’s okay, that’s good to hear.
228 00:27:30.860 ⇒ 00:27:34.719 Awaish Kumar: Yeah, we do ask these questions, like, they come up
229 00:27:35.080 ⇒ 00:27:37.799 Awaish Kumar: That’s okay. These are kind of expectations.
230 00:27:39.690 ⇒ 00:27:44.120 Awaish Kumar: Okay, I think I’m good with my question, like, yeah, if you would like to ask anything.
231 00:27:44.700 ⇒ 00:27:55.279 Bijil: Yeah, I mean, maybe, just… I mean, you already briefly touched on in the last couple of questions there, and the style of working. So how does the style of working work? So, you have so many people who are across the world.
232 00:27:55.380 ⇒ 00:28:01.239 Bijil: Is it many people going into one project, or one person going into one project, or how does that sort of structure look like?
233 00:28:01.240 ⇒ 00:28:08.430 Awaish Kumar: Like, when you… we always have few people. We work with a model where like…
234 00:28:08.670 ⇒ 00:28:12.489 Awaish Kumar: There are always, like, 3 people on one project.
235 00:28:15.090 ⇒ 00:28:24.509 Awaish Kumar: like, that’s the, like, ideal scenario, and that’s what Redford’s considering. We do have some exceptions where…
236 00:28:24.710 ⇒ 00:28:31.340 Awaish Kumar: We are supporting them for something, and that we have to, like.
237 00:28:32.540 ⇒ 00:28:36.440 Awaish Kumar: Just one person can do that, but normally we want to have
238 00:28:36.930 ⇒ 00:28:45.339 Awaish Kumar: clients, where at least we can… we can have 3 people, to work on that, to deliver, like, the…
239 00:28:45.490 ⇒ 00:28:52.610 Awaish Kumar: deliver the work in a more, like, robust way. And apart from that.
240 00:28:53.770 ⇒ 00:28:57.220 Awaish Kumar: Also, like, we do want some,
241 00:28:58.240 ⇒ 00:29:12.600 Awaish Kumar: So, like, if somebody leaves, if somebody is sick, like, how we are going to cover our client work. So we do have that, like, the process for handling all of those situations.
242 00:29:13.650 ⇒ 00:29:17.290 Bijil: Okay, sounds good to me. One last question, I got a couple minutes.
243 00:29:17.430 ⇒ 00:29:21.919 Bijil: Just, could you, like, maybe talk through, talk to me about one project that you’re proud of?
244 00:29:22.290 ⇒ 00:29:33.009 Bijil: That you can kind of think of. Like, doesn’t have to go into details, because obviously client… client details and stuff like that, but just high level, what sort of projects that you guys take on, just to get an idea.
245 00:29:34.840 ⇒ 00:29:40.439 Awaish Kumar: Yeah, like, in my current job, like, there are a lot of projects which I’m working on.
246 00:29:40.650 ⇒ 00:29:42.080 Awaish Kumar: Which we are proud of.
247 00:29:42.240 ⇒ 00:29:49.769 Awaish Kumar: I think, The recent one which I’m working on is, like, we have some…
248 00:29:49.920 ⇒ 00:29:55.609 Awaish Kumar: We are building a data foundation for a CPG company, which is, like,
249 00:29:56.250 ⇒ 00:30:00.000 Awaish Kumar: With, 100 plus billion of revenue.
250 00:30:00.240 ⇒ 00:30:07.589 Awaish Kumar: They have… they have different channels to operate, like retail, e-commerce, wholesale.
251 00:30:09.280 ⇒ 00:30:16.540 Awaish Kumar: And they are making money, but they don’t know… they don’t have a way to report on it right now, don’t have the data.
252 00:30:16.690 ⇒ 00:30:19.499 Awaish Kumar: From e-comm, they might have something, but…
253 00:30:19.920 ⇒ 00:30:23.110 Awaish Kumar: Nothing for wholesale, nothing for retail.
254 00:30:23.320 ⇒ 00:30:26.509 Awaish Kumar: So we are building, you know, foundations for them.
255 00:30:26.690 ⇒ 00:30:31.150 Awaish Kumar: To break in all the data, which is spread across various sources.
256 00:30:31.200 ⇒ 00:30:47.690 Awaish Kumar: for retail, maybe something that’s… is it Snowflake, something coming in S3, FTP, emails, right? We have my Shopify, like, coming from Shopify, Amazon, where to go, stored.
257 00:30:48.020 ⇒ 00:30:50.669 Awaish Kumar: Gorgeous, like, customer service data.
258 00:30:50.850 ⇒ 00:30:54.740 Awaish Kumar: Customer support data, and then, we have,
259 00:30:55.090 ⇒ 00:31:00.850 Awaish Kumar: Salesforce Marketing Cloud, and all the marketing connectors, so…
260 00:31:01.120 ⇒ 00:31:04.200 Awaish Kumar: Like, a whole bunch of data from various sources.
261 00:31:04.530 ⇒ 00:31:11.830 Awaish Kumar: Is… we are adjusting that. So, obviously, we are using tools, ingestion tools, for most of it.
262 00:31:12.280 ⇒ 00:31:20.330 Awaish Kumar: But we also have Dexter, so if some… something like, for some… Like, what’d you say, small?
263 00:31:21.470 ⇒ 00:31:24.709 Awaish Kumar: API advice, we don’t want our…
264 00:31:25.000 ⇒ 00:31:30.069 Awaish Kumar: Partners to build the collectors. We actually go and write scripts ourselves.
265 00:31:30.170 ⇒ 00:31:34.750 Awaish Kumar: That is orchestrated through text, that we have.
266 00:31:35.370 ⇒ 00:31:41.179 Awaish Kumar: Slow Flake, DVT, And, apart from that, I think…
267 00:31:42.330 ⇒ 00:31:45.919 Awaish Kumar: We don’t get a vac… like, most of our jobs
268 00:31:46.330 ⇒ 00:31:52.810 Awaish Kumar: are on the dbt, like, the GitHub Actions, because most of it is… lives in the dbt.
269 00:31:53.050 ⇒ 00:31:56.429 Awaish Kumar: So, normally, what, like…
270 00:31:57.780 ⇒ 00:32:09.929 Awaish Kumar: We have Dexter for something which orchestrates, but that’s very kind of a little part of it. Most of our business logic, the bottling, all of it lives in dbt.
271 00:32:10.070 ⇒ 00:32:13.430 Awaish Kumar: We don’t want to move We don’t move to…
272 00:32:13.710 ⇒ 00:32:18.200 Awaish Kumar: the Python or any other programming language, until it’s really…
273 00:32:18.440 ⇒ 00:32:32.869 Awaish Kumar: necessary. So, most of the logic lives in the dbt, and that’s why, we can easily run it using dbt code and Git of actions. Some clients prefer cloud, but we probably see…
274 00:32:32.980 ⇒ 00:32:36.299 Awaish Kumar: with dbt cold, Then we have…
275 00:32:36.670 ⇒ 00:32:45.840 Awaish Kumar: dashboarding tools, we have a team of analysts, which can basically help us with Excel reporting, OBD, Diablo, Power BI, whatever
276 00:32:46.090 ⇒ 00:32:55.600 Awaish Kumar: And yeah, that’s basically what we do here. Like, there’s a lot of communication, discovery, training of metrics, standardizing that.
277 00:32:56.090 ⇒ 00:33:05.090 Awaish Kumar: So we, like, do all… kind of, like, business VI kind of… Kind of work.
278 00:33:05.320 ⇒ 00:33:11.840 Awaish Kumar: Yeah, that’s… I think that’s… that’s what… Our projects look like, totally.
279 00:33:12.180 ⇒ 00:33:15.109 Awaish Kumar: As I mentioned, like, we…
280 00:33:15.650 ⇒ 00:33:30.110 Awaish Kumar: to have some clients where they want, like, okay, I need some data, like, time of architecture, I need some real-time injections, but yeah, it’s… it’s very rare. Not everybody needs real-time data.
281 00:33:32.210 ⇒ 00:33:42.539 Bijil: Perfect. Awesome, sounds good to me. Everything looks good, man. I know we are out on time now, just, curious, just, next steps, what that looks like, and, timeline for all of that.
282 00:33:42.850 ⇒ 00:33:45.669 Awaish Kumar: Yeah, I’m going to share my feedback with the…
283 00:33:46.210 ⇒ 00:33:49.039 Awaish Kumar: operations team, and based on that.
284 00:33:49.170 ⇒ 00:33:53.599 Awaish Kumar: Rico from Oman Precious Team will follow up on the next steps.
285 00:33:54.760 ⇒ 00:33:56.100 Bijil: Okay, sounds good.
286 00:33:56.220 ⇒ 00:33:57.360 Awaish Kumar: That’s awesome.
287 00:33:57.830 ⇒ 00:33:58.580 Awaish Kumar: Thank you.
288 00:33:59.130 ⇒ 00:34:00.850 Bijil: Appreciate it. Have a nice day. Bye.