Meeting Title: Brainforge x Urban Stem Deprecation Planning Date: 2025-06-23 Meeting participants: Caio Velasco, Uttam Kumaran
WEBVTT
1 00:00:18.430 ⇒ 00:00:19.140 Uttam Kumaran: Hey!
2 00:00:19.910 ⇒ 00:00:20.790 Caio Velasco: Hello!
3 00:00:22.050 ⇒ 00:00:23.360 Uttam Kumaran: Hey? How’s everything?
4 00:00:23.760 ⇒ 00:00:24.900 Caio Velasco: Good! How are you?
5 00:00:25.390 ⇒ 00:00:26.240 Uttam Kumaran: Good.
6 00:00:29.620 ⇒ 00:00:31.350 Caio Velasco: How’s your Monday started.
7 00:00:32.390 ⇒ 00:00:36.982 Uttam Kumaran: Monday starting. Good. I’m just gonna take the dog out for a quick walk.
8 00:00:37.620 ⇒ 00:00:38.070 Caio Velasco: Right.
9 00:00:41.020 ⇒ 00:00:45.510 Uttam Kumaran: And then, yeah, this week’s gonna be really good. We have, I think
10 00:00:46.770 ⇒ 00:00:53.019 Uttam Kumaran: I mean these things you never know. But I think we have 2 more clients that are gonna sign, at least in the next
11 00:00:53.160 ⇒ 00:00:54.569 Uttam Kumaran: 3 days.
12 00:00:54.800 ⇒ 00:01:03.115 Uttam Kumaran: I texted. I texted both of them yesterday, and they’re both look pretty, positive, so like fingers crossed.
13 00:01:04.455 ⇒ 00:01:05.069 Caio Velasco: Quick!
14 00:01:05.618 ⇒ 00:01:07.830 Uttam Kumaran: So that would be good, so
15 00:01:08.390 ⇒ 00:01:15.320 Uttam Kumaran: like it’ll be a really strong month. If if so, we will have those 2 starting, and then
16 00:01:16.216 ⇒ 00:01:17.690 Uttam Kumaran: we’ll have.
17 00:01:17.860 ⇒ 00:01:24.610 Uttam Kumaran: We have an additional 2 that are moving from sort of like an audit to a larger engagement.
18 00:01:24.750 ⇒ 00:01:33.190 Uttam Kumaran: so that should hopefully be net 4 new clients this month, which will be really, really
19 00:01:33.750 ⇒ 00:01:41.020 Uttam Kumaran: great. I’m also going to. I’m going to New York to go visit Robert on on Wednesday. So.
20 00:01:42.070 ⇒ 00:01:42.610 Caio Velasco: Hold on!
21 00:01:42.610 ⇒ 00:01:44.340 Uttam Kumaran: Yeah, there.
22 00:01:44.660 ⇒ 00:01:48.109 Caio Velasco: Yeah, I feel nice. Where do you go?
23 00:01:48.620 ⇒ 00:01:52.639 Uttam Kumaran: I’m going on Wednesday, Wednesday, Wednesday morning.
24 00:01:53.210 ⇒ 00:01:55.780 Caio Velasco: Maybe you can try also to meet with Alexander.
25 00:01:56.470 ⇒ 00:02:03.900 Uttam Kumaran: I’m gonna yeah, we’re gonna meet. I’m gonna meet him, too, on I think I think we’re gonna meet on Thursday or Wednesday evening. Yeah.
26 00:02:04.250 ⇒ 00:02:04.950 Caio Velasco: Hmm!
27 00:02:05.340 ⇒ 00:02:06.650 Caio Velasco: That’s nice, really nice.
28 00:02:06.650 ⇒ 00:02:12.099 Uttam Kumaran: Yeah, yeah, he’s he’s he’s been helping a lot. I I think you’ll be. I mean.
29 00:02:12.460 ⇒ 00:02:15.540 Uttam Kumaran: I mean, you know how organized he is, I think you’ll
30 00:02:15.967 ⇒ 00:02:24.210 Uttam Kumaran: we’re doing a lot of similar stuff, I’m sure, to what you’re used to for the last org. But it’s he’s helping Amber a lot. Actually.
31 00:02:24.580 ⇒ 00:02:31.395 Caio Velasco: That’s nice. Yeah, no. I remember that he was a great addition. So it was crazy that they let him go.
32 00:02:32.084 ⇒ 00:02:40.110 Uttam Kumaran: Yeah, I don’t know who would let him. I mean he he clearly bounced up like Bridgewater is like a great
33 00:02:40.110 ⇒ 00:02:40.943 Uttam Kumaran: thank you.
34 00:02:42.920 ⇒ 00:02:43.620 Caio Velasco: Right.
35 00:02:46.300 ⇒ 00:02:47.520 Uttam Kumaran: Awesome.
36 00:02:47.520 ⇒ 00:02:48.280 Caio Velasco: Yeah.
37 00:02:48.280 ⇒ 00:02:49.940 Uttam Kumaran: How about you? How’s your weekend.
38 00:02:51.170 ⇒ 00:02:56.740 Caio Velasco: Well, my weekend was good. Actually, my, my family came visit me. So they are staying.
39 00:02:57.870 ⇒ 00:03:00.510 Caio Velasco: Yeah, they came from Portugal.
40 00:03:01.040 ⇒ 00:03:06.580 Caio Velasco: They, my mom has been here one time, but my grandparents, they have so 1st time for them in
41 00:03:06.770 ⇒ 00:03:09.489 Caio Velasco: in Barcelona. They’ve been to Madrid, but not Barcelona.
42 00:03:10.200 ⇒ 00:03:13.550 Uttam Kumaran: Oh, nice! Oh, I didn’t know. Your grandparents also came. That’s great!
43 00:03:13.830 ⇒ 00:03:16.100 Caio Velasco: Yes, they are also in Portugal.
44 00:03:16.538 ⇒ 00:03:23.179 Caio Velasco: So yeah, it was like super nice to have them around, even though now I have, you know another job as well.
45 00:03:24.630 ⇒ 00:03:25.570 Uttam Kumaran: Nice.
46 00:03:25.960 ⇒ 00:03:35.029 Caio Velasco: But yeah, it’s been super nice. The weekend was just about going out and like touristing a bit, showing them like a new, a few new places.
47 00:03:35.380 ⇒ 00:03:41.319 Caio Velasco: So yeah, we’re super good. Then it’s now I can say that summer is, it’s here because it’s super hot.
48 00:03:41.730 ⇒ 00:03:43.230 Uttam Kumaran: Okay. Great. Great.
49 00:03:43.430 ⇒ 00:03:44.020 Caio Velasco: Yep.
50 00:03:45.810 ⇒ 00:03:46.520 Uttam Kumaran: Nice.
51 00:03:48.990 ⇒ 00:03:57.339 Uttam Kumaran: Okay, cool. Well, tell me where we want to get started. I mean, I’m happy to give you any background. I think, for me today. I just want to give you as much knowledge about
52 00:03:57.710 ⇒ 00:04:02.180 Uttam Kumaran: sort of redshift related migrations. I’ve done a couple of these so
53 00:04:02.727 ⇒ 00:04:05.559 Uttam Kumaran: I can kind of at least explain a lot of the
54 00:04:05.770 ⇒ 00:04:11.610 Uttam Kumaran: the risks and things, and like sort of how we should probably try to try to handle things.
55 00:04:12.170 ⇒ 00:04:12.903 Caio Velasco: Cool, cool.
56 00:04:13.935 ⇒ 00:04:18.924 Caio Velasco: So yeah, I, I kind of had separated this into 2 parts.
57 00:04:19.560 ⇒ 00:04:25.099 Caio Velasco: we can just basically talk about deprecation. But then, if we have time, we can also talk a little bit about modeling.
58 00:04:25.420 ⇒ 00:04:31.119 Caio Velasco: although I also had a meeting today with Emilia, which was quite clear, but still.
59 00:04:31.740 ⇒ 00:04:32.840 Uttam Kumaran: Okay.
60 00:04:33.030 ⇒ 00:04:49.580 Caio Velasco: So the deprecation. I basically now that I’ve done some work on it, I see that I basically separated the work into 2 parts like the Dbt mapping with redshift and the Dbt mapping with look.
61 00:04:50.823 ⇒ 00:04:51.649 Caio Velasco: The first.st
62 00:04:51.650 ⇒ 00:04:52.380 Uttam Kumaran: Yes.
63 00:04:52.550 ⇒ 00:04:54.560 Caio Velasco: The Dbt with redshift.
64 00:04:54.750 ⇒ 00:04:59.053 Caio Velasco: Well, well, let me start with the one I started first.st So dbt, with looker,
65 00:05:00.200 ⇒ 00:05:06.330 Caio Velasco: basically what I what I understood from from Lucer that we can always access system activity
66 00:05:06.480 ⇒ 00:05:09.139 Caio Velasco: and get a lot of information from there.
67 00:05:09.320 ⇒ 00:05:09.930 Caio Velasco: And
68 00:05:10.280 ⇒ 00:05:18.929 Caio Velasco: at the beginning I was a bit, I mean lost with Luca, because I never used. But as you mentioned, it’s it’s basic you have explored and fuse
69 00:05:20.240 ⇒ 00:05:21.140 Caio Velasco: and
70 00:05:21.250 ⇒ 00:05:33.679 Caio Velasco: in the views, if you look at the looker, folder, looker, repo, and github. You can see that those tags, the SQL. Table name, and then different
71 00:05:33.860 ⇒ 00:05:37.780 Caio Velasco: derived table. That’s where either
72 00:05:37.980 ⇒ 00:05:49.370 Caio Velasco: the full path to a redshift table is defined, or a sequel, and then you get from the from clauses or from the join clauses, whatever you want.
73 00:05:49.820 ⇒ 00:05:52.650 Caio Velasco: So that was okay. But then I noticed that
74 00:05:52.890 ⇒ 00:05:58.830 Caio Velasco: my script was not able to get the explorers which come from.
75 00:05:59.560 ⇒ 00:06:03.210 Caio Velasco: Let’s say, dot model dot looker.
76 00:06:03.820 ⇒ 00:06:04.360 Uttam Kumaran: Yes.
77 00:06:04.360 ⇒ 00:06:20.090 Caio Velasco: Or not from a dot view dot booker. And that’s like, okay, I have to update this. So I updated. And then in the explorers. They are basically pointing to views, and the views will point back to to the path, to to Ratshift tables. So then, after that.
78 00:06:20.200 ⇒ 00:06:24.339 Caio Velasco: I was able to do this link between
79 00:06:24.840 ⇒ 00:06:32.710 Caio Velasco: dashboards, their names and ids. And also all this associated tables in redship
80 00:06:32.890 ⇒ 00:06:36.210 Caio Velasco: was a lot of work, but was really nice to learn.
81 00:06:36.580 ⇒ 00:06:40.539 Caio Velasco: and I did with the scripts with Chat Gpt helping me out.
82 00:06:41.480 ⇒ 00:06:44.929 Uttam Kumaran: Yeah. And how did the script work? It was a script like in your browser or.
83 00:06:45.140 ⇒ 00:06:54.029 Caio Velasco: So what I know actually, what I do is I I created a repo in our Brainford account. If I’m not wrong
84 00:06:54.190 ⇒ 00:06:59.129 Caio Velasco: in in Github. And then I connect via I always use Vs code.
85 00:06:59.300 ⇒ 00:07:09.479 Caio Velasco: So I connect via Vs code. And then over there, I basically decided to do with python scripts. So I have, like some
86 00:07:09.660 ⇒ 00:07:13.360 Caio Velasco: few libraries that I install like not like maybe 3 or 4,
87 00:07:13.540 ⇒ 00:07:21.350 Caio Velasco: and with those, for example, Pandas would be one of them with those I can write some script to
88 00:07:21.490 ⇒ 00:07:26.500 Caio Velasco: basically, let’s say, from Lucer, I have.
89 00:07:26.640 ⇒ 00:07:30.720 Caio Velasco: I can just like export Csvs.
90 00:07:30.970 ⇒ 00:07:32.889 Caio Velasco: And then I okay and then run it over.
91 00:07:32.890 ⇒ 00:07:33.340 Uttam Kumaran: That.
92 00:07:33.340 ⇒ 00:07:38.060 Caio Velasco: Exactly. And then I ran over there. So that was would be like the input to the scripts.
93 00:07:38.190 ⇒ 00:07:38.730 Caio Velasco: Okay.
94 00:07:39.610 ⇒ 00:07:46.829 Caio Velasco: yeah. And then the the output would be other Csv. That I would just at the end copy and paste to our spreadsheet.
95 00:07:47.740 ⇒ 00:07:48.289 Uttam Kumaran: I see.
96 00:07:48.290 ⇒ 00:07:48.810 Uttam Kumaran: Okay.
97 00:07:48.810 ⇒ 00:07:52.969 Caio Velasco: Yeah. So it was like a bit more manual, but still automated enough.
98 00:07:53.670 ⇒ 00:07:54.000 Uttam Kumaran: Okay.
99 00:07:55.200 ⇒ 00:07:55.830 Caio Velasco: And.
100 00:07:55.830 ⇒ 00:08:09.580 Uttam Kumaran: Yeah, I mean, I think you basically nailed it. I mean, look, looker is just like A, it’s just a adds flexibility on it’s basically a sequel generator. Right? So all it’s doing is it’s creating its own representation of the tables.
101 00:08:09.850 ⇒ 00:08:13.460 Uttam Kumaran: It’s you could layer on additional expressions.
102 00:08:13.956 ⇒ 00:08:29.089 Uttam Kumaran: And then the explorers are just joins, and then, as soon as you run a query in an explore as soon as you run a query in an explore it. It creates a sequel. It creates a SQL. Query, and then it issues it, and it pulls the results back.
103 00:08:29.200 ⇒ 00:08:40.199 Uttam Kumaran: So it’s just a semantic layer on top of the tables. You’re exactly right. The the problem with looker is that because you can join something
104 00:08:40.409 ⇒ 00:08:47.630 Uttam Kumaran: like 10 times through different explorers if you don’t have any governance right? If you don’t have a process on like
105 00:08:47.770 ⇒ 00:09:12.699 Uttam Kumaran: one team creating explores, there’s guidelines on when to create an explorer versus joining a view doesn’t explore already exist. That satisfies your needs. You get like a hundred explores right? And then you get it. And then when you create a dashboard dashboards, even crazier, because dashboard can hold things from multiple explores right? And so
106 00:09:12.850 ⇒ 00:09:16.600 Uttam Kumaran: it’s dashboard is probably the tougher thing to
107 00:09:17.430 ⇒ 00:09:22.120 Uttam Kumaran: to deprecate because it could have things from explorers that are
108 00:09:22.380 ⇒ 00:09:26.770 Uttam Kumaran: that you want to kill off and explores that need to still be live.
109 00:09:28.220 ⇒ 00:09:38.520 Uttam Kumaran: But yeah, this is just like what happens after using looker, for I mean, their situation is really bad. But like this is like what happens when you have no
110 00:09:38.710 ⇒ 00:09:43.890 Uttam Kumaran: sort of central governance on, when to create stuff. You know, you get this sort of asset sprawl.
111 00:09:44.740 ⇒ 00:09:46.549 Caio Velasco: Yeah, yeah, no, that’s right. That’s right.
112 00:09:46.947 ⇒ 00:09:48.729 Caio Velasco: Okay, no. I got it. Then
113 00:09:49.840 ⇒ 00:09:56.650 Caio Velasco: let me see if there’s any other thing. Oh, okay. So then given that I had this mapping between the red stripe tables and
114 00:09:57.350 ⇒ 00:09:59.090 Caio Velasco: the dashboards.
115 00:10:01.720 ⇒ 00:10:09.429 Caio Velasco: I well, I went back to the spreadsheet, and I remember that we have a sheet a tab.
116 00:10:10.740 ⇒ 00:10:18.789 Caio Velasco: For that in the that we’re listing everything in Dbt, so we have Dbt models Dbt seed
117 00:10:19.160 ⇒ 00:10:25.100 Caio Velasco: and the Bt. Sources. So then, what I did was I just selected the models, and then
118 00:10:25.600 ⇒ 00:10:31.839 Caio Velasco: in that subset? That was where I was doing the match between dashboard and the tables.
119 00:10:32.040 ⇒ 00:10:35.950 Caio Velasco: because I also learned that Dbt is
120 00:10:36.660 ⇒ 00:10:51.960 Caio Velasco: materializing things, objects as analytics dot analytics. So then I was able to, you know from the model name I get the full path, and then from that full path I go in my script, make the join, and then I find the dashboards.
121 00:10:53.105 ⇒ 00:10:53.540 Uttam Kumaran: Right.
122 00:10:53.780 ⇒ 00:11:01.910 Caio Velasco: So yeah, so it seems to be okay. I, I was expecting to see more dashboards for each
123 00:11:02.280 ⇒ 00:11:11.709 Caio Velasco: model. But there’s a lot of models that have no dashboards at all pointing to it. I was, maybe I’m wrong, but I mean I run it a couple of times.
124 00:11:13.699 ⇒ 00:11:26.600 Caio Velasco: But yeah, it seems to be that like, yeah, a lot. Actually, yeah. And then you see that, for example components. Except, it’s yeah. They have like 100 dashboards pointing to that. So then it’s like
125 00:11:26.940 ⇒ 00:11:27.720 Uttam Kumaran: Yes.
126 00:11:27.720 ⇒ 00:11:28.780 Caio Velasco: Yes. Yeah.
127 00:11:30.201 ⇒ 00:11:33.270 Uttam Kumaran: So that, okay, so that’s basically done. And then
128 00:11:33.270 ⇒ 00:11:38.129 Uttam Kumaran: the yeah, the other thing is that like, of course, dbt, is
129 00:11:38.964 ⇒ 00:11:53.870 Uttam Kumaran: purely the ex like execution layer. Right? So ideally, what we’re doing is like one if we kill it from dB, if like, you have to sort of pick which direction you’re coming from, based on the use case. Like, if you’re
130 00:11:53.940 ⇒ 00:12:16.179 Uttam Kumaran: if you’re if you have like, hey, there’s these, there’s like a hundred dashboards here, and they’re none of them are getting any usage. Let’s just delete them. You don’t really have to do anything in Dbt, right? So there’s like content. There’s removing content just because it’s unused. Right? That’s like one category. There’s also like, Hey, these are. These are models that are no longer
131 00:12:16.280 ⇒ 00:12:19.300 Uttam Kumaran: either exist or are no longer supported.
132 00:12:19.530 ⇒ 00:12:30.160 Uttam Kumaran: So then you have to go. Stop the Dbt. Job, delete, and then go delete all the necessary looker, and then finally delete the redshift table
133 00:12:30.750 ⇒ 00:12:32.480 Uttam Kumaran: right like. That’s the steps.
134 00:12:32.820 ⇒ 00:12:34.489 Uttam Kumaran: Those are the kind of the steps for that.
135 00:12:34.851 ⇒ 00:12:39.479 Uttam Kumaran: But we could walk through some. We could totally walk through some scenarios, and I think there’s just gonna be
136 00:12:39.660 ⇒ 00:12:42.040 Uttam Kumaran: a couple of different angles to come in.
137 00:12:42.494 ⇒ 00:12:44.439 Uttam Kumaran: Just to make sure that, like
138 00:12:45.000 ⇒ 00:12:49.020 Uttam Kumaran: you try to, it will happen. But you just try to prevent like.
139 00:12:49.170 ⇒ 00:12:52.560 Uttam Kumaran: hey, I was using this like, where is it? Type thing, you know.
140 00:12:53.350 ⇒ 00:12:59.588 Caio Velasco: Yeah. Okay, okay, so let though, let me restart from from the beginning, from what Emily has been doing.
141 00:13:00.686 ⇒ 00:13:18.009 Caio Velasco: So then we have another sheet, which is just a dashboard, is a dashboard level sheet, listing all dashboards, and then in this last 2 weeks she was working on it. And now as I posted last week, we have a final list of dashboard to be deprecated.
142 00:13:18.851 ⇒ 00:13:21.660 Caio Velasco: So let’s say we already have this.
143 00:13:22.290 ⇒ 00:13:39.489 Caio Velasco: can we? I mean, of course, we can trust because she did it. But what would be the the measure here for us to say, like, Yeah, okay, we are going to deprecate this 700 dashboards. Is that what we are doing first, st or we are still going through the connections with the Dbt stuff in the redshift tables.
144 00:13:39.710 ⇒ 00:13:42.880 Uttam Kumaran: So I would, I mean, so there’s
145 00:13:43.120 ⇒ 00:13:49.019 Uttam Kumaran: there’s sort of different opinions on, like how to do these types of deprecations for me, just because I have a lot
146 00:13:49.820 ⇒ 00:13:51.640 Uttam Kumaran: our tissue from
147 00:13:51.780 ⇒ 00:13:57.110 Uttam Kumaran: messing this up like a lot of times. I would start with the easiest thing to reverse
148 00:13:57.470 ⇒ 00:14:02.670 Uttam Kumaran: alright. So the easiest thing to reverse is taking those dashboards
149 00:14:02.910 ⇒ 00:14:05.539 Uttam Kumaran: and just unsharing them, meaning like
150 00:14:05.640 ⇒ 00:14:13.829 Uttam Kumaran: they’re no longer. They’re just no longer available, right? So just making sure that they
151 00:14:14.990 ⇒ 00:14:18.879 Uttam Kumaran: like they can’t be. They just can’t be used anymore. Right? That’s an easy.
152 00:14:19.050 ⇒ 00:14:23.634 Uttam Kumaran: That’s a just a completely easy thing to do, right? Just make sure that they’re
153 00:14:24.540 ⇒ 00:14:29.780 Uttam Kumaran: They’re moved to us to an archive space or something like that, right like
154 00:14:29.940 ⇒ 00:14:36.690 Uttam Kumaran: that’s probably what I would suggest. And then unsharing them, because that’s easy to revert. If if
155 00:14:37.680 ⇒ 00:14:39.879 Uttam Kumaran: someone’s like. Oh, I was using that right
156 00:14:40.210 ⇒ 00:14:43.850 Uttam Kumaran: like that’s what I would. That’s what I would suggest.
157 00:14:44.600 ⇒ 00:14:52.739 Caio Velasco: Okay, okay, no, that’s great. That makes sense. And well, let’s say that we have that approach. And then another layer would be the connection with
158 00:14:53.100 ⇒ 00:14:54.719 Caio Velasco: the Dbt motives.
159 00:14:56.580 ⇒ 00:15:00.320 Caio Velasco: Let’s say that we deprecate those, and no one
160 00:15:00.700 ⇒ 00:15:02.989 Caio Velasco: says anything. So seems to be okay.
161 00:15:03.730 ⇒ 00:15:08.880 Uttam Kumaran: Yeah, I think I would. Also, I would also consider the next change to be
162 00:15:09.100 ⇒ 00:15:15.380 Uttam Kumaran: getting rid of. Are there any explores that are like 100% unused in dashboard
163 00:15:19.030 ⇒ 00:15:25.200 Uttam Kumaran: cause. That’s that’s that’s what I would suggest is the next layer, because ultimately what gets put in the
164 00:15:25.340 ⇒ 00:15:36.430 Uttam Kumaran: the explorers are what get used to create the dashboard components. So if there are explores that have note, so the way to calculate is like, look at the explorers with
165 00:15:36.870 ⇒ 00:15:41.599 Uttam Kumaran: with no dashboards associated with them.
166 00:15:42.160 ⇒ 00:15:44.460 Uttam Kumaran: If so, we should get rid of that next.
167 00:15:46.750 ⇒ 00:15:54.469 Caio Velasco: Okay. Okay, okay, this part I haven’t. I don’t think we have a connection with from exports to
168 00:15:55.630 ⇒ 00:15:56.580 Caio Velasco: do lead.
169 00:15:58.160 ⇒ 00:16:01.480 Uttam Kumaran: So the dashboards themselves will have what’s called looks.
170 00:16:01.860 ⇒ 00:16:07.279 Uttam Kumaran: Looks is the tiles, and then tiles will pull from an explorer
171 00:16:08.020 ⇒ 00:16:13.500 Uttam Kumaran: like that. That should be the sort of the join linking between. Then.
172 00:16:13.750 ⇒ 00:16:20.079 Caio Velasco: So I do have a a list of dash a Csv, that is, dashboards, and they’re explores and
173 00:16:20.370 ⇒ 00:16:21.010 Caio Velasco: and.
174 00:16:21.010 ⇒ 00:16:21.540 Uttam Kumaran: Okay.
175 00:16:21.730 ⇒ 00:16:22.900 Caio Velasco: And leave. Yeah.
176 00:16:23.240 ⇒ 00:16:31.100 Uttam Kumaran: So I would basically look, I, yeah, you could just cross reference like, look at all the explorers there. Compare it to the list of all the known explorers
177 00:16:31.830 ⇒ 00:16:36.219 Uttam Kumaran: like, just select all explorers. You’ll see if there’s any that.
178 00:16:36.630 ⇒ 00:16:44.639 Uttam Kumaran: because if they’re not used then we can. We can also consider removing those, because that way we’ll.
179 00:16:44.640 ⇒ 00:16:48.810 Caio Velasco: Not used. What do you mean like, with the user.
180 00:16:49.570 ⇒ 00:16:58.690 Uttam Kumaran: They’ve never been queried via Dashboard, or they never been queried like recently, just from the
181 00:16:59.060 ⇒ 00:17:02.799 Uttam Kumaran: explore page, you know, in Looker, you can go to the top and click, explore.
182 00:17:03.870 ⇒ 00:17:13.120 Uttam Kumaran: And you can basically like, just select an explorer we just want to find, because that way, what you’re able to do is, let’s say there’s 5 tables.
183 00:17:13.410 ⇒ 00:17:17.510 Uttam Kumaran: Join into one explorer, and you can crush the explore.
184 00:17:17.750 ⇒ 00:17:21.900 Uttam Kumaran: Then it’s easy to go deprecate right? Because, you know, it’s not going to be used.
185 00:17:22.079 ⇒ 00:17:25.810 Uttam Kumaran: So you’re almost able to take take out more
186 00:17:26.319 ⇒ 00:17:29.599 Uttam Kumaran: stuff in one go is what would be my my suggestion.
187 00:17:30.980 ⇒ 00:17:31.490 Caio Velasco: Okay.
188 00:17:31.490 ⇒ 00:17:36.080 Caio Velasco: Okay. Okay? No. I think I see, it’s because we were only having
189 00:17:36.530 ⇒ 00:17:48.269 Caio Velasco: the the other sheets we have. They look into dashboard usage. So we have dashboard that are used, their dashboard that are way less used. But still that doesn’t mean that an explorer inside
190 00:17:48.630 ⇒ 00:17:53.689 Caio Velasco: a dashboard would be used or not, because that’s more granular.
191 00:17:54.820 ⇒ 00:18:01.390 Uttam Kumaran: Yeah. So that’s what I would look to see if you can find which explores you can deprecate there. May that may be a smaller subset.
192 00:18:01.860 ⇒ 00:18:04.899 Caio Velasco: And then the next thing that I would suggest.
193 00:18:05.040 ⇒ 00:18:09.070 Uttam Kumaran: Is. Yeah, starting to go through the views.
194 00:18:09.634 ⇒ 00:18:12.059 Uttam Kumaran: When you depre when you deprecate
195 00:18:12.440 ⇒ 00:18:17.880 Uttam Kumaran: sort of my suggestion would be 1st like turning.
196 00:18:19.070 ⇒ 00:18:21.910 Uttam Kumaran: Yeah, like, you basically have to turn off the view
197 00:18:22.110 ⇒ 00:18:25.269 Uttam Kumaran: and then uncomment it from the explorer.
198 00:18:26.510 ⇒ 00:18:27.850 Uttam Kumaran: You see what I mean?
199 00:18:28.617 ⇒ 00:18:36.049 Uttam Kumaran: And then I then I. Then I would remove it from redshift, first, st because again try to do the thing that’s easiest to roll back
200 00:18:36.190 ⇒ 00:18:39.350 Uttam Kumaran: like if if we and again, like we could.
201 00:18:39.610 ⇒ 00:18:42.860 Uttam Kumaran: The the way this works is like we can prep all the data. But like
202 00:18:42.980 ⇒ 00:18:49.269 Uttam Kumaran: at some point, someone’s gonna be like, Hey, where did this thing go, and we want to be able to go trace back what happened. So
203 00:18:49.420 ⇒ 00:18:53.960 Uttam Kumaran: my suggestion would be first, st if you’re like, okay, the all these sort of models.
204 00:18:54.150 ⇒ 00:19:00.129 Uttam Kumaran: All these views are scheduled to be deprecated. Let’s go ahead and comment comment them out
205 00:19:00.630 ⇒ 00:19:09.090 Uttam Kumaran: right? So you can just go into Looker. You can just control a comment, the entire file, and then go into the explore that joined.
206 00:19:09.210 ⇒ 00:19:14.740 Uttam Kumaran: And then you can comment out the individual view itself.
207 00:19:15.000 ⇒ 00:19:25.349 Uttam Kumaran: What the impact that makes is those those views, those the dimensions and metrics from that view will no longer be available
208 00:19:26.150 ⇒ 00:19:31.520 Uttam Kumaran: for any in any basically report where those are queried.
209 00:19:31.820 ⇒ 00:19:39.279 Uttam Kumaran: Right? But given your data, those aren’t being queried. So there shouldn’t be, you know, an issue.
210 00:19:43.870 ⇒ 00:19:50.770 Caio Velasco: Okay. Okay. Now, I do see new things here, and I’ll have to take a look into them. But I kind of get like the.
211 00:19:50.770 ⇒ 00:19:54.850 Uttam Kumaran: Yeah. And I think you’ll have this recording. So I feel like, it’ll yeah, once you.
212 00:19:54.850 ⇒ 00:19:57.891 Uttam Kumaran: which is, yeah, that that’ll be helpful. And then the other thing is
213 00:19:58.510 ⇒ 00:20:01.979 Uttam Kumaran: in Looker, there’s this thing called Content validator.
214 00:20:02.740 ⇒ 00:20:09.510 Uttam Kumaran: I would. I would take a look and maybe just look at the documentation about that. Basically, what it does is
215 00:20:11.130 ⇒ 00:20:24.650 Uttam Kumaran: you’re able to take a look at all the errors in all of the content. And what are what are content errors, so content errors is like, Oh, dashboard exists. But then the explorer that it was pulling from no longer exists.
216 00:20:24.800 ⇒ 00:20:32.830 Uttam Kumaran: or the dashboard exists, and the tile exists. But like 3 of the metrics
217 00:20:33.270 ⇒ 00:20:41.119 Uttam Kumaran: are now deprecated. So there’s an error. Right? So so this is probably gonna be what we’re look what we’re gonna look to.
218 00:20:41.530 ⇒ 00:20:44.829 Uttam Kumaran: This is how you find out if there’s problems
219 00:20:44.960 ⇒ 00:20:50.759 Uttam Kumaran: is like, once we make a change, we should run content. Validator, it will tell you like.
220 00:20:50.970 ⇒ 00:20:59.140 Uttam Kumaran: hey? We just removed this view, and it’s still referenced in these like 5 dashboards. And so those dashboards now have errors.
221 00:20:59.410 ⇒ 00:21:01.639 Uttam Kumaran: And so now what we have to do is either
222 00:21:01.880 ⇒ 00:21:06.380 Uttam Kumaran: we have to go into that dashboard and remove whatever’s being referenced.
223 00:21:06.760 ⇒ 00:21:14.130 Uttam Kumaran: or that dashboard needs to be deprecated. But I think you’ll find that this content validator will be sort of
224 00:21:14.260 ⇒ 00:21:16.490 Uttam Kumaran: pretty helpful if you take a look at it.
225 00:21:17.920 ⇒ 00:21:23.999 Caio Velasco: Okay. And I given this this new thing that you’re telling me. For example. I know that
226 00:21:24.150 ⇒ 00:21:40.430 Caio Velasco: Amber would probably be expecting me to end this like today or tomorrow, or something like that, because we’ve been already doing a lot of work in this mapping between dashboards and models and everything. But I mean, from what you’re saying, I would need like a day or 2 to to like. Go in.
227 00:21:40.430 ⇒ 00:21:45.899 Uttam Kumaran: Yeah, I, the way I would break it up is like by each. I would probably suggest breaking
228 00:21:46.040 ⇒ 00:22:09.889 Uttam Kumaran: this up with each scenario that we talked about right? So there’s 1 scenario with like completely unused dashboards. There’s another scenario with unused views. There’s another scenario with unused explores like I would break up the tickets there because you’ve actually done all the work to do the planning. And I wanna we we I’m I’m gonna propose that we run this through Zack before we execute
229 00:22:10.614 ⇒ 00:22:16.370 Uttam Kumaran: but then there’s also tickets to like actually go turn everything off.
230 00:22:17.080 ⇒ 00:22:25.690 Uttam Kumaran: Those I would totally suggest, like we have. We should do with Emily like sometime before Wednesday.
231 00:22:27.740 ⇒ 00:22:33.780 Uttam Kumaran: Because if we do this on a Thursday or Friday, someone on the weekend is gonna call us
232 00:22:34.150 ⇒ 00:22:39.369 Uttam Kumaran: so the earlier in the week we can try to do these sorts of deprecations
233 00:22:39.690 ⇒ 00:22:41.909 Uttam Kumaran: the more risk mitigation we have.
234 00:22:42.756 ⇒ 00:22:54.640 Uttam Kumaran: So my suggestion for for amber and for the team is is for you to mention. Like, Hey, we, we have these 3 scenarios of deprecations that need to happen
235 00:22:54.830 ⇒ 00:23:07.299 Uttam Kumaran: right? Like we need to drop explores. We need to drop dashboards, and then we also need to drop views of views are one to one mapping with tables, which means, once we drop the view. We can go drop the table.
236 00:23:07.680 ⇒ 00:23:11.960 Uttam Kumaran: Those 3 tickets need to get executed individually.
237 00:23:12.100 ⇒ 00:23:16.749 Uttam Kumaran: starting with the dashboards and explores because those are the easiest to roll back?
238 00:23:18.930 ⇒ 00:23:21.700 Uttam Kumaran: And then the 3rd one is is that yeah?
239 00:23:21.810 ⇒ 00:23:26.790 Uttam Kumaran: And and that that again is just like going through moving it, moving things around.
240 00:23:27.476 ⇒ 00:23:29.380 Uttam Kumaran: And yeah, that that’s
241 00:23:30.040 ⇒ 00:23:34.210 Uttam Kumaran: that. That’s that. That’d be. My suggestion is those 3 scenarios are the biggest ones.
242 00:23:35.490 ⇒ 00:23:38.050 Uttam Kumaran: Okay, no, okay, that makes total sense. Makes no sense.
243 00:23:38.050 ⇒ 00:23:51.631 Uttam Kumaran: Part of part of this, really, I would say, is, and you know, is this risk mitigation aspect? So I would just try to be methodical in the way we chop things out, because the more sort of like,
244 00:23:52.130 ⇒ 00:24:10.169 Uttam Kumaran: course we are. The the risk of it coming back to us, for example, if someone over the weekend or let’s say someone in 3 weeks is like, Hey like a couple of years ago we had this explorer that did. Xyz. I can’t find it anymore. And I need it.
245 00:24:11.040 ⇒ 00:24:15.620 Uttam Kumaran: That’s like an example of what could happen right? So then, in that situation, what do we do
246 00:24:16.050 ⇒ 00:24:40.069 Uttam Kumaran: so at that point we have to say, Okay, which explorer was it? Or what were the types of things you’re doing? Can we do that in something that already exists. If not okay, we could rebuild it, and then there’s some. We can go look through our code that we did to rebuild it. But one example is like, let’s say, there’s data from A, from a Etl, from like something coming from stitch. And then we go and drop it.
247 00:24:40.320 ⇒ 00:24:42.019 Uttam Kumaran: That data is like lost.
248 00:24:42.350 ⇒ 00:24:45.279 Uttam Kumaran: Right? So probably my suggestion, even in this, would be like.
249 00:24:45.380 ⇒ 00:24:48.260 Uttam Kumaran: Don’t drop any data, for now
250 00:24:48.550 ⇒ 00:24:54.210 Uttam Kumaran: I think what we’ll do is move everything to some to like an archive or a cold storage.
251 00:24:55.660 ⇒ 00:25:01.379 Uttam Kumaran: because the one thing that in this whole process we can get looker code back, we can build dashboards.
252 00:25:01.520 ⇒ 00:25:09.439 Uttam Kumaran: but we can’t get. If we already etl the data in, it’s gonna be really rough to try to find it again. If we drop those tables.
253 00:25:09.780 ⇒ 00:25:17.050 Uttam Kumaran: and the storage cost is not that high? So I think what what we should consider is like, we don’t drop any tables.
254 00:25:17.310 ⇒ 00:25:20.679 Uttam Kumaran: I think we should drop it from Dbt code
255 00:25:22.188 ⇒ 00:25:25.030 Uttam Kumaran: or archive the Dbt code.
256 00:25:25.300 ⇒ 00:25:29.950 Uttam Kumaran: And then, second, we should put those tables into an archive section
257 00:25:30.120 ⇒ 00:25:36.299 Uttam Kumaran: right? And I can. I can help with that. But those would be, think those would be tickets that I would also suggest making
258 00:25:36.600 ⇒ 00:25:42.399 Uttam Kumaran: that way. We don’t run the risk of like losing any data that we can’t. We can’t reverse, you know.
259 00:25:43.680 ⇒ 00:25:45.900 Caio Velasco: Yeah, yeah, no. Perfect.
260 00:25:46.070 ⇒ 00:25:47.380 Caio Velasco: I got the idea.
261 00:25:47.700 ⇒ 00:25:48.840 Uttam Kumaran: Okay.
262 00:25:50.380 ⇒ 00:25:53.973 Caio Velasco: Okay. So I think, let me see, what else is here?
263 00:25:56.320 ⇒ 00:26:08.209 Caio Velasco: yeah. Well, we talked about dbt, look, but we also talked about dbt ratchet. So I think for those, it’s okay. I’ll write all this down and then propose it to Amber and then start working on it.
264 00:26:09.830 ⇒ 00:26:19.040 Caio Velasco: for the modeling part. I saw that I already have a ticket to start looking into the revenue model, or at least the.
265 00:26:19.040 ⇒ 00:26:19.380 Uttam Kumaran: Yes.
266 00:26:19.380 ⇒ 00:26:25.879 Caio Velasco: Let’s say, March, because I don’t think. Don’t even think a net revenue model exists in our restance.
267 00:26:26.571 ⇒ 00:26:30.000 Caio Velasco: They try to calculate revenue in many ways with
268 00:26:30.220 ⇒ 00:26:33.286 Caio Velasco: many tables, but it seems a bit disorganized.
269 00:26:34.880 ⇒ 00:26:40.270 Caio Velasco: And then I know that the lad is, has more knowledge in the inventory part.
270 00:26:40.730 ⇒ 00:26:43.500 Caio Velasco: So then I was talking to him today about it.
271 00:26:43.760 ⇒ 00:26:49.470 Caio Velasco: And my idea it’s although he already told me that there is a table.
272 00:26:49.640 ⇒ 00:26:56.140 Caio Velasco: an important one downstream, which is called tableau items, except
273 00:26:56.270 ⇒ 00:26:59.720 Caio Velasco: and from there, if you look at the lineage. Yes, there’s a lot of things
274 00:27:00.336 ⇒ 00:27:06.929 Caio Velasco: so my deal with. I’ve never done this, but when I heard that what we want is to be able to calculate revenue
275 00:27:07.503 ⇒ 00:27:24.579 Caio Velasco: which it’s at the end of the day, an equation that, you know, theoretically, ideally, would be price times quantity. But obviously it’s not so. So I started to have a lot of questions first, st like I learned that they are using only shopify now. So then.
276 00:27:24.790 ⇒ 00:27:27.889 Caio Velasco: my question, my 1st question would be, for example,
277 00:27:28.960 ⇒ 00:27:40.050 Caio Velasco: these things probably are defined by shopify like when you sell something. What happens? I don’t know how the shipping is processed, or or discounts, or.
278 00:27:40.050 ⇒ 00:27:40.620 Uttam Kumaran: Yes.
279 00:27:40.620 ⇒ 00:27:45.290 Caio Velasco: You know. So all those things theoretically shopify have them.
280 00:27:46.018 ⇒ 00:27:57.169 Caio Velasco: But this is already something that because I’m not, I’m not super experienced in this, but on the urban stem side they are also calculating with lots of crazy sequel. So what is the best
281 00:27:57.760 ⇒ 00:27:59.160 Caio Velasco: to start this?
282 00:27:59.360 ⇒ 00:28:13.789 Caio Velasco: Because in my mind, I would at some point be looking at the blank notion page and say, Okay, what is revenue factors affects revenue? How do we trace an order from scratch, etc, etc, etc. How would you do.
283 00:28:13.790 ⇒ 00:28:19.690 Uttam Kumaran: Yeah, so a couple of different, you have a couple of different options. So one is
284 00:28:20.462 ⇒ 00:28:25.309 Uttam Kumaran: in the code. Right now, what you’re gonna see is like they have.
285 00:28:25.560 ⇒ 00:28:35.029 Uttam Kumaran: The the main problem here is that they have. They moved over to shopify recently, but historically, the data is coming from another place. So
286 00:28:35.330 ⇒ 00:28:39.049 Uttam Kumaran: one problem is just like, Okay, where’s live? Revenue coming from?
287 00:28:39.180 ⇒ 00:28:56.697 Uttam Kumaran: Another problem is, where is can we trace back where all revenue came from? Right. So you may have to tie shopify with whatever the old system. So. But like, let’s put that, let’s put that problem in the back, like, if we just talk about live revenue. So the 1st thing to do is
288 00:28:57.250 ⇒ 00:28:59.729 Uttam Kumaran: to basically trace back.
289 00:29:00.160 ⇒ 00:29:06.970 Uttam Kumaran: Take the metric that’s being used as revenue. Right? So that’s probably something in tableau items Xf or whatever.
290 00:29:07.120 ⇒ 00:29:17.930 Uttam Kumaran: and literally walk it all the way back. And while you go back, take all of the transformations that are happening. And that will basically build you
291 00:29:18.150 ⇒ 00:29:21.129 Uttam Kumaran: like from source to
292 00:29:21.320 ⇒ 00:29:34.640 Uttam Kumaran: tableau items, how revenue is built, what you’re gonna find there is gonna be really brutal like. There’s so much crappy sequel written that
293 00:29:35.090 ⇒ 00:29:40.529 Uttam Kumaran: this this file, or whatever the view you’re gonna write, is just gonna have a lot of
294 00:29:40.810 ⇒ 00:29:56.670 Uttam Kumaran: SQL transformations, mainly because they just didn’t model this end to end, they sort of kept adding, adding, adding, adding, adding, right? So the 1st thing is like, Okay, how can we simplify the distance between shopify source and
295 00:29:57.120 ⇒ 00:30:03.210 Uttam Kumaran: the whatever? The final revenue table is right like right now, if it’s let’s say it’s 30 hops like
296 00:30:03.530 ⇒ 00:30:09.450 Uttam Kumaran: one table, one table, one table, one table. Okay, can we do that like 4? Right? So one thing is like.
297 00:30:09.660 ⇒ 00:30:17.099 Uttam Kumaran: okay, it’s clear that, like right now, the metric jumps through all these. But looking at the source, it’s primarily coming from this table.
298 00:30:17.220 ⇒ 00:30:23.210 Uttam Kumaran: And roughly, I could now map all the different logic pieces that are happening.
299 00:30:23.986 ⇒ 00:30:37.120 Uttam Kumaran: And then we start to build the latest table. My my goal for you is to show some value that you can start to debug, and compare to as fast as you can, right?
300 00:30:37.490 ⇒ 00:30:55.730 Uttam Kumaran: Meaning instead of spending the time like. Okay, I traced revenue all the way back through 30 models back to shopify. And I want to try to really, meticulously create the value, recreate the value. What you’re gonna find is their value may be wrong.
301 00:30:55.860 ⇒ 00:31:07.139 Uttam Kumaran: We don’t know yet, and so I’m my suggestion would be show a number saying, Hey, all I did simply was like, We have this tab. We have the shopify orders.
302 00:31:07.640 ⇒ 00:31:11.670 Uttam Kumaran: You know we’ve done this for a couple of other clients before we built our revenue model.
303 00:31:11.990 ⇒ 00:31:18.239 Uttam Kumaran: And let’s start from there and then let’s start to question, like what assumptions are happening, and why those 2 numbers are different.
304 00:31:18.500 ⇒ 00:31:24.370 Uttam Kumaran: In that moment when you start to question the assumptions is when we can start to write the documentation.
305 00:31:24.570 ⇒ 00:31:38.909 Uttam Kumaran: Oh, like we forgot to account for X, or like oh, we we have, like some ancillary revenue coming from another place, or oh, we always this type of discount code X thing happens like those are all.
306 00:31:39.110 ⇒ 00:31:43.390 Uttam Kumaran: Those are all nonstandard logic things that we have to document.
307 00:31:43.740 ⇒ 00:31:52.030 Uttam Kumaran: so that anytime we look at, hey? Why is our revenue different from just select revenue from shopify? It’s Xyz ABC reasons.
308 00:31:53.020 ⇒ 00:32:01.720 Uttam Kumaran: So that’s like, how I would probably go through this with the key thing being like, show a number that the team can react and give feedback to
309 00:32:01.990 ⇒ 00:32:10.030 Uttam Kumaran: quickly, because that conversation is what’s gonna help you get like the next caveat versus.
310 00:32:10.340 ⇒ 00:32:14.079 Uttam Kumaran: yeah, it’s it’s too large to spend like
311 00:32:14.390 ⇒ 00:32:17.170 Uttam Kumaran: so much time trying to replicate.
312 00:32:17.720 ⇒ 00:32:24.519 Uttam Kumaran: Because a lot of the existing logic sucks really really bad, like by the time you try to replicate it.
313 00:32:24.973 ⇒ 00:32:29.980 Uttam Kumaran: There’s like, it’s not worth even replicating it like we should completely start from scratch.
314 00:32:32.590 ⇒ 00:32:37.830 Caio Velasco: Okay. Okay, no, it’s it’s yeah, no, it’s a lot of work for sure. But.
315 00:32:37.830 ⇒ 00:32:47.859 Uttam Kumaran: It’s also it’s also not an easy. It’s not a definitely, not an easy thing. But this is where like it’s worth starting from. My suggestion is to start from scratch.
316 00:32:48.220 ⇒ 00:32:53.890 Uttam Kumaran: and like we can find different ways of getting the answer. But.
317 00:32:53.890 ⇒ 00:32:54.490 Caio Velasco: Oops!
318 00:32:54.710 ⇒ 00:33:05.980 Uttam Kumaran: I’m telling you when you go trace that thing back. It was gonna be so many views, and it’s like some of the models are like 5 lines. Some of them are like 2,000 lines. It’s just like.
319 00:33:06.846 ⇒ 00:33:09.619 Uttam Kumaran: I don’t know. I don’t see the worth
320 00:33:09.880 ⇒ 00:33:13.480 Uttam Kumaran: in referencing any of their stuff, you know.
321 00:33:14.570 ⇒ 00:33:15.160 Caio Velasco: Okay.
322 00:33:15.390 ⇒ 00:33:25.129 Caio Velasco: okay, no perfect. And a question that I had also when I was talking with them a lot is that one of the things you can find there is that
323 00:33:25.738 ⇒ 00:33:35.309 Caio Velasco: they have something that they call buffer, that they usually send like 30 flowers, because they know that’s only gonna arrive? 25 for whatever reason?
324 00:33:36.690 ⇒ 00:33:49.660 Caio Velasco: And then when as well, he told me that I was like, do they, how do they predict that do they have like? Do they go into machine learning, modeling to predict those things? This is something we can aim for the future
325 00:33:50.600 ⇒ 00:33:52.450 Caio Velasco: just because I’m interested.
326 00:33:52.670 ⇒ 00:34:02.580 Uttam Kumaran: Yeah, yeah, no, you. So you should so, Perry, I think it’s Perry or Felipe, one of the analysts on their team runs this they are not using
327 00:34:03.020 ⇒ 00:34:15.980 Uttam Kumaran: the buffer model is, is just sequel, like, they’re not using any sort of predictive analytics to produce that. In fact, what they do is they have a calculation where they like.
328 00:34:16.120 ⇒ 00:34:30.010 Uttam Kumaran: They basically create some sort of factor, and it adds a certain amount, and then they order it. They’ve never. I don’t think they’ve done an analysis on like the waste coming out of that so long story. Long answer is like or short answer is like
329 00:34:30.210 ⇒ 00:34:32.179 Uttam Kumaran: they’re not doing anything. Fancy
330 00:34:33.429 ⇒ 00:34:37.759 Uttam Kumaran: you’ll see that in the data, though they’re like, it’s just a SQL. Query which is like.
331 00:34:38.639 ⇒ 00:34:46.880 Uttam Kumaran: really like, I agree with you, and that we should, it would be awesome to measure like, hey, how much are we spending on buffer? And
332 00:34:47.139 ⇒ 00:34:52.639 Uttam Kumaran: is there any way to to make the buffer more accurate. Right? Like the amount of buffer we order more accurate.
333 00:34:53.170 ⇒ 00:34:54.860 Uttam Kumaran: It’s a great project. Yeah.
334 00:34:55.400 ⇒ 00:34:56.426 Caio Velasco: Okay. Okay.
335 00:34:56.940 ⇒ 00:34:58.860 Uttam Kumaran: That’s gonna be for the renewal.
336 00:35:05.220 ⇒ 00:35:07.629 Caio Velasco: Let me see what else?
337 00:35:08.110 ⇒ 00:35:12.610 Caio Velasco: Yeah, no, I think. Well, I think you you told me like a lot of good things.
338 00:35:13.380 ⇒ 00:35:14.070 Caio Velasco: yeah, I think.
339 00:35:14.070 ⇒ 00:35:32.590 Uttam Kumaran: I think this video will help you like, keep referencing it. It’ll some of the things I said, I think, will become a little bit more clear once you’re sort of map it out. But my, my number one thing is like, go slow and go slow and make sure that there’s no problems come back to bite us, because
340 00:35:32.710 ⇒ 00:35:40.559 Uttam Kumaran: these migrations always tend to end in like oh, we accidentally dropped like a hundred things, and like we have no way to get back to it, and then
341 00:35:40.870 ⇒ 00:35:47.050 Uttam Kumaran: it’s really painful, and really try and try to aim not to do anything. Thursday, Fridays.
342 00:35:47.200 ⇒ 00:35:50.088 Uttam Kumaran: Otherwise the weekend will become drama.
343 00:35:51.900 ⇒ 00:35:58.989 Caio Velasco: That’s very true. 1 1 quick thing, just because I was working on the on estimating costs for.
344 00:35:58.990 ⇒ 00:35:59.310 Uttam Kumaran: Yeah.
345 00:35:59.310 ⇒ 00:36:08.889 Caio Velasco: Turned off. I started with the 1st table that we, with the 1st pipeline that we turned off, which is also just one table. That was like Amazon.
346 00:36:09.020 ⇒ 00:36:11.159 Caio Velasco: whatever whatever. And
347 00:36:12.150 ⇒ 00:36:20.259 Caio Velasco: I was trying to understand how well I went to stitch, and I saw that there was no data there anymore. And then, apparently, you can’t access.
348 00:36:21.010 ⇒ 00:36:26.910 Caio Velasco: There you usage extraction loading data because it was already turned off.
349 00:36:27.716 ⇒ 00:36:34.550 Caio Velasco: So that’s like, okay, how should I do this. And then I was like, Okay, maybe redshift right? Go into redshift and see the batches
350 00:36:34.760 ⇒ 00:36:45.559 Caio Velasco: over there. And then when I checked the minimum and maximum dates. Actually it appeared that the last one was like last year, November.
351 00:36:45.830 ⇒ 00:36:51.309 Caio Velasco: So then I was like, Okay, but it was turned on. Do you have any like knowledge in in why
352 00:36:51.440 ⇒ 00:36:58.270 Caio Velasco: it was on, and we turned off. But the data was not even being replic ingested anymore. Since last year.
353 00:36:59.300 ⇒ 00:37:01.610 Uttam Kumaran: Yeah, it may have been on, and then
354 00:37:01.740 ⇒ 00:37:08.269 Uttam Kumaran: no, there’s no schedule. So that that could be one thing it could be just like, turn on, and it’s ad hoc. Second, the connector.
355 00:37:08.380 ⇒ 00:37:11.499 Uttam Kumaran: like. Whatever the connection was, it could have been broken.
356 00:37:13.210 ⇒ 00:37:18.760 Uttam Kumaran: Right, then another way to also estimate this is like what is like. It’s not only the cost of like
357 00:37:19.040 ⇒ 00:37:23.360 Uttam Kumaran: oh, it’s a hundred 1,000 rows on redshift, but like, what is the complexity cost?
358 00:37:23.610 ⇒ 00:37:27.040 Uttam Kumaran: Right? And this is where you may have to go. Look at some papers, or like.
359 00:37:27.290 ⇒ 00:37:31.679 Uttam Kumaran: look at some like estimates. But what is the cost of having
360 00:37:31.890 ⇒ 00:37:39.900 Uttam Kumaran: additional connector? Additional model? Additional dB, like, what is the actual like engineering like? How much time does it add
361 00:37:40.010 ⇒ 00:37:49.840 Uttam Kumaran: when we go to debug something, or when we go to add a new feature right like that, I think, is the hot gonna be the higher cost than like the redshift storage cost
362 00:37:49.990 ⇒ 00:38:08.320 Uttam Kumaran: right. And this is where I think, like we have to do some storytelling. But I think you should try to find, like some estimate of like, Hey, every additional table and connector we have is an additional like $10 a month in like all costs, included complexity
363 00:38:08.580 ⇒ 00:38:23.249 Uttam Kumaran: and storage and runtime, and so, like everything we remove. We’re saving this much money, because guess what? If if every time we go look for something, and we have to scroll through a hundred 1,000 things. That’s time, and that’s money
364 00:38:23.670 ⇒ 00:38:24.960 Uttam Kumaran: coming from the company.
365 00:38:25.280 ⇒ 00:38:32.560 Uttam Kumaran: right? So those are things that I would. Maybe that’s probably where I would suggest the actual bulk of the cost is gonna come from.
366 00:38:33.560 ⇒ 00:38:48.089 Caio Velasco: Okay? And when you’re saying about this estimation, do you mean like end end to end like from when we start like ingesting, so speak, cost rows cost, and then everything else that is said would be related to
367 00:38:48.260 ⇒ 00:38:55.150 Caio Velasco: it, being materialized, for example, by Dbt. And everything along until it gets to a dashboard. That’s what I mean.
368 00:38:55.850 ⇒ 00:39:02.870 Uttam Kumaran: Yeah. But I would say, like the stitch, like the redshift cost to store a million rows is very cheap.
369 00:39:03.050 ⇒ 00:39:05.800 Uttam Kumaran: right? And like the cost to
370 00:39:06.040 ⇒ 00:39:12.669 Uttam Kumaran: keep the code, and Dbt is very cheap and looker. There’s no incremental cost for another dashboard.
371 00:39:12.860 ⇒ 00:39:17.219 Uttam Kumaran: So where the cost is coming from, though, is from how disorganized
372 00:39:17.620 ⇒ 00:39:24.140 Uttam Kumaran: and how bloated the system is. That’s what’s actually cost cost money here, right.
373 00:39:24.740 ⇒ 00:39:26.130 Caio Velasco: Okay, okay.
374 00:39:26.130 ⇒ 00:39:31.460 Caio Velasco: right? The money comes from like, how long it’s gonna take us to figure things out when a problem happens.
375 00:39:32.030 ⇒ 00:39:38.350 Uttam Kumaran: That’s the true cost, right? But this is where it’s like I’m not. You’ll you’ll have to think about this a little bit.
376 00:39:38.490 ⇒ 00:39:46.670 Uttam Kumaran: But ultimately, I don’t know if you, if you, if you just make an estimate for one, and then you scale that up across all the stuff we do.
377 00:39:47.110 ⇒ 00:39:57.699 Uttam Kumaran: But that is actually what I think the the real savings are in time. And basically engineering hours takes it takes to build anything
378 00:39:57.810 ⇒ 00:39:59.539 Uttam Kumaran: or or debug anything.
379 00:40:03.420 ⇒ 00:40:10.890 Uttam Kumaran: This is gonna be a few dollars, maybe per year. It’s just it’s just not. But like, you know, that that’s not right. Like
380 00:40:11.450 ⇒ 00:40:14.960 Uttam Kumaran: there, there’s some like, hey! What is the cost of having, like
381 00:40:15.170 ⇒ 00:40:20.097 Uttam Kumaran: all this? Shit everywhere, like what is it right there? So there’s some number.
382 00:40:21.640 ⇒ 00:40:26.600 Caio Velasco: Yeah, no, I get it. I get it. There’s like more, something more intangible. There. Yeah.
383 00:40:26.600 ⇒ 00:40:37.029 Caio Velasco: that was on my mind when I when I started. This because of the ticket was that I thought I was just really calculating the like. If.
384 00:40:37.030 ⇒ 00:40:44.140 Uttam Kumaran: No, I think I think you’re right as well. I didn’t. I just thought about this last week where I was kind of trying to think about what are the
385 00:40:44.260 ⇒ 00:40:49.530 Uttam Kumaran: like cause I saw. You’re like, Hey, like Demo? Lotto is also saying like, Can we put together some sort of like impact.
386 00:40:49.660 ⇒ 00:40:53.309 Uttam Kumaran: And I do think that
387 00:40:54.030 ⇒ 00:40:58.460 Uttam Kumaran: it needs. I do think that it needs to be more about like engineering hours saved
388 00:40:59.180 ⇒ 00:41:02.539 Uttam Kumaran: in addition to technology cost impact
389 00:41:02.840 ⇒ 00:41:05.849 Uttam Kumaran: for it to actually be a meaningful number.
390 00:41:07.010 ⇒ 00:41:08.370 Caio Velasco: Okay. Okay.
391 00:41:08.690 ⇒ 00:41:09.490 Caio Velasco: Okay. Okay.
392 00:41:09.490 ⇒ 00:41:14.179 Uttam Kumaran: It’s like it’s a little bit of storytelling, though, like this is where, like I I don’t know.
393 00:41:14.610 ⇒ 00:41:28.069 Uttam Kumaran: Look, we don’t need to be. I don’t. I just don’t. I wouldn’t go for like a hundred percent accurate, verifiable like. But I would say that has to be directionally correct, which is like like, what is the savings like? Let’s say you were to play
394 00:41:28.810 ⇒ 00:41:30.589 Uttam Kumaran: pay brain forge.
395 00:41:31.050 ⇒ 00:41:34.579 Uttam Kumaran: What? And we came in. How faster would we have gone
396 00:41:34.690 ⇒ 00:41:37.030 Uttam Kumaran: if there was all this stuff was cleaned up
397 00:41:37.370 ⇒ 00:41:43.430 Uttam Kumaran: and like, what is the cost right like? That’s the sort of. But it’s kind of a weird thing like I don’t know how else we can
398 00:41:43.880 ⇒ 00:41:52.139 Uttam Kumaran: demonstrate that. But payroll costs like engineering cost is the for the for the folks are the number one cost center like
399 00:41:52.530 ⇒ 00:41:54.489 Uttam Kumaran: beyond any tool we’re using.
400 00:41:54.760 ⇒ 00:41:57.699 Uttam Kumaran: And so that’s really what I told them is like, look.
401 00:41:58.300 ⇒ 00:42:02.390 Uttam Kumaran: I think your costs for your tools are okay, like we’ll save a lot. But
402 00:42:02.890 ⇒ 00:42:07.279 Uttam Kumaran: really the cost is how long it just takes to do simple thing. Here, you know.
403 00:42:10.840 ⇒ 00:42:23.810 Caio Velasco: Yeah, no, that that’s interesting. My mind was basically. Well, if you’re turning off an injection, pipeline is because you’re not using. So if you’re not using what was what were the costs that associated with it? No one.
404 00:42:25.230 ⇒ 00:42:26.250 Uttam Kumaran: Yeah.
405 00:42:27.300 ⇒ 00:42:28.550 Caio Velasco: Exactly.
406 00:42:30.770 ⇒ 00:42:31.530 Caio Velasco: Okay.
407 00:42:32.210 ⇒ 00:42:36.530 Caio Velasco: Okay. So I think we we covered maybe basically everything I had.
408 00:42:36.984 ⇒ 00:42:48.919 Caio Velasco: The only other question that I would have would would be something personal from what you message me because I was just curious. So when you, when you said about like a potential full time role or something.
409 00:42:49.060 ⇒ 00:42:56.900 Caio Velasco: Do you have anything? Are they more concrete in your mind, or what are you considering? I mean? I don’t have. I don’t expect any final answer. I’m just curious.
410 00:42:57.490 ⇒ 00:43:08.300 Uttam Kumaran: Yeah. So let me give you the sort of the math on our side. So for everybody, for everybody on the engineering side, our goal is to get you between 80 to 100%
411 00:43:08.610 ⇒ 00:43:15.620 Uttam Kumaran: sort of billable meaning on any given week. 80% of your time is
412 00:43:15.860 ⇒ 00:43:39.770 Uttam Kumaran: going towards client work. I think, like we did a lot of review of, like, everybody’s clockify hours. And currently like you do have. Of course, like you’re you’re gonna say, yeah, there’s bandwidth like, it’s clear in the in the data as well that you have bandwidth right? So my job before I can move to making that full time offer is to make sure that you have
413 00:43:40.300 ⇒ 00:43:42.470 Uttam Kumaran: like that stable work.
414 00:43:44.250 ⇒ 00:43:57.920 Uttam Kumaran: the otherwise like. There’s no there. There isn’t like extra money to make that change right now. So my 1st goal is like, how can I reliably get you to 80% billable for at least
415 00:43:58.130 ⇒ 00:43:59.409 Uttam Kumaran: a month or 2.
416 00:44:01.320 ⇒ 00:44:07.690 Uttam Kumaran: right that way, like you’ll start to. You’ll start to basically say, cool. I am working anywhere from like 30 to 40 HA week.
417 00:44:09.190 ⇒ 00:44:19.630 Uttam Kumaran: And that’s the time at which, like I, I’m I’m more comfortable saying, Okay, cool. We. We’ve reached some stability and you feel comfortable, and then we’ll move to
418 00:44:19.900 ⇒ 00:44:22.510 Uttam Kumaran: move to an offer.
419 00:44:24.900 ⇒ 00:44:39.250 Caio Velasco: Okay, okay, I think I got it. Yeah, okay, perfect. Because so far, I’m considering more or less like what we discussed before that I would be working like around 20 h for this client per week at least. That’s what
420 00:44:39.727 ⇒ 00:44:46.270 Caio Velasco: but I well, of course, for the last week, and maybe last 2 weeks I I have put more, much more out.
421 00:44:46.610 ⇒ 00:44:48.400 Caio Velasco: Want to learn, and that’s 1 b.
422 00:44:48.520 ⇒ 00:45:07.350 Caio Velasco: But I also just want you to know that that I am working kind of full time with. Yes, but I prefer for now to keep what we discussed, and then deliver value and quality, because I don’t like to deliver work without quality. It’s just not in my nature.
423 00:45:07.730 ⇒ 00:45:10.360 Caio Velasco: and even if I spend more hours, it’s fine.
424 00:45:10.600 ⇒ 00:45:15.420 Caio Velasco: But then also consider that between those things you see in clockify maybe.
425 00:45:15.420 ⇒ 00:45:16.059 Uttam Kumaran: Yeah, yeah.
426 00:45:16.060 ⇒ 00:45:22.960 Caio Velasco: 1020, 30% more over there that I’m putting on my own. So then, well, when we get to full time, then maybe I would be more
427 00:45:23.711 ⇒ 00:45:25.700 Caio Velasco: connected in this. You know what.
428 00:45:25.700 ⇒ 00:45:32.509 Uttam Kumaran: It makes sense like, I don’t want you to have to double think on like, oh, shit, I’m not gonna get paid for these hours.
429 00:45:33.156 ⇒ 00:45:36.210 Uttam Kumaran: I also want to make sure that
430 00:45:36.400 ⇒ 00:45:43.220 Uttam Kumaran: for every hour you’re working like the company is able to make make money for that, you know. So I don’t think we’re very far off.
431 00:45:43.390 ⇒ 00:45:59.240 Uttam Kumaran: but I hear you like i i 1, of course, like I, I actually am very similar, like, even if I’m not getting paid like I just the work needs to needs to happen. And and for you, that’s a bet on us, right? And and for me, my job is to
432 00:45:59.380 ⇒ 00:46:10.760 Uttam Kumaran: get you on one of these clients that’s coming up. Fill the rest of your time, so that if you do go a little bit over or under. It’s like there’s no difference. Right? At least you’re you have expectations set
433 00:46:11.246 ⇒ 00:46:22.953 Uttam Kumaran: and then that’s like, that’s what we’re trying to do with everybody right? Because for the core group of folks that’s here right now, my goal is to have everybody as close to full time as possible.
434 00:46:23.830 ⇒ 00:46:28.530 Uttam Kumaran: part of the math on our side is, I have to be able to pull that trigger once we
435 00:46:28.920 ⇒ 00:46:32.270 Uttam Kumaran: have cash coming in from clients given, just like
436 00:46:33.060 ⇒ 00:46:44.690 Uttam Kumaran: the really the clients are. The problem here is that a lot of payments from clients are always delayed, and so, if I promise to send money out, I really don’t want to have to delay that.
437 00:46:44.990 ⇒ 00:46:53.169 Uttam Kumaran: So I’m really trying to bias towards like, okay, let’s get the client in. Let’s get folks working. Let’s collect the money, and then
438 00:46:53.530 ⇒ 00:47:06.530 Uttam Kumaran: I’m much more comfortable, being like, make a full time offer, which is really sad, because, like, I would love this to be like more real time, like fine comes in. Okay, I know the money’s gonna come. It’s like, No, I can pay you. So let’s move to something.
439 00:47:06.890 ⇒ 00:47:13.110 Uttam Kumaran: But the number one struggle in the last few months has been just collecting money from clients like it’s so brutal
440 00:47:13.290 ⇒ 00:47:15.490 Uttam Kumaran: that people don’t pay us on time.
441 00:47:15.958 ⇒ 00:47:19.639 Uttam Kumaran: And so I really don’t. I just don’t want to
442 00:47:19.890 ⇒ 00:47:28.030 Uttam Kumaran: do what’s happened in the last few months where I have to delay things like it’s just not something I like to do at all. So that’s why I’m being more
443 00:47:28.150 ⇒ 00:47:29.410 Uttam Kumaran: conservative.
444 00:47:31.470 ⇒ 00:47:35.989 Uttam Kumaran: Like, that’s that’s that’s the only sort of reason on my side.
445 00:47:37.420 ⇒ 00:47:39.569 Caio Velasco: Okay, no, perfect. Understood? Understood?
446 00:47:39.700 ⇒ 00:47:51.312 Caio Velasco: Okay, no problem. It makes sense. And definitely, I’m on board. So when you have more news, let me know, and then I also can plan like my month, so that I don’t take things that I shouldn’t.
447 00:47:51.610 ⇒ 00:47:53.270 Uttam Kumaran: Perfect. Perfect. Okay.
448 00:47:53.860 ⇒ 00:47:54.290 Caio Velasco: Perfect.
449 00:47:54.290 ⇒ 00:47:54.780 Uttam Kumaran: Okay.
450 00:47:54.780 ⇒ 00:47:55.499 Caio Velasco: Yes, ma’am, thank you.
451 00:47:55.900 ⇒ 00:47:57.739 Caio Velasco: Thank you. Thank you. I appreciate.
452 00:47:58.250 ⇒ 00:47:59.270 Uttam Kumaran: Yeah. Talk. Soon.
453 00:47:59.910 ⇒ 00:48:00.550 Caio Velasco: Thanks.