Meeting Title: Data Platform Chat - Async & Observability Date: 2025-04-30 Meeting participants: Awaish Kumar, Uttam Kumaran
WEBVTT
1 00:01:22.870 ⇒ 00:01:23.620 Awaish Kumar: Hello!
2 00:01:24.650 ⇒ 00:01:25.320 Uttam Kumaran: A
3 00:01:37.260 ⇒ 00:01:39.170 Uttam Kumaran: gotta wait for a couple more people.
4 00:02:44.810 ⇒ 00:02:50.290 Uttam Kumaran: Okay, maybe let’s do you want to go ahead, Aisha? Maybe we can just walk through the observability. Doc.
5 00:02:50.847 ⇒ 00:03:05.490 Uttam Kumaran: I left some comments. I think the biggest thing is, I wanted to sort of turn it all the docs into sort of one similar format. So I do. I did really enjoy what demalade and kayo produce in terms of their format. It was almost like, basically like a Prd.
6 00:03:07.820 ⇒ 00:03:12.640 Uttam Kumaran: So yeah, I don’t know if you want to share. Maybe we can walk through that either way.
7 00:03:15.190 ⇒ 00:03:15.700 Uttam Kumaran: Sure.
8 00:04:12.430 ⇒ 00:04:13.719 Awaish Kumar: Yeah. Here.
9 00:04:15.570 ⇒ 00:04:22.900 Uttam Kumaran: Okay? Great. Yeah. I think the only piece of feedback I have is
10 00:04:24.900 ⇒ 00:04:31.069 Uttam Kumaran: basically trying to like, did did you have a chance to look at how their document was structured
11 00:04:32.010 ⇒ 00:04:32.650 Uttam Kumaran: at all.
12 00:04:32.650 ⇒ 00:04:33.070 Awaish Kumar: Alright.
13 00:04:34.950 ⇒ 00:04:44.180 Awaish Kumar: I think I click share. But I can. Can I open kind of
14 00:04:54.500 ⇒ 00:04:56.430 Awaish Kumar: okay, this this one.
15 00:04:57.300 ⇒ 00:04:58.010 Uttam Kumaran: Yes.
16 00:04:59.630 ⇒ 00:05:05.940 Uttam Kumaran: so I think they just like had a couple of concepts here that were helpful, like, what are the core principles?
17 00:05:06.310 ⇒ 00:05:10.470 Uttam Kumaran: What is the problem. So I think I think part of this
18 00:05:11.460 ⇒ 00:05:21.560 Uttam Kumaran: just helps anyone sort of understand, like, why we made the decisions. So I don’t know. I felt like this. It was they kind of went. They kind of had a lot. So I think they’ll end up cleaning some up. But
19 00:05:21.660 ⇒ 00:05:27.020 Uttam Kumaran: I think the brains, references, brainstorming, the references, core principles.
20 00:05:27.390 ⇒ 00:05:38.480 Uttam Kumaran: And like basically the problem and why and the end goal are, I think, like, very, very helpful. And then that way, we can start to break this out, break it out into like tickets, you know.
21 00:05:39.780 ⇒ 00:05:47.746 Awaish Kumar: Okay, like, how like? So this is like, more like theoretical things like,
22 00:05:48.490 ⇒ 00:05:51.150 Awaish Kumar: what is possibility? What are the
23 00:05:51.500 ⇒ 00:05:54.110 Awaish Kumar: like? We I also mentioned like, what are the
24 00:05:54.300 ⇒ 00:05:56.550 Awaish Kumar: what I want to, Mayo? What is the.
25 00:05:56.820 ⇒ 00:06:03.439 Uttam Kumaran: I think it’s just a formatting. I think it’s just purely like a formatting thing. I think you actually have all the all the information.
26 00:06:04.941 ⇒ 00:06:10.749 Uttam Kumaran: I think I just want to. I think this is this is more about like the Mvps.
27 00:06:12.140 ⇒ 00:06:12.559 Awaish Kumar: There we go!
28 00:06:12.560 ⇒ 00:06:19.389 Uttam Kumaran: I think, from the top section. I think, for example, if we’re going to go to like amber and say like, Hey, here’s an explanation of like, what is.
29 00:06:19.740 ⇒ 00:06:24.800 Uttam Kumaran: okay, yeah, this section makes sense. So maybe we can just combine both of these like
30 00:06:25.440 ⇒ 00:06:27.620 Uttam Kumaran: documents into one basically.
31 00:06:28.510 ⇒ 00:06:36.020 Awaish Kumar: Okay, I. So I had these steps and basically that are going to.
32 00:06:37.250 ⇒ 00:06:40.320 Awaish Kumar: Basically, if I put all in one, it’s going to
33 00:06:40.550 ⇒ 00:06:45.260 Awaish Kumar: be a lot. That’s why I just created like this baseline in a separate page
34 00:06:45.370 ⇒ 00:06:47.449 Awaish Kumar: where it says, what is the current stage.
35 00:06:49.510 ⇒ 00:06:53.020 Awaish Kumar: what tools we are using? What? Where the gap is.
36 00:06:53.210 ⇒ 00:06:55.469 Awaish Kumar: And I actually think it’s.
37 00:06:55.470 ⇒ 00:06:59.569 Uttam Kumaran: Fine for it to all be in one cause. Then we’ll add a table of contents to the top.
38 00:07:00.260 ⇒ 00:07:01.040 Awaish Kumar: Okay.
39 00:07:03.690 ⇒ 00:07:10.530 Uttam Kumaran: Like, what do you think? Cause I I want us to. I want it to be able to just be like a working document, that that every everything’s in that one place.
40 00:07:10.850 ⇒ 00:07:14.219 Uttam Kumaran: you know versus the nested documents, because people are, I think, are just gonna miss it.
41 00:07:15.630 ⇒ 00:07:18.910 Awaish Kumar: Okay? Yeah. Then I can restructure it.
42 00:07:20.230 ⇒ 00:07:25.159 Awaish Kumar: And and put it in our in one view.
43 00:07:25.430 ⇒ 00:07:30.134 Uttam Kumaran: Yeah. And then at the top. What I’ll do is I’m just gonna create like a I’ll just add a
44 00:07:32.220 ⇒ 00:07:34.450 Uttam Kumaran: table of contents to the top here.
45 00:07:34.880 ⇒ 00:07:35.940 Uttam Kumaran: So that’s there.
46 00:07:35.940 ⇒ 00:07:36.500 Awaish Kumar: Okay.
47 00:07:38.955 ⇒ 00:07:43.820 Uttam Kumaran: Okay, so let’s go. Maybe let’s go through the the the core,
48 00:07:45.910 ⇒ 00:07:54.049 Uttam Kumaran: sort of items. And then let’s maybe let’s we can spend a little bit of time in linear and like sort of start to ticket some stuff out.
49 00:07:55.940 ⇒ 00:07:56.850 Awaish Kumar: You know.
50 00:07:57.599 ⇒ 00:08:06.159 Awaish Kumar: So this one is like it tells us what tools we are using and what visibility we have like.
51 00:08:06.280 ⇒ 00:08:14.030 Awaish Kumar: So for these tools, for example, their engineering tools. We have low visibility, because unless we go into the
52 00:08:15.161 ⇒ 00:08:22.750 Awaish Kumar: tool and check it, we don’t know if anything failed and delayed. Dbt. We did did try to add some tests.
53 00:08:22.930 ⇒ 00:08:25.739 Awaish Kumar: They have some limited coverage there and then
54 00:08:26.040 ⇒ 00:08:30.960 Awaish Kumar: for the dashboards also, we don’t have any checks right out.
55 00:08:32.770 ⇒ 00:08:34.880 Awaish Kumar: So this is going to state about
56 00:08:35.049 ⇒ 00:08:37.940 Awaish Kumar: same thing in a more descriptive way.
57 00:08:41.950 ⇒ 00:08:45.149 Awaish Kumar: And we we are saying like, what is the next step here?
58 00:08:45.870 ⇒ 00:08:51.759 Awaish Kumar: So we want to add observability tooling by layer. So we want to monitor
59 00:08:52.230 ⇒ 00:08:57.779 Awaish Kumar: the extraction layer, then Dbt, and then the dashboard layer as well.
60 00:08:58.500 ⇒ 00:09:05.750 Awaish Kumar: And how like what we want to mirror at every stages, want to see for freshness.
61 00:09:05.940 ⇒ 00:09:12.840 Awaish Kumar: see, look for failures, and then we want to measure. If if there’s any failure or
62 00:09:13.460 ⇒ 00:09:19.219 Awaish Kumar: some test. Fresh test fails, then how quickly we can respond.
63 00:09:21.650 ⇒ 00:09:23.430 Awaish Kumar: So then.
64 00:09:23.430 ⇒ 00:09:24.030 Uttam Kumaran: Yeah, yeah.
65 00:09:25.020 ⇒ 00:09:29.339 Awaish Kumar: Yeah, so that this is the thing you you are talking about, like.
66 00:09:29.640 ⇒ 00:09:31.880 Awaish Kumar: who is going to onboard? Right?
67 00:09:33.393 ⇒ 00:09:42.950 Awaish Kumar: So we have to define the data stewardship responsibilities which says, like, Okay, these DVD models or this dashboard
68 00:09:43.895 ⇒ 00:09:50.080 Awaish Kumar: is the the people person A is responsible to to look for
69 00:09:50.290 ⇒ 00:09:54.790 Awaish Kumar: failures, and Fisher stays for these models and these dashboards.
70 00:09:55.320 ⇒ 00:10:00.850 Awaish Kumar: So that’s why that’s how we are. We deliver the responsibilities.
71 00:10:09.890 ⇒ 00:10:11.240 Uttam Kumaran: So for this
72 00:10:13.620 ⇒ 00:10:15.850 Uttam Kumaran: One thing that I’m curious about is
73 00:10:16.200 ⇒ 00:10:21.694 Uttam Kumaran: is like, how this is actually going to get into the
74 00:10:25.640 ⇒ 00:10:31.680 Uttam Kumaran: like, how are how are we gonna actually move these logs into like snowflake? For example, to to do the measurement.
75 00:10:33.560 ⇒ 00:10:43.740 Awaish Kumar: And like, we’ve so like, if, like, what we are talking about here is like the
76 00:10:43.980 ⇒ 00:10:50.990 Awaish Kumar: if we are looking for observability, or we are want to mayor for responses.
77 00:10:51.310 ⇒ 00:10:52.670 Awaish Kumar: Incident, response.
78 00:10:56.230 ⇒ 00:10:57.692 Uttam Kumaran: Yeah, I guess.
79 00:11:01.070 ⇒ 00:11:02.839 Awaish Kumar: Like, so I guess.
80 00:11:02.840 ⇒ 00:11:06.460 Uttam Kumaran: Yeah, we we should set together like what the sla’s are.
81 00:11:06.590 ⇒ 00:11:07.760 Uttam Kumaran: And then
82 00:11:08.160 ⇒ 00:11:12.589 Uttam Kumaran: I do want to sort of get a baseline on on everything. But I’m just sort of like, what?
83 00:11:12.720 ⇒ 00:11:17.789 Uttam Kumaran: Trying to think of like, okay? Like, I agree with that. What’s the next like sort of thing we should try to do.
84 00:11:18.490 ⇒ 00:11:27.010 Awaish Kumar: Like like. So, for example, by layer like X. If we want to have observability on extraction layer.
85 00:11:27.120 ⇒ 00:11:35.650 Awaish Kumar: how we can achieve it. So previously in my job. How I did it was I, instead of triggering
86 00:11:37.430 ⇒ 00:11:47.560 Awaish Kumar: these runs on by like 5 trend runs by inside the 5 trend, I try to trigger it from, for example, airflow.
87 00:11:47.910 ⇒ 00:11:48.890 Awaish Kumar: So
88 00:11:49.080 ⇒ 00:11:56.260 Awaish Kumar: if that ran fails airflow will tell me, and it’s in maybe message in slack as well, that this
89 00:11:56.430 ⇒ 00:11:59.300 Awaish Kumar: extension run on 5, 10 failed.
90 00:12:04.470 ⇒ 00:12:05.769 Uttam Kumaran: I see. Okay.
91 00:12:05.960 ⇒ 00:12:07.970 Awaish Kumar: So we, this flow.
92 00:12:08.620 ⇒ 00:12:20.319 Awaish Kumar: this scheduling part isn’t at one place in airflow, so it runs the extraction part, then it will run the dbt. Part, then it will run the dashboard to refresh part.
93 00:12:21.990 ⇒ 00:12:26.940 Uttam Kumaran: So then I guess let’s let’s break this down. So then we have, like EL.
94 00:12:27.230 ⇒ 00:12:29.090 Uttam Kumaran: We have dbt.
95 00:12:31.870 ⇒ 00:12:34.809 Uttam Kumaran: We have dbt runs. We have dbt tests.
96 00:12:38.120 ⇒ 00:12:38.569 Awaish Kumar: I know we.
97 00:12:38.570 ⇒ 00:12:38.960 Uttam Kumaran: So.
98 00:12:38.960 ⇒ 00:12:40.240 Awaish Kumar: Same license.
99 00:12:40.970 ⇒ 00:12:42.390 Uttam Kumaran: And then dashboards.
100 00:12:42.940 ⇒ 00:12:46.950 Uttam Kumaran: So I think, basically, I want to break down into like, what is the next?
101 00:12:47.320 ⇒ 00:12:51.689 Uttam Kumaran: So for El, you’re basically going to like.
102 00:12:52.280 ⇒ 00:13:05.620 Uttam Kumaran: I guess what I’m what I’m thinking is like, you’re gonna have 5 train and polytomic have some sort of structured way of like sending in alerts somewhere on failure. Or basically, there’s always we just have a document that’s like how we’re gonna do. El related failures.
103 00:13:07.655 ⇒ 00:13:11.389 Awaish Kumar: Yeah, yes. So like.
104 00:13:11.520 ⇒ 00:13:20.649 Awaish Kumar: So one of the 2 ways. Either we have a single place which is going to trigger those runs in 5, 10, or polyatomic.
105 00:13:20.750 ⇒ 00:13:32.800 Awaish Kumar: Other way is that we integrate channel like the slack channels with 5 train. If there’s any failure, it’s going to respond there, and we define Sla document, which says, if there is any
106 00:13:33.160 ⇒ 00:13:41.454 Awaish Kumar: extraction, failure, like, it takes the priority of which, like urgent or whatever and
107 00:13:42.110 ⇒ 00:13:49.430 Awaish Kumar: and we maybe define some on call person based on days which is going to look for these values.
108 00:13:52.130 ⇒ 00:13:52.810 Uttam Kumaran: Okay?
109 00:14:03.650 ⇒ 00:14:05.450 Uttam Kumaran: And then, so okay.
110 00:14:06.960 ⇒ 00:14:12.530 Uttam Kumaran: so give me an example. So like, if polyatomic or run fails, they, they’re gonna send an email alert.
111 00:14:12.710 ⇒ 00:14:13.510 Uttam Kumaran: All right.
112 00:14:14.640 ⇒ 00:14:20.690 Uttam Kumaran: Polyatomic extract bails, right email or.
113 00:14:20.920 ⇒ 00:14:23.159 Awaish Kumar: Send yeah. Slack alerts.
114 00:14:24.100 ⇒ 00:14:28.680 Uttam Kumaran: Well, but this is the thing is like they may not. I don’t know if they have a slack integration.
115 00:14:30.210 ⇒ 00:14:31.010 Awaish Kumar: Okay.
116 00:14:31.010 ⇒ 00:14:32.739 Uttam Kumaran: But we should think about like
117 00:14:33.110 ⇒ 00:14:42.130 Uttam Kumaran: what the output is always gonna be right. Output is black, alert, to which channel right?
118 00:14:43.904 ⇒ 00:14:54.100 Uttam Kumaran: Like, who is immediate triage run book, right? Basically something like that.
119 00:14:58.570 ⇒ 00:15:03.890 Awaish Kumar: Yes, okay, so they they will send some alerts, and we will have
120 00:15:06.980 ⇒ 00:15:16.670 Awaish Kumar: a person who is going to look for those failures immediately and and like.
121 00:15:17.180 ⇒ 00:15:19.220 Awaish Kumar: if it is, it can be solved
122 00:15:20.090 ⇒ 00:15:23.309 Awaish Kumar: easily. We should solve it. And if there is.
123 00:15:23.430 ⇒ 00:15:26.330 Awaish Kumar: after some kind of testing, we figure out that like it’s a.
124 00:15:26.610 ⇒ 00:15:31.120 Awaish Kumar: it’s a major issue like we can create a follow. Follow up again.
125 00:15:33.470 ⇒ 00:15:34.130 Uttam Kumaran: Okay.
126 00:15:39.670 ⇒ 00:15:54.170 Awaish Kumar: Yeah, same. For like, DVD tests, like, if it, if a test failed, can it be easily resolved? Or is it like requires extra investigation or like. So how we have to create, follow up ticket and, like
127 00:15:54.480 ⇒ 00:15:57.760 Awaish Kumar: fine, spend some time on solving
128 00:15:58.200 ⇒ 00:16:00.450 Awaish Kumar: on an urgent issue, and then
129 00:16:00.720 ⇒ 00:16:05.469 Awaish Kumar: share the investigations and findings and then create a follow up ticket.
130 00:16:19.930 ⇒ 00:16:22.209 Uttam Kumaran: Okay, that makes sense.
131 00:16:22.390 ⇒ 00:16:29.130 Uttam Kumaran: But like. So then for Dbt run. Similarly, I think part of this is
132 00:16:33.980 ⇒ 00:16:37.126 Uttam Kumaran: so okay, so that makes sense, I think.
133 00:16:37.690 ⇒ 00:16:44.450 Uttam Kumaran: Another piece of this is just starting to measure El Dbt
134 00:16:44.620 ⇒ 00:16:48.439 Uttam Kumaran: failures in one central location right.
135 00:16:49.950 ⇒ 00:16:50.690 Awaish Kumar: Yeah, yeah.
136 00:16:57.770 ⇒ 00:16:59.699 Awaish Kumar: This is a good one.
137 00:17:06.069 ⇒ 00:17:09.059 Uttam Kumaran: begin documenting key data assets.
138 00:17:13.900 ⇒ 00:17:17.130 Awaish Kumar: Yeah, like, it’s kind of same. Last 2 points.
139 00:17:17.130 ⇒ 00:17:17.569 Uttam Kumaran: Yeah.
140 00:17:17.579 ⇒ 00:17:23.549 Awaish Kumar: We have the what data marks we have. Who’s going to own it? And what key dashboards?
141 00:17:23.949 ⇒ 00:17:25.429 Awaish Kumar: Who’s the owner
142 00:17:30.689 ⇒ 00:17:38.799 Awaish Kumar: like this is one of the ways like the thing we we talked about like.
143 00:17:39.379 ⇒ 00:17:43.625 Awaish Kumar: what if on call person like? So this is something
144 00:17:44.479 ⇒ 00:17:50.099 Awaish Kumar: data stewardship will normally do. They are responsible for, like
145 00:17:51.319 ⇒ 00:17:53.909 Awaish Kumar: 3, 5 models, 3 dashboards, and
146 00:17:54.109 ⇒ 00:17:57.769 Awaish Kumar: if if there’s any failure related to that, we are going to look for it.
147 00:18:01.430 ⇒ 00:18:06.560 Uttam Kumaran: Do you like do you think on call? Is the best strategy.
148 00:18:12.590 ⇒ 00:18:19.080 Awaish Kumar: Yeah, like, I, I mentioned 2 strategies here. One is on call. Second. One is this one
149 00:18:19.770 ⇒ 00:18:26.810 Awaish Kumar: defining data stewardship is that like, I’ve worked on product product sales summary. And I’m the now
150 00:18:27.020 ⇒ 00:18:29.609 Awaish Kumar: on an offend. So now, if any
151 00:18:30.380 ⇒ 00:18:34.340 Awaish Kumar: any failures or test failures related to that model.
152 00:18:35.340 ⇒ 00:18:44.949 Awaish Kumar: I’m the I’m I’m the one responsible to look and investigate that come like. That’s how they define
153 00:18:45.180 ⇒ 00:18:50.570 Awaish Kumar: data stewardships and and the companies like Uber Netflix.
154 00:18:51.250 ⇒ 00:18:58.339 Uttam Kumaran: Hmm, okay? So then get by the given like our given, like, you know, we
155 00:18:58.540 ⇒ 00:19:06.890 Uttam Kumaran: in our world like any, are we just gonna rotate on call like one person every week, basically or like, how do you think one person, every cycle, or what do you want to do.
156 00:19:13.230 ⇒ 00:19:21.020 Awaish Kumar: I think one person, if we go with on call strategy. I think one person on like
157 00:19:21.170 ⇒ 00:19:24.099 Awaish Kumar: we rotate it every day is is what I think.
158 00:19:27.670 ⇒ 00:19:28.680 Uttam Kumaran: Every day is a lot.
159 00:19:30.350 ⇒ 00:19:34.520 Awaish Kumar: Like we define it, like everyone chooses a day like of
160 00:19:34.720 ⇒ 00:19:40.060 Awaish Kumar: so like this gives them flexibility, like, if I know on Tuesdays, I might have to.
161 00:19:40.740 ⇒ 00:19:44.799 Awaish Kumar: Oh, work on flexible hours, and I
162 00:19:45.281 ⇒ 00:19:51.759 Awaish Kumar: so on. If I’m on a call on Monday, I have to be with my computer all the time
163 00:19:53.800 ⇒ 00:19:58.920 Awaish Kumar: and next days I am flexible to like move here and there my hours.
164 00:19:59.650 ⇒ 00:20:03.390 Awaish Kumar: But if I just set a full week, then yeah, we’re stuck for week.
165 00:20:05.380 ⇒ 00:20:10.199 Uttam Kumaran: But that’s the thing is like every doing it for every day.
166 00:20:13.650 ⇒ 00:20:17.113 Uttam Kumaran: Yeah, I don’t know. I guess this is a good question for everybody.
167 00:20:31.240 ⇒ 00:20:32.250 Uttam Kumaran: hmm!
168 00:20:37.530 ⇒ 00:20:40.379 Awaish Kumar: Yeah, that’s like, I’m telling what I,
169 00:20:41.600 ⇒ 00:20:48.039 Awaish Kumar: the studies I’ve worked in. Like I, I was working with a company, and we were building data
170 00:20:48.140 ⇒ 00:20:53.390 Awaish Kumar: web scrappers, and every day some skippers, Finn.
171 00:20:53.640 ⇒ 00:20:58.459 Awaish Kumar: And that’s how we have it. Have a strategy to like 2 person every day on call
172 00:20:59.260 ⇒ 00:21:06.630 Awaish Kumar: to look for web scrapping failures, and like they wrote it every day.
173 00:21:08.950 ⇒ 00:21:13.529 Uttam Kumaran: But that’s where it’s like, I I don’t know. I feel like that’s we’re since we’re not building products.
174 00:21:14.180 ⇒ 00:21:19.660 Uttam Kumaran: and we’re we’re we’re mainly just on one client. They’re only maybe, like 2 or 3 people on a given client. Right?
175 00:21:20.650 ⇒ 00:21:30.970 Uttam Kumaran: And the the goal of sort of on call is mainly just a triage like you may not be the person solving it, but you’re the your goal is to work with the Pm. To basically say, like.
176 00:21:31.150 ⇒ 00:21:36.420 Uttam Kumaran: what is the problem like who’s? And then basically like, do the investigation
177 00:21:36.990 ⇒ 00:21:43.210 Uttam Kumaran: that’s really the goal of the on call person. And then you can solve it if you can, but otherwise it just gets ticketed.
178 00:21:46.300 ⇒ 00:21:47.570 Uttam Kumaran: So
179 00:21:51.640 ⇒ 00:22:00.279 Uttam Kumaran: I don’t know. I I think we should be at least on a weekly basis, if not on a cycle like a 2 week cycle where someone’s on call for the cycle.
180 00:22:00.950 ⇒ 00:22:02.090 Uttam Kumaran: Because
181 00:22:02.380 ⇒ 00:22:07.480 Uttam Kumaran: what this helps is that if there’s an error, then everybody else knows that whoever’s on call is gonna handle it.
182 00:22:07.880 ⇒ 00:22:08.680 Uttam Kumaran: you know.
183 00:22:09.000 ⇒ 00:22:11.819 Uttam Kumaran: So it’s not like 10 people running into every problem.
184 00:22:15.820 ⇒ 00:22:22.380 Uttam Kumaran: But also we, it’s tough because we are sort of Async, so it’s a little bit difficult.
185 00:22:30.490 ⇒ 00:22:31.140 Awaish Kumar: Like.
186 00:22:31.340 ⇒ 00:22:39.479 Awaish Kumar: okay, yeah, that that’s okay. Also. Okay, like, if we can move weekly. But like, if if a client only has 1 80, then.
187 00:22:40.560 ⇒ 00:22:41.090 Uttam Kumaran: Yeah.
188 00:22:41.640 ⇒ 00:22:45.850 Awaish Kumar: He’s the in charge, like every second.
189 00:22:46.340 ⇒ 00:22:46.990 Uttam Kumaran: Yeah.
190 00:23:12.590 ⇒ 00:23:15.199 Uttam Kumaran: okay, I don’t know. Let’s think about it. More.
191 00:23:15.960 ⇒ 00:23:20.320 Uttam Kumaran: maybe. Okay. So I I hear that on the current landscape. So the next piece is
192 00:23:20.650 ⇒ 00:23:24.759 Uttam Kumaran: define observability, vision and principles.
193 00:23:26.940 ⇒ 00:23:29.069 Uttam Kumaran: Okay, so this is where, yeah, we have.
194 00:23:37.440 ⇒ 00:23:39.559 Awaish Kumar: Yeah, like this key dimension that
195 00:23:40.550 ⇒ 00:23:42.530 Awaish Kumar: in my, that we want to move.
196 00:23:52.640 ⇒ 00:23:57.010 Uttam Kumaran: I think out of these like this is a huge priority.
197 00:23:58.520 ⇒ 00:24:03.950 Uttam Kumaran: I think these 2 are like also a huge priority.
198 00:24:05.760 ⇒ 00:24:17.360 Uttam Kumaran: And this one, right, like I think we do a pretty good job of communication.
199 00:24:18.460 ⇒ 00:24:27.259 Uttam Kumaran: but and then the next. So I think these these 4, I really think, are super high. So tell me, like, is is meta plane, basically are like
200 00:24:28.620 ⇒ 00:24:32.400 Uttam Kumaran: our like goals are like the next step here.
201 00:24:34.410 ⇒ 00:24:41.110 Awaish Kumar: And yes, like so like, I’ve worked with only 2 tools right now, like foundational.
202 00:24:41.360 ⇒ 00:24:50.220 Awaish Kumar: which which doesn’t work for us. And then the Meta plane. But I also recently came across a tool called data tiff.
203 00:24:51.020 ⇒ 00:24:54.480 Awaish Kumar: Yeah, I know data diff by the data fold guys.
204 00:24:54.840 ⇒ 00:24:57.510 Awaish Kumar: I have not used it. But maybe this.
205 00:24:58.040 ⇒ 00:25:04.159 Awaish Kumar: this can help us with this, this impacts mailing impacts on Prs, I don’t know.
206 00:25:05.770 ⇒ 00:25:09.300 Awaish Kumar: Like Meta plan also helps us with source to
207 00:25:11.610 ⇒ 00:25:15.630 Awaish Kumar: destination checks. They they might help us with.
208 00:25:15.930 ⇒ 00:25:22.660 Awaish Kumar: Staging was just broad. But I’m not sure they they do this later, if for individual Prs.
209 00:25:24.610 ⇒ 00:25:28.219 Awaish Kumar: But yeah, like, Meta blend pretty much can help us with.
210 00:25:30.410 ⇒ 00:25:31.110 Uttam Kumaran: Hmm.
211 00:25:31.110 ⇒ 00:25:33.579 Awaish Kumar: With, like, you can see here
212 00:25:34.160 ⇒ 00:25:37.569 Awaish Kumar: 90% of the things we want to do right now.
213 00:25:54.010 ⇒ 00:26:00.000 Uttam Kumaran: Okay, I mean, why don’t like I don’t know how much data fold costs, but
214 00:26:00.470 ⇒ 00:26:05.450 Uttam Kumaran: I think we should just go with Meta. I think we should. Just. I’ll I can get us a demo
215 00:26:06.210 ⇒ 00:26:07.640 Uttam Kumaran: of data fold
216 00:26:12.290 ⇒ 00:26:15.950 Uttam Kumaran: But I think we should just move forward with Meta plane. For now.
217 00:26:16.340 ⇒ 00:26:19.650 Uttam Kumaran: just like cause. This is a problem right now. So
218 00:26:20.740 ⇒ 00:26:23.810 Uttam Kumaran: I don’t know. Like, what do you think? Or do you want to talk.
219 00:26:26.730 ⇒ 00:26:31.390 Awaish Kumar: I think, like I I would like love to explore like
220 00:26:32.510 ⇒ 00:26:35.449 Awaish Kumar: I think we should use metaplane for one client.
221 00:26:35.730 ⇒ 00:26:36.420 Uttam Kumaran: Okay.
222 00:26:36.420 ⇒ 00:26:39.790 Awaish Kumar: Completely and then make a decision.
223 00:26:42.420 ⇒ 00:26:46.870 Uttam Kumaran: Okay, so let’s enable Meta plane or
224 00:26:52.320 ⇒ 00:26:54.570 Uttam Kumaran: like. Which client do we want to try for.
225 00:26:56.940 ⇒ 00:27:03.710 Awaish Kumar: Yeah, like, I started off with Javi. But they are not active now, so we can go with maybe hidden.
226 00:27:04.350 ⇒ 00:27:06.730 Uttam Kumaran: Yeah, let’s try enable Meta plane for Eden.
227 00:27:13.200 ⇒ 00:27:16.710 Awaish Kumar: Yeah, so these were like main
228 00:27:17.220 ⇒ 00:27:21.719 Awaish Kumar: things, 2 points which which are going to help us with the implementation.
229 00:27:21.850 ⇒ 00:27:28.929 Awaish Kumar: Other ones are more like top level things like, how, what? What.
230 00:27:29.660 ⇒ 00:27:36.089 Awaish Kumar: which domains or data sets? We want to monitor, to identify which tools we are going to use
231 00:27:37.960 ⇒ 00:27:40.219 Awaish Kumar: and build some proof of concept.
232 00:27:40.480 ⇒ 00:27:43.560 Awaish Kumar: So yeah, with Meta plan, that’s what we are going to do.
233 00:27:46.870 ⇒ 00:27:47.540 Uttam Kumaran: Okay.
234 00:27:59.330 ⇒ 00:28:04.589 Uttam Kumaran: okay, so next steps is enable Meta plane for Eden. Decide on meta plane versus data. Fold
235 00:28:07.040 ⇒ 00:28:10.300 Uttam Kumaran: And then how are you going to like.
236 00:28:10.400 ⇒ 00:28:12.960 Uttam Kumaran: how are you going to get the data for each of these.
237 00:28:14.970 ⇒ 00:28:16.129 Awaish Kumar: On the data for what?
238 00:28:16.390 ⇒ 00:28:20.669 Uttam Kumaran: Like, are you gonna get data out of Meta plane to look at like
239 00:28:21.650 ⇒ 00:28:24.330 Uttam Kumaran: how often alerts are going off or.
240 00:28:31.370 ⇒ 00:28:32.939 Awaish Kumar: Like meta plan.
241 00:28:35.810 ⇒ 00:28:40.609 Uttam Kumaran: For example, if I wanna if I wanna ask, okay, across all of our clients like.
242 00:28:41.000 ⇒ 00:28:43.569 Uttam Kumaran: are our error? Is our error rate going down
243 00:28:43.930 ⇒ 00:28:48.990 Uttam Kumaran: like. That’s 1 kind of overall question that I want to start measuring. If we can.
244 00:28:50.110 ⇒ 00:28:55.570 Awaish Kumar: Yeah, but that we can measure like data. Meta plan shows how many incident occur
245 00:28:55.770 ⇒ 00:29:01.019 Awaish Kumar: like, how many tests failed this week or last month or this this kind of thing.
246 00:29:07.300 ⇒ 00:29:11.550 Awaish Kumar: So they have the history of failures.
247 00:29:49.010 ⇒ 00:29:49.470 Uttam Kumaran: Okay.
248 00:29:49.470 ⇒ 00:29:50.200 Awaish Kumar: No.
249 00:29:51.940 ⇒ 00:29:59.349 Uttam Kumaran: Okay. So I, I feel okay about this. Maybe just to quickly go through the next 3, because I know it’s a jump. So
250 00:29:59.680 ⇒ 00:30:02.749 Uttam Kumaran: identify a high impact tool. Yeah, so.
251 00:30:04.060 ⇒ 00:30:05.330 Awaish Kumar: Yeah, that’s what I.
252 00:30:06.430 ⇒ 00:30:09.310 Uttam Kumaran: Okay, that makes sense tooling strategy
253 00:30:11.020 ⇒ 00:30:15.130 Uttam Kumaran: that makes sense. Poc, around key use cases.
254 00:30:18.710 ⇒ 00:30:20.401 Uttam Kumaran: So I think out of
255 00:30:25.050 ⇒ 00:30:32.210 Uttam Kumaran: yeah, I mean, out of all these, I think the observability and the baseline is like, really, really important.
256 00:30:33.730 ⇒ 00:30:34.350 Awaish Kumar: Every week.
257 00:30:34.780 ⇒ 00:30:42.979 Uttam Kumaran: So I want to really prioritize anything around documentation, anything around fixing errors fast. I want to prioritize.
258 00:30:44.380 ⇒ 00:30:47.370 Awaish Kumar: So like we should go with Meta plan first.st
259 00:30:48.460 ⇒ 00:30:53.010 Uttam Kumaran: Yeah, I I mean, I don’t know. I just I just think that like we should just go with Meta plane.
260 00:30:55.210 ⇒ 00:30:55.890 Awaish Kumar: Okay.
261 00:30:57.250 ⇒ 00:30:58.720 Uttam Kumaran: Because this is just like.
262 00:30:59.410 ⇒ 00:31:05.939 Uttam Kumaran: Yeah, I think we should just go with them. I’ll get. I’ll I’ll send you the link you can book, a demo with datafold, and you can tell me what you think.
263 00:31:07.900 ⇒ 00:31:13.140 Uttam Kumaran: But yeah, the tooling strategy I’m less concerned with. And then
264 00:31:14.780 ⇒ 00:31:21.209 Uttam Kumaran: I really want to try to get some of some of this ticketed out. You know soon. So
265 00:31:21.660 ⇒ 00:31:25.349 Uttam Kumaran: ideally, I think if we can get everything into this one document,
266 00:31:27.370 ⇒ 00:31:31.100 Uttam Kumaran: like out of the out of the nested docs. And then
267 00:31:31.210 ⇒ 00:31:36.659 Uttam Kumaran: basically start to look at. Okay, what is our plan for the next 6 weeks?
268 00:31:36.960 ⇒ 00:31:39.689 Uttam Kumaran: And like, how can we just tick it out and start to.
269 00:31:40.300 ⇒ 00:31:42.479 Uttam Kumaran: you know, get get going on some of these.
270 00:31:43.520 ⇒ 00:31:44.599 Uttam Kumaran: So I think we’re closer.
271 00:31:47.210 ⇒ 00:31:49.569 Uttam Kumaran: Big thing is, I think, about the the on call.
272 00:31:49.960 ⇒ 00:31:51.639 Uttam Kumaran: Think about metaplane.
273 00:31:52.322 ⇒ 00:31:59.660 Uttam Kumaran: And then, yeah, I want to quickly run a proof of concept with like Eden. And then basically say, cool, we’re happy with this, how this looks like.
274 00:32:00.180 ⇒ 00:32:00.910 Uttam Kumaran: yeah.
275 00:32:03.920 ⇒ 00:32:11.610 Awaish Kumar: So and so, yeah, like, only thing which I’m really concerned about the the thing which I talked about like.
276 00:32:11.890 ⇒ 00:32:18.770 Awaish Kumar: So Meta plan helps helps with monitoring and testing.
277 00:32:18.910 ⇒ 00:32:20.570 Awaish Kumar: But how?
278 00:32:21.760 ⇒ 00:32:28.350 Awaish Kumar: How we are going to monitor extraction, layer failures, and the
279 00:32:30.860 ⇒ 00:32:32.719 Awaish Kumar: and the decision on on call.
280 00:32:36.540 ⇒ 00:32:41.670 Awaish Kumar: So, yeah, like, I think, then these things can. We can maybe look out after that.
281 00:32:43.670 ⇒ 00:32:45.819 Uttam Kumaran: Okay, yeah, I agree.
282 00:32:48.180 ⇒ 00:32:49.410 Awaish Kumar: Okay. Sure.
283 00:32:49.930 ⇒ 00:32:52.509 Uttam Kumaran: Okay, thanks, guys. I know we have the AI
284 00:32:52.800 ⇒ 00:32:57.490 Uttam Kumaran: meeting now, but I’ll I’ll spend some time later talking to some folks about this and the Async
285 00:32:57.680 ⇒ 00:33:06.060 Uttam Kumaran: call as well. And then, yeah, let’s keep pushing. So okay, thanks, Kyle, thanks.