Meeting Title: Redshift Monitoring Setup Sync Date: 2025-08-20 Meeting participants: Uttam Kumaran, Emily Giant
WEBVTT
1 00:00:32.689 ⇒ 00:00:33.560 Emily Giant: Hi!
2 00:00:33.730 ⇒ 00:00:46.180 Uttam Kumaran: Hello! Okay, so let’s… Keep going on this, so… yeah, so, so Redshift… Basically, it’s prop… is…
3 00:00:46.770 ⇒ 00:01:00.420 Uttam Kumaran: it’s gonna help us. So, for example, if, like, we were wanting… if we want to take, like, a new table, for example, in Redshift, let’s say Dim Calendar is a new one, I can actually go into, like, calendar date and, like.
4 00:01:01.400 ⇒ 00:01:04.840 Uttam Kumaran: Create a new monitor just on this one column.
5 00:01:04.980 ⇒ 00:01:14.869 Uttam Kumaran: Which can monitor several things. It can monitor, like, you can monitor a column, you can write custom SQL and monitor it. I think there’s other stuff, like…
6 00:01:15.920 ⇒ 00:01:21.540 Uttam Kumaran: group by… I don’t know, there’s a… there’s a… they have a lot of things that you can do, which is…
7 00:01:21.730 ⇒ 00:01:26.710 Uttam Kumaran: And this is, like, add source to target. I haven’t… I’ve only done, like, the basics, so….
8 00:01:26.710 ⇒ 00:01:27.210 Emily Giant: Hmm.
9 00:01:27.210 ⇒ 00:01:30.420 Uttam Kumaran: We could probably go a step deeper. …
10 00:01:30.650 ⇒ 00:01:38.270 Uttam Kumaran: But one thing that I wanted to share in Slack is that, we should start to see like…
11 00:01:38.600 ⇒ 00:01:40.699 Uttam Kumaran: We should start to see these…
12 00:01:40.960 ⇒ 00:01:43.220 Uttam Kumaran: alerts come in, and so for a.
13 00:01:43.220 ⇒ 00:01:43.620 Emily Giant: Yeah.
14 00:01:43.620 ⇒ 00:02:01.079 Uttam Kumaran: because the permissions stuff broke, these are now offline, which is great, because now I know that these aren’t working, and I can go update them. Additionally, we should start to see, like, both new issues come up here, and then we can ask the question immediately here.
15 00:02:01.220 ⇒ 00:02:08.689 Uttam Kumaran: … So that’s kind of, like, what I’m… what I’m hopeful for, is that we have a channel where
16 00:02:08.910 ⇒ 00:02:11.730 Uttam Kumaran: Alerts come in, and also, …
17 00:02:12.080 ⇒ 00:02:21.660 Uttam Kumaran: Ideally, the dbt jobs issues come through here, because it’s actually quite… it’s a little bit easier to triage here in Metaplane.
18 00:02:22.920 ⇒ 00:02:23.340 Uttam Kumaran: Also.
19 00:02:23.340 ⇒ 00:02:23.920 Emily Giant: Yeah, that’s….
20 00:02:23.920 ⇒ 00:02:31.230 Uttam Kumaran: We can do is auto-create linear tickets, like, for triage when an issue incident comes up, and create runbooks.
21 00:02:31.470 ⇒ 00:02:37.360 Uttam Kumaran: But I’m really, really… I think this is a good step in the right direction, especially given that
22 00:02:37.620 ⇒ 00:02:40.890 Uttam Kumaran: We’re gonna be able to start putting alerts on
23 00:02:41.080 ⇒ 00:02:50.640 Uttam Kumaran: a lot of core columns. Really, the biggest thing that I’m gonna try to push for is that all of our new MARTS models have the last
24 00:02:51.090 ⇒ 00:03:04.099 Uttam Kumaran: have a… ideally, like, have a last run at, and then also, most of them should have time series, which is, like, when is the la- when is the latest data updated at? …
25 00:03:06.080 ⇒ 00:03:10.330 Uttam Kumaran: And, like, I think those… at least if we can have those monitors for the top
26 00:03:11.130 ⇒ 00:03:13.340 Uttam Kumaran: you know, models, I think, like, we should….
27 00:03:13.340 ⇒ 00:03:13.830 Emily Giant: Yeah.
28 00:03:13.830 ⇒ 00:03:14.980 Uttam Kumaran: unstop.
29 00:03:16.270 ⇒ 00:03:27.400 Emily Giant: Yeah, I think this is great. I think one of the questions they might ask is, like, will they be in this channel? Will they have any responsibility when it comes to monitoring? Which the answer is yes.
30 00:03:27.520 ⇒ 00:03:31.359 Emily Giant: Probably no, but it’s more or less to…
31 00:03:31.530 ⇒ 00:03:34.670 Emily Giant: give them reassurance, right? Like, not to…
32 00:03:34.850 ⇒ 00:03:38.880 Emily Giant: Introduce anything that they would be monitoring.
33 00:03:39.610 ⇒ 00:03:41.990 Uttam Kumaran: Mmm, yeah, yeah, I agree.
34 00:03:42.960 ⇒ 00:03:48.789 Uttam Kumaran: Mainly, I just want to show them that, like, we’re now kind of looking at everything, there is a process.
35 00:03:49.120 ⇒ 00:03:52.120 Uttam Kumaran: And my hope is that we catch anything before they do.
36 00:03:52.540 ⇒ 00:04:00.559 Emily Giant: Yeah, I think they’re gonna dig it. They’ll really like this. I can just foresee them being like, wait, am I supposed to do something?
37 00:04:00.560 ⇒ 00:04:01.020 Uttam Kumaran: Okay.
38 00:04:01.020 ⇒ 00:04:02.620 Emily Giant: Yeah. Yeah.
39 00:04:02.620 ⇒ 00:04:03.680 Uttam Kumaran: Okay, okay.
40 00:04:05.130 ⇒ 00:04:07.039 Emily Giant: Alright, yeah, I think this is great.
41 00:04:07.180 ⇒ 00:04:10.150 Emily Giant: And I’m excited about it, this will make my life a lot easier.
42 00:04:10.150 ⇒ 00:04:12.250 Uttam Kumaran: Yeah, and I’ll invite you to this.
43 00:04:12.700 ⇒ 00:04:15.219 Uttam Kumaran: As well, so you can log in and poke around.
44 00:04:15.390 ⇒ 00:04:19.290 Uttam Kumaran: … But yeah, you should just see some…
45 00:04:19.560 ⇒ 00:04:24.400 Uttam Kumaran: this week will be kind of a bigger calibration week on this sort of software.
46 00:04:24.400 ⇒ 00:04:29.180 Emily Giant: Yeah. Yeah, I’m kind of hopeful that if you like it, and it seems to be working.
47 00:04:29.220 ⇒ 00:04:34.830 Uttam Kumaran: I’m happy to… I’ll probably make the pitch to Zach next week, and… See if we can… Yeah?
48 00:04:35.990 ⇒ 00:04:37.040 Emily Giant: That sounds great.
49 00:04:37.040 ⇒ 00:04:41.950 Uttam Kumaran: Tried out a lot of them, like, there’s a lot of, sort of, observability tools. We did a whole.
50 00:04:41.950 ⇒ 00:04:42.550 Emily Giant: That’s right.
51 00:04:42.550 ⇒ 00:04:45.110 Uttam Kumaran: Or we, like, looked into all of them, and these guys…
52 00:04:45.300 ⇒ 00:04:50.359 Uttam Kumaran: We’re just the most cost-effective and, like, had the most flexibility.
53 00:04:50.570 ⇒ 00:04:51.660 Uttam Kumaran: …
54 00:04:52.710 ⇒ 00:04:57.299 Uttam Kumaran: Yeah, I would say there’s a lot of other ones, but they’re really expensive, and they’re just not, like.
55 00:04:57.470 ⇒ 00:05:02.180 Uttam Kumaran: This one, as you… if you have it running over time, it gets smarter and smarter over time, so….
56 00:05:02.420 ⇒ 00:05:08.010 Emily Giant: Yeah, we just have a need for this. It’s, like, very apparent to me that, like, this could have…
57 00:05:08.210 ⇒ 00:05:16.300 Emily Giant: helped me, enormously over the last couple months. So, like, I don’t think Zach’s gonna give a lot of pushback when it comes to this particular tool.
58 00:05:16.300 ⇒ 00:05:17.210 Uttam Kumaran: Okay, okay.
59 00:05:17.540 ⇒ 00:05:18.160 Emily Giant: Yeah.
60 00:05:19.480 ⇒ 00:05:19.980 Uttam Kumaran: Okay.
61 00:05:19.980 ⇒ 00:05:22.810 Emily Giant: Well, I’m excited, I think they’ll be really excited, too.
62 00:05:22.970 ⇒ 00:05:24.389 Uttam Kumaran: Okay, okay, awesome.
63 00:05:24.390 ⇒ 00:05:26.400 Emily Giant: Cool, alright, thank you.
64 00:05:26.400 ⇒ 00:05:27.789 Uttam Kumaran: Thank you. I’ll talk to you soon.
65 00:05:27.790 ⇒ 00:05:29.049 Emily Giant: Bye, talk to you soon.