Meeting Title: Data Platform Weekly Date: 2025-04-14 Meeting participants: Annie Yu, Uttam Kumaran, Amber Lin, Ryan Luke, Demilade Agboola, Awaish Kumar, Ryan, Caio Velasco


WEBVTT

1 00:02:19.690 00:02:20.650 Demilade Agboola: Alright guys.

2 00:02:21.910 00:02:25.300 Uttam Kumaran: Hey, hey, everyone

3 00:02:28.320 00:02:33.203 Uttam Kumaran: cool? I think this is the crew, hey? I know. I think Luke is out. He has

4 00:02:33.690 00:02:40.609 Uttam Kumaran: a family thing today. But yeah, this is sort of the data platform

5 00:02:40.750 00:02:49.310 Uttam Kumaran: squad. I basically wanted to just bring everyone, you know, that’s internal on our data team together. I know

6 00:02:49.630 00:02:53.040 Uttam Kumaran: each of us probably spend time more heads down

7 00:02:53.597 00:02:58.889 Uttam Kumaran: on client work, but I wanted to make sure that we all talk.

8 00:02:59.200 00:03:04.009 Uttam Kumaran: you know once a week, and find ways to dedicate a little bit of time towards just like

9 00:03:04.160 00:03:06.368 Uttam Kumaran: our platform as a whole.

10 00:03:07.020 00:03:13.960 Uttam Kumaran: What do you? What do I mean by platform? So this is where like decisions around

11 00:03:14.080 00:03:17.540 Uttam Kumaran: what tools should we use? How do we do? Documentation?

12 00:03:20.712 00:03:24.360 Uttam Kumaran: Things like like?

13 00:03:25.484 00:03:29.889 Uttam Kumaran: How Dbt should run! How snowflake should be structured! Those are all decisions that

14 00:03:30.440 00:03:53.849 Uttam Kumaran: previously I was like a data platform team of one. So I made a lot of those decisions. But now that we all have spent, you know, around a month working in our system. You all know the pros and cons of a lot of those decisions. For me, instead of going in thinking through what’s wrong, what’s working and sort of applying that I want to loop everybody in

15 00:03:54.490 00:04:00.840 Uttam Kumaran: different than our clients. However, this team can move a little bit more methodically.

16 00:04:01.798 00:04:15.709 Uttam Kumaran: And a little bit more patiently, I think for this squad we’re working on, basically enabling how all enabling all of us to work more efficiently with less stress.

17 00:04:16.166 00:04:22.749 Uttam Kumaran: And so I think a little bit about how we can enable that the other piece I think about

18 00:04:23.288 00:04:35.130 Uttam Kumaran: is around, how we can continue to deliver more for our clients and have them have more confidence in the sort of work that we’re we’re developing. So

19 00:04:36.970 00:04:43.880 Uttam Kumaran: yeah, maybe any questions or thoughts there and then I have a little bit of a doc to go over. But

20 00:04:45.830 00:04:52.050 Uttam Kumaran: yeah, I guess any thoughts or anyone have questions on, sort of like, what the goal of this

21 00:04:53.170 00:04:54.919 Uttam Kumaran: this team is.

22 00:05:01.530 00:05:03.709 Amber Lin: So to clarify. This is

23 00:05:04.296 00:05:10.820 Amber Lin: this is not an internal project with developing. It’s more of enabling everyone to

24 00:05:11.120 00:05:17.669 Amber Lin: use the tools and sort of consolidate our ideas on how to work more efficiently is that the goal.

25 00:05:19.127 00:05:33.830 Uttam Kumaran: This is sort of like you think about what is a data platform. So data platform is everything that our data folks use to do their job, whether it’s like our, how our tickets structured. It’s everything from like, how.

26 00:05:33.960 00:05:39.220 Uttam Kumaran: what kind of work do we take on, and how our tickets structured all the way to how do we structure? Dbt.

27 00:05:39.410 00:05:41.749 Uttam Kumaran: well, how do we do documentation

28 00:05:41.850 00:05:44.680 Uttam Kumaran: like, how do we run? Pr process?

29 00:05:44.790 00:05:51.060 Uttam Kumaran: Those are all like things that are cross client, like decisions we have to make.

30 00:05:51.180 00:05:59.019 Uttam Kumaran: And so for me, I’ve made a lot of those decisions historically, just on, based on my past, but now we are but

31 00:05:59.260 00:06:10.259 Uttam Kumaran: I previously I I’ve sort of considered this running like how our other teams, but we are interesting where we have several clients we’re supporting. And then this team really just supports all of us.

32 00:06:10.901 00:06:19.299 Uttam Kumaran: Right? So this can be. This can be things about how we structure a dbt, how? Yeah? Basically like process standardization. Exactly.

33 00:06:24.350 00:06:31.489 Uttam Kumaran: So I feel like for. But for me, what I know is that like, I want to make sure everybody’s involved in making these some of these decisions. Because then we can go test.

34 00:06:32.320 00:06:37.559 Uttam Kumaran: We can go test some of these out and make sure that they’re actually like, they’re actually working.

35 00:06:38.056 00:06:41.730 Uttam Kumaran: My lens for all this is starting from the clients. Right? So

36 00:06:42.160 00:06:45.930 Uttam Kumaran: is the client happy? Why or why not? Why not, and how much of that is.

37 00:06:46.487 00:06:49.340 Uttam Kumaran: How much can we solve as a platform team?

38 00:06:49.480 00:06:56.859 Uttam Kumaran: Additionally, the second most important pieces are all of we are all of us when we go and develop a piece of

39 00:06:57.544 00:07:00.459 Uttam Kumaran: when we go and act on a ticket.

40 00:07:01.060 00:07:06.699 Uttam Kumaran: Is that clear? Do we end up succeeding? How often do we succeed, and what else could we have done to support that?

41 00:07:14.470 00:07:19.820 Uttam Kumaran: I know I’ve talked to a wish I talked to Kyle a little bit about this, but any other. Any other folks have

42 00:07:20.000 00:07:22.059 Uttam Kumaran: have questions. Otherwise I can sort of

43 00:07:22.970 00:07:27.427 Uttam Kumaran: maybe sort of lead a little bit of a discussion around a couple of things, and then

44 00:07:28.270 00:07:31.429 Uttam Kumaran: and we could just talk about how we want to action on some items.

45 00:07:42.560 00:07:51.505 Uttam Kumaran: Okay, cool. Well, if not, I sent this note. I sent this link in in the zoom chat. And I’ll send it also here in

46 00:07:52.700 00:07:58.720 Uttam Kumaran: in slack basically.

47 00:07:58.980 00:08:04.639 Uttam Kumaran: You know, I work. I worked with AI. And I sort of came up with some of these questions just to sort of allow us to

48 00:08:04.740 00:08:09.709 Uttam Kumaran: think about what what should be on our mind as a data platform team.

49 00:08:11.410 00:08:15.580 Uttam Kumaran: I think, like a good way of doing this is to sort of like, go around the Horn, and

50 00:08:15.720 00:08:23.499 Uttam Kumaran: if anyone has like. Okay, if I could wave a wand and fix one thing about our platform or the way we do things, what what would it be

51 00:08:23.996 00:08:28.983 Uttam Kumaran: you know I I would love for this meeting to be really, really collaborative.

52 00:08:29.850 00:08:38.429 Uttam Kumaran: so I don’t know. Does that make sense? Does anyone have any any other ideas on, you know, ways we can make this session more productive before we kick off.

53 00:08:39.556 00:08:48.520 Awaish Kumar: Yeah, like one of the thing which I have been thinking about streamlining our data pipeline so.

54 00:08:48.680 00:08:55.749 Awaish Kumar: and observability of our data pipeline. So one of the connector in Javi like for Amazon.

55 00:08:56.447 00:08:59.019 Awaish Kumar: It has been delayed for 5 h.

56 00:08:59.210 00:09:03.350 Awaish Kumar: So right now we don’t have a way to find out that.

57 00:09:03.620 00:09:05.910 Awaish Kumar: And like, we have.

58 00:09:06.030 00:09:17.509 Awaish Kumar: if multiple tools which we are using like portable polytomic, 5 trend and on some like a connector on 5 trend is delayed, and we we don’t have a

59 00:09:18.414 00:09:28.410 Awaish Kumar: observability on that like, if that is finished, we just run our Dbt models without our raw data is refreshed.

60 00:09:29.204 00:09:34.799 Awaish Kumar: Like out, like. One of the thing we could do is that like we could introduce one of the

61 00:09:35.060 00:09:39.930 Awaish Kumar: orchestrator like airflow, where we basically build the workflows.

62 00:09:40.100 00:09:49.090 Awaish Kumar: So like instead of like waiting on timing like a A connector on 5 trend finishes.

63 00:09:49.320 00:09:53.600 Awaish Kumar: And then we run our Dpd models like, we can basically set up that

64 00:09:53.710 00:10:12.480 Awaish Kumar: a every time a sync runs and and it succeeds, we run the downstream tasks, and we have a observability that we. If if any connector is delayed, we get a notification on slack. The data for this connector is delayed for some time or whatever, and if anything fails. Also, we get that

65 00:10:13.130 00:10:20.589 Awaish Kumar: I mean that way, like we are on top of things. If anything fail. For example, North Beam has been failing for some time

66 00:10:20.820 00:10:24.080 Awaish Kumar: on portable for Javi also. So yeah.

67 00:10:29.270 00:10:34.250 Uttam Kumaran: Okay, so definitely. So there’s a big piece around data source observability, monitoring.

68 00:10:35.970 00:10:46.760 Awaish Kumar: Yeah, like, and the like, the what to say, like creating tags like a workflow where

69 00:10:47.240 00:10:53.100 Awaish Kumar: we we are. We are more moving towards, like kind of

70 00:10:54.960 00:10:57.500 Awaish Kumar: if on based on events. Like, if a

71 00:10:57.970 00:11:01.849 Awaish Kumar: sync succeeds, we run downstream tasks

72 00:11:02.000 00:11:19.090 Awaish Kumar: right now, like I’m I’m just guessing that if like we now move for Javi, we move to 6 h synchronization. So I’m guessing on time that okay, if if a run started on 4 Am. It might finish by 6. Then I have to run dbt.

73 00:11:19.627 00:11:26.139 Awaish Kumar: to to get the data into our model. Mars. So yeah, that can be like automated.

74 00:11:26.490 00:11:32.910 Awaish Kumar: And also we we can save. We don’t have to run Dbt from downstream down like the upstream task is failed.

75 00:11:35.930 00:11:36.580 Uttam Kumaran: Okay.

76 00:11:42.080 00:11:44.560 Uttam Kumaran: yeah. And I think maybe what we’ll do, I think

77 00:11:45.258 00:11:50.729 Uttam Kumaran: amber is gonna pull this together right now is, maybe we’ll just do something in fig jam, because I know this document.

78 00:11:51.191 00:11:55.890 Uttam Kumaran: There’s a lot of questions. So maybe. But I, basically, anyone else want to sort of talk about

79 00:11:56.430 00:12:00.050 Uttam Kumaran: our platform as a whole, and like what works? Sort of what sucks

80 00:12:02.910 00:12:07.093 Uttam Kumaran: Or if there’s something in particular out of these categories that that sort of

81 00:12:08.320 00:12:12.379 Uttam Kumaran: you’re like, okay, if this, if if we’re able to solve this, this will be a huge help.

82 00:12:17.420 00:12:21.209 Awaish Kumar: Yeah. And and one more thing on this, like we talked about

83 00:12:21.460 00:12:26.510 Awaish Kumar: that we want to move away from the Github actions to running our DVD models

84 00:12:26.620 00:12:31.379 Awaish Kumar: somewhere else, so that this will also solve this problem, that

85 00:12:31.799 00:12:35.630 Awaish Kumar: we will be running our models on some of our own machines

86 00:12:35.730 00:12:38.350 Awaish Kumar: instead of relying on Github actions.

87 00:12:42.930 00:12:44.340 Uttam Kumaran: Okay. I’ll note that down.

88 00:12:57.732 00:13:02.380 Demilade Agboola: Is is the focus only on like data modeling, like the

89 00:13:02.810 00:13:12.799 Demilade Agboola: everything from ingestion to the warehouse, because sometimes some things also happen. You know, after the warehouse, in terms of like the reports themselves.

90 00:13:13.280 00:13:14.140 Demilade Agboola: is Dabio.

91 00:13:14.140 00:13:21.659 Uttam Kumaran: No, yeah, yeah. I think the I think the way to frame what’s in focus here is the problem has to be cross client.

92 00:13:22.660 00:13:25.660 Uttam Kumaran: I think everything anything that’s like

93 00:13:25.910 00:13:29.350 Uttam Kumaran: is potentially cross. Client is up for debate on.

94 00:13:29.630 00:13:33.820 Uttam Kumaran: But I think it’s worth worth shouting out the Prom, and then we’ll see like whether?

95 00:13:35.010 00:13:45.050 Uttam Kumaran: yeah. But I I think it end to end. But not only just like it happens at data. But then, if there’s something that happens on the business side, or is relate to communication. Everything’s up up for grabs.

96 00:14:00.100 00:14:10.120 Caio Velasco: So one thing that I could say I think, would be even repetitive, because I remember that I mentioned something like a few weeks ago when I started is that

97 00:14:11.000 00:14:13.760 Caio Velasco: I saw that it was easier to

98 00:14:14.520 00:14:39.579 Caio Velasco: build something on the engineering side when we had a centralized place for either the client or the Po, or whoever is talking to the client to just go and say like, Hey, these are. This is my similar to what Akash is doing actually every time he shares his excel. And it’s very similar to that. Something simple like that. We have knowledge about everything that is happening.

99 00:14:40.020 00:14:49.229 Caio Velasco: or everything that needs to be done, or new requests. And you know, if they have any question regarding a data model or a new metric, or a new change, or

100 00:14:49.380 00:14:52.850 Caio Velasco: it’s centralized. So you go there. You want you you check.

101 00:14:53.100 00:14:58.770 Caio Velasco: And well, of course, you always have to make sure that they are enough

102 00:14:59.190 00:15:13.776 Caio Velasco: information there. For example, when when I did the Cloud view the 1st version of the Cloud View data model, I had a ticket. It was like there was some information in there which was enough to begin with.

103 00:15:14.850 00:15:20.139 Caio Velasco: But then I I saw that now there are some other tickets to, I think to any

104 00:15:20.710 00:15:23.740 Caio Velasco: on top of that, because I think she has to build a

105 00:15:24.080 00:15:26.370 Caio Velasco: dashboard or something, or 1st mock.

106 00:15:26.500 00:15:30.570 Caio Velasco: and there was so much more information there than I could have used this before.

107 00:15:30.800 00:15:38.689 Caio Velasco: so somehow integrating that whenever something in start starting, I think, guides the whole process.

108 00:15:38.890 00:15:41.060 Caio Velasco: That’s my my view.

109 00:15:43.470 00:15:55.789 Uttam Kumaran: Perfect. Yeah. And and how is feedback on like this? I know we developed this a while ago. I sort of am, you know, I don’t. Wanna. I don’t wanna solutionize too much in this call more. I just wanna hear.

110 00:15:56.270 00:15:59.069 Uttam Kumaran: All the problems. So we can sort of prioritize. But

111 00:15:59.642 00:16:20.960 Uttam Kumaran: you know, for me in this document like, I do like that. And this isn’t. Probably this is something that we would need to fill out. But we have, like sort of all our data sources, all. And then the dashboard and the metrics. I don’t know. I feel like it’s maybe a little bit too deep. But this is sort of a core metrics where this is filled out by the Pm.

112 00:16:23.290 00:16:30.529 Uttam Kumaran: And I kind of like this a little bit more. But does that? Is anyone like have it have any like

113 00:16:31.230 00:16:37.239 Uttam Kumaran: opinions on on this. And maybe there’s some low hanging fruit decisions we can make on just like.

114 00:16:37.380 00:16:41.909 Uttam Kumaran: okay, every client needs a Google sheet like this with

115 00:16:42.050 00:16:46.869 Uttam Kumaran: at least core metrics, data sources and like information about

116 00:16:47.620 00:16:50.440 Uttam Kumaran: the data platform and the costs and things like that.

117 00:16:57.080 00:17:03.059 Ryan Luke: I think this works, but I think it needs to be improved somehow. I’m I’m just not sure how, yet

118 00:17:03.290 00:17:06.210 Ryan Luke: like having it in sheets.

119 00:17:07.640 00:17:10.620 Ryan Luke: isn’t, I guess, the best way to do it.

120 00:17:11.089 00:17:16.679 Demilade Agboola: Yeah, I also feel like this, there are ways definitely ways we should be able to automate this. I feel this is a bit.

121 00:17:16.680 00:17:17.030 Uttam Kumaran: Yeah.

122 00:17:17.400 00:17:21.559 Demilade Agboola: And it’s 2 words we can try and make it a visual process.

123 00:17:22.124 00:17:30.745 Demilade Agboola: I maybe we could look into Dvt and like column column, lineage, and kind of see how that helps as well.

124 00:17:32.270 00:17:35.300 Demilade Agboola: And also, I think he also relies on

125 00:17:35.400 00:17:38.209 Demilade Agboola: the constant updating in this asset.

126 00:17:38.210 00:17:38.970 Uttam Kumaran: Yes.

127 00:17:38.970 00:17:44.889 Demilade Agboola: Make a change. You have to come back here, and you know, remember to do that. I think that can be very, you know, tiresome.

128 00:17:46.260 00:17:49.370 Uttam Kumaran: I agree. I mean, that’s my perspective on like

129 00:17:50.610 00:17:57.090 Uttam Kumaran: that’s that’s a huge piece of my perspective. Is that like I’d maintaining, this is harder than creating it.

130 00:17:58.220 00:18:02.587 Uttam Kumaran: And so I try to think a little bit about like what we can do there.

131 00:18:03.150 00:18:13.600 Uttam Kumaran: cause cause. Okay? So if you take a look at like, if you take a look at that document, right? At least the data source documentation and the data platform. These don’t change that frequently.

132 00:18:13.960 00:18:18.300 Uttam Kumaran: Right. This is just the tools we’re using. Who the owner is, how much it costs.

133 00:18:18.710 00:18:22.589 Uttam Kumaran: This is definitely a little bit more rough to maintain.

134 00:18:25.390 00:18:30.549 Uttam Kumaran: this is where we can try some stuff with AI to maybe take in the repo and write some of this.

135 00:18:33.520 00:18:40.399 Uttam Kumaran: But like, yeah, the the we tried some stuff like this. It’s just really, it’s really hard to maintain.

136 00:18:49.660 00:18:55.210 Uttam Kumaran: But certainly, like, if we talk a little bit further on documentation

137 00:18:55.780 00:19:02.790 Uttam Kumaran: one is, I think we have multiple pieces of like documentation. Right? So we have like pipes.

138 00:19:03.050 00:19:18.120 Uttam Kumaran: We have like looms. We have github code. We have like our just our like client documentation.

139 00:19:18.590 00:19:23.288 Uttam Kumaran: right? Like one of the things that we are trying to do for some clients. Is this like

140 00:19:24.180 00:19:29.229 Uttam Kumaran: client documentation, like, for example.

141 00:19:29.780 00:19:35.589 Uttam Kumaran: for stack Blitz. What we did is we? We just work through an FAQ process where we just have

142 00:19:36.130 00:19:38.770 Uttam Kumaran: question, answer, question, answer, question, answer.

143 00:19:41.580 00:19:46.640 Uttam Kumaran: Something like this, I think, is easy for AI to sort of keep updated and maintain.

144 00:19:47.070 00:19:52.549 Uttam Kumaran: It’s also easy for people to just go to one place and see all the information about different topics.

145 00:19:53.835 00:19:55.959 Uttam Kumaran: How do we feel about something like this?

146 00:19:56.550 00:20:03.170 Uttam Kumaran: Just one document with like, sort of all this information that we keep updated. But this isn’t going to be as specific as

147 00:20:03.680 00:20:09.249 Uttam Kumaran: we just need multiple types. Right? So, but as yeah, I don’t know, I’m trying to think through what the best option is here.

148 00:20:12.540 00:20:14.442 Ryan: Yeah, I think this is great.

149 00:20:14.970 00:20:21.450 Ryan: but I I definitely think this needs to happen before we start making the models. Because, like.

150 00:20:22.320 00:20:27.284 Ryan: yeah, we we just created this after the fact. Right? So. But like,

151 00:20:28.620 00:20:34.060 Ryan: if we have this before creating the models that way, we have context on point. And like more

152 00:20:36.190 00:20:45.170 Ryan: like we, we get to know what the actual business logic it is for some of the metrics like what total sales means and stuff like that.

153 00:20:45.700 00:20:46.380 Uttam Kumaran: Okay.

154 00:20:48.860 00:20:53.530 Ryan: That’s just my yeah, my-, my phone, just based on like

155 00:20:53.860 00:20:58.900 Ryan: what I’m experiencing at the moment for pool parts and stack bits, for example like that.

156 00:20:59.380 00:21:02.509 Ryan: Do our Doc our models first, st and then

157 00:21:02.970 00:21:05.050 Ryan: share it with a client? And then the

158 00:21:06.210 00:21:11.469 Ryan: they did. They don’t really know, or like we. We didn’t really know what the actual business logic is.

159 00:21:11.630 00:21:17.140 Ryan: so maybe it would be great to have the business logic set in place first.st

160 00:21:17.910 00:21:18.580 Uttam Kumaran: Okay.

161 00:21:32.900 00:21:35.569 Uttam Kumaran: Annie, or demalade. Any thoughts.

162 00:21:44.490 00:21:46.480 Demilade Agboola: I think also.

163 00:21:49.520 00:21:56.419 Demilade Agboola: we could also rely on Dvt. Docs in the sense that Dvc. Allows us to do Dvt. Docs generate.

164 00:21:56.710 00:21:59.779 Demilade Agboola: which allows, like a local HTML, file.

165 00:22:00.790 00:22:06.610 Demilade Agboola: and if you have Dbt cloud, it actually does create like a web page to allow you to actually view

166 00:22:07.382 00:22:16.799 Demilade Agboola: the actual dbt, documentation of the flow as well, and I think the advantage of that is that can be a bit more

167 00:22:17.660 00:22:23.489 Demilade Agboola: that can be easier to maintain. Because, like, you’re just working within, like the Dvc space, make those changes.

168 00:22:23.700 00:22:27.830 Demilade Agboola: We write about the changes and it reflects in the Dpc Docs.

169 00:22:28.542 00:22:35.150 Demilade Agboola: potentially, Dbt also allows like Dbt exposure. So we can also say what

170 00:22:35.490 00:22:46.699 Demilade Agboola: we can also explicitly put what the marks are feeding like, what dashboards they are feeding. So now we have a complete view from sources all the way, till.

171 00:22:46.880 00:22:51.600 Demilade Agboola: the dashboards being used, we can have a view of everything in one place.

172 00:22:54.140 00:22:54.660 Ryan: So.

173 00:22:54.660 00:22:55.190 Uttam Kumaran: Yeah, if I.

174 00:22:55.190 00:23:00.109 Ryan: It’s worth like exploring Dbt. Cloud as another option.

175 00:23:01.070 00:23:02.469 Uttam Kumaran: Yeah, I think my

176 00:23:03.010 00:23:09.290 Uttam Kumaran: my pushback there is that I I just would be very surprised if, like it works for our clients

177 00:23:09.780 00:23:15.509 Uttam Kumaran: like, you know, I considered using Dbt. Cloud and the documentation there. But

178 00:23:15.930 00:23:20.240 Uttam Kumaran: I think there’s 2 pieces that I I guess I would be concerned about. One is

179 00:23:20.460 00:23:26.950 Uttam Kumaran: like, how much effort is it for us to maintain like column level definition everywhere?

180 00:23:27.710 00:23:29.689 Uttam Kumaran: Second piece is like our.

181 00:23:29.810 00:23:37.790 Uttam Kumaran: you know. Think about. Think about a client. Think about one of your stakeholders on one of your clients like, are they gonna log into Dbt cloud when they have a question about a metric?

182 00:23:38.320 00:23:45.590 Uttam Kumaran: I think they may just ask. They may just like ask in slack, right? So what is our option at that point to point them to something?

183 00:23:47.650 00:23:54.299 Uttam Kumaran: you know I I don’t think I I don’t. You know. One thing I’ve learned is, I don’t think a single client is going to go into Github and look

184 00:23:54.860 00:23:59.189 Uttam Kumaran: so certainly like I don’t know. I have a little bit of an aversion to

185 00:23:59.660 00:24:02.670 Uttam Kumaran: referencing anything that has to live as code.

186 00:24:02.980 00:24:10.050 Uttam Kumaran: although your point about maintenance is true, but that would be my sort of push back there.

187 00:24:16.710 00:24:22.400 Uttam Kumaran: But I hear you. I guess there’s there needs to be some solution that’s ex for me. It’s like accessibility, right? Like

188 00:24:23.310 00:24:29.540 Uttam Kumaran: that. That sort of FAQ. Doc. That I shared is really accessible like a Pm. Can go edit it right

189 00:24:30.050 00:24:37.560 Uttam Kumaran: anyone on the client can side can edit it. Any of our engineers can edit it. We can have eventually try to have AI sort of maintain part of it.

190 00:24:38.250 00:24:43.090 Uttam Kumaran: So it seems like a a good non code medium, you know.

191 00:24:45.710 00:24:48.320 Demilade Agboola: Definitely. I agree. I think also, there’s

192 00:24:49.680 00:24:58.400 Demilade Agboola: so I know it’s I know it’s not free. But I I there’s data. Hub. I know data hub was something I’ve used in the previous company, and it’s pretty was pretty helpful to have

193 00:24:59.155 00:25:01.229 Demilade Agboola: things in one space. Also.

194 00:25:01.230 00:25:01.820 Uttam Kumaran: Okay.

195 00:25:02.320 00:25:07.510 Demilade Agboola: Actually, being able to have.

196 00:25:08.170 00:25:23.849 Demilade Agboola: So okay, so like the way data hub also works is like a Dvc docs will persist into data hub. So all of a sudden, you now have like this place where, like, you can see the entire lineage, and you can also see the documentations in like easily readable format.

197 00:25:24.657 00:25:31.409 Demilade Agboola: And you. You can also add things like, I know, when we had, like us, a team where we’re maintaining

198 00:25:31.640 00:25:40.899 Demilade Agboola: the entire Dbt space, we’re able to add things like the owners of different models. So like, say, Hey, if this model breaks.

199 00:25:41.060 00:25:45.779 Demilade Agboola: the the person responsible for this model is this or the person who made the last change? Was this.

200 00:25:45.880 00:25:48.550 Demilade Agboola: so that’s also something that could also like

201 00:25:49.180 00:25:59.749 Demilade Agboola: we we like we did. But again, that requires, like a dedicated team, and like people who are like on it consistently. But what I was going to also was that

202 00:26:00.740 00:26:06.039 Demilade Agboola: if we have, as if we have a lot of context within our Dvt space.

203 00:26:06.170 00:26:26.040 Demilade Agboola: it’s, I think, at that point it’s easier to have, like an AI model that can then read the entire space and answer questions. It might not always be 100% accurate, but it can start answering things about in what tables, for instance, or in what matters are we using this column.

204 00:26:26.170 00:26:27.460 Demilade Agboola: or in what matter

205 00:26:27.570 00:26:33.470 Demilade Agboola: and or like? Does this exist in this table? Does this exist in that table? What is it.

206 00:26:33.470 00:26:46.269 Uttam Kumaran: Exactly. Yeah, I I totally agree. And so one of the things that I that we are working on the AI team is doing that because part of the point of documentation is that it’s used when the question is asked

207 00:26:46.420 00:26:53.279 Uttam Kumaran: right. But what if you would just ask the AI, then the the need for documentation just goes as a way to like

208 00:26:53.470 00:27:02.479 Uttam Kumaran: organize it. But for the most part, like people don’t care whether they have to go into notion. In fact, that’s the barrier. Right is actually going to the right place.

209 00:27:02.810 00:27:14.809 Uttam Kumaran: Can we agree on that right, like going to the right place, knowing where it is is basically all this stuff we’re trying to solve here. And so part of what I, what I was experimenting with is, there’s this library called repo. Mix.

210 00:27:15.230 00:27:18.429 Uttam Kumaran: It just takes the entire repo and puts it into one file.

211 00:27:18.590 00:27:23.800 Uttam Kumaran: And so I actually wrote a workflow that does this for the ABC home client.

212 00:27:24.392 00:27:27.530 Uttam Kumaran: We’re basically anytime a new Pr gets merged.

213 00:27:28.260 00:27:32.400 Uttam Kumaran: It puts the entire repo into one large file.

214 00:27:32.640 00:27:39.770 Uttam Kumaran: And this is exactly it. So the way it works is it creates A. It creates a a summary at the top.

215 00:27:39.960 00:27:45.383 Uttam Kumaran: And then this is something that you should. You could just literally copy paste into AI, and it has the entire context of the repo.

216 00:27:45.750 00:27:49.129 Uttam Kumaran: So it shows the directory structure.

217 00:27:49.280 00:27:58.040 Uttam Kumaran: and then it goes and shows every file, the path, and then the entire code. And so this look, the whole repo in this, for this is like only 1,500 lines.

218 00:27:58.450 00:28:00.740 Uttam Kumaran: easily can shove that into AI.

219 00:28:01.090 00:28:04.550 Uttam Kumaran: The if we’re interested like, I can just

220 00:28:04.720 00:28:07.369 Uttam Kumaran: slap this workflow onto every repo.

221 00:28:07.810 00:28:10.200 Uttam Kumaran: And folks can test like

222 00:28:11.270 00:28:18.359 Uttam Kumaran: we can basically test out like, Hey, putting this directly into AI and and having an ask questions. That’s what I suggested. The AI team do.

223 00:28:24.770 00:28:27.939 Ryan: Yeah, I think that’s a good idea. It also like pushes

224 00:28:28.350 00:28:33.580 Ryan: the customers and like our clients to use AI and like, it’s also like

225 00:28:33.930 00:28:37.679 Ryan: some sort of marketing from our side that we’re using AI.

226 00:28:38.510 00:28:44.419 Ryan: They like it, probably one other AI services as well, or something like that. So yeah.

227 00:28:44.420 00:28:45.180 Uttam Kumaran: Yeah.

228 00:28:48.710 00:28:49.550 Uttam Kumaran: okay.

229 00:28:52.701 00:29:07.389 Amber Lin: Just to chime in here, put the mirror board in our chat. Do we wanna have a say 5 to 10 min where we drop in all the issues and ideas for each section. So we can sort of look at it visually.

230 00:29:07.730 00:29:10.530 Amber Lin: Were you guys able to access from your board.

231 00:29:11.210 00:29:14.080 Uttam Kumaran: Not everyone has Miro. Can we do this in fig jam?

232 00:29:14.840 00:29:15.699 Uttam Kumaran: I don’t even

233 00:29:16.340 00:29:27.040 Uttam Kumaran: okay. It’s fine. If not like I I don’t even know why. Like I I told the AI team not to stop using Miro. I don’t know why we’re using Miro, because we everybody in the company is on fig jam. So we should just use that.

234 00:29:27.260 00:29:28.380 Amber Lin: Oh, okay.

235 00:29:29.230 00:29:35.030 Uttam Kumaran: But it’s fine. I I don’t know. I think we can we? We don’t have a ton of time left. Maybe we can just keep going on this I don’t know.

236 00:29:35.030 00:29:35.650 Amber Lin: Sure.

237 00:29:36.000 00:29:43.270 Uttam Kumaran: Okay. But like, maybe just send that into. And maybe if you want to send it to me, Amber, and I can move that to to a fake jam’s point.

238 00:29:43.610 00:29:49.960 Amber Lin: Yeah, I just want to make sure that we cover all of these points and know what we’re gonna do for next meetings.

239 00:29:50.720 00:29:53.325 Uttam Kumaran: Yeah, I will. Yeah, I’ll I’ll

240 00:29:53.940 00:29:57.000 Uttam Kumaran: I’ll get to that, probably just with 15 min left in the meeting.

241 00:29:57.000 00:29:58.520 Amber Lin: Okay. Sounds good.

242 00:29:58.740 00:29:59.440 Uttam Kumaran: Cool.

243 00:30:02.240 00:30:07.750 Uttam Kumaran: Okay? So I think it seems like a big piece is documentation and knowledge sharing.

244 00:30:07.970 00:30:13.250 Uttam Kumaran: How did we all feel about like the message, Robertson, about the.

245 00:30:13.980 00:30:16.939 Uttam Kumaran: you know, recording looms and Pr reviews.

246 00:30:19.860 00:30:24.099 Uttam Kumaran: I will. I’ll just copy this

247 00:30:25.590 00:30:28.710 Uttam Kumaran: to the bottom here, so we can all see it in one place. But

248 00:30:29.190 00:30:32.679 Uttam Kumaran: how do we all think about this like inter team communication.

249 00:30:41.410 00:30:48.950 Ryan: Yeah, I think we definitely need to improve our asynchronous messaging, especially like considering

250 00:30:50.000 00:30:59.550 Ryan: like, not of all of us are like working at the same time, like I noticed some people like working earlier and being off earlier as well, and stuff like that. So

251 00:31:00.040 00:31:05.670 Ryan: maybe that’s something we need to standardize on, like how to me.

252 00:31:06.890 00:31:13.559 Ryan: slack communication as work, as asynchronous, like as possible, or something.

253 00:31:17.610 00:31:26.890 Demilade Agboola: I’m curious in terms of like Prs and stuff. Do do we have issues? I don’t know. Again, I I haven’t necessarily seen that. But do we have issues with

254 00:31:27.370 00:31:31.359 Demilade Agboola: understanding and being able to follow what’s happening in people’s Prs

255 00:31:33.170 00:31:38.650 Demilade Agboola: cause, I know, like, it’s definitely an issue with dashboards. I’m not necessarily sure if it’s an issue with prs.

256 00:31:42.870 00:31:45.399 Uttam Kumaran: Yeah, maybe anyone else want to go, and then I can go.

257 00:31:47.310 00:31:51.420 Caio Velasco: So I can give you just an example. For example. I’m

258 00:31:52.210 00:31:57.290 Caio Velasco: working as a I think, secondary in in pool parts with Luke.

259 00:31:57.590 00:32:05.379 Caio Velasco: and but I mean he’s he’s doing 99.9 9 9 9% of the work. So I’m more like in Javi.

260 00:32:05.510 00:32:33.719 Caio Velasco: And and then sometimes I say that I need to review some of his Pr. But since I’m honestly don’t have much idea of what is happening, or what what his beauty and model A, B or C, but I have to review them. So one thing that was helpful that we hop on a call, and he gave me some explanations. And so if the if that thing could be a loom video would be helpful but then I don’t know how far we need to go with it.

261 00:32:34.278 00:32:43.600 Caio Velasco: Because could be just because I don’t have the initial knowledge. But then, when I have probably won’t be there, and it won’t be needed, but

262 00:32:44.070 00:32:46.519 Caio Velasco: that’s what I can share. That happened with me.

263 00:32:52.650 00:32:55.030 Uttam Kumaran: And it was else thoughts on Vrs.

264 00:32:57.940 00:33:07.639 Demilade Agboola: Okay? I think in that case, yeah, it might be helpful to just like dual looms where we can like walk through the high level concept of what’s happening in the Pr

265 00:33:08.898 00:33:11.651 Demilade Agboola: potentially, I feel that

266 00:33:13.350 00:33:15.880 Demilade Agboola: In that case we might not necessarily need

267 00:33:16.140 00:33:24.049 Demilade Agboola: descriptions per se, like, because I mean, that’s the idea of a Pr description in in there. So you can kind of explain what’s happening.

268 00:33:24.460 00:33:24.810 Demilade Agboola: Yeah.

269 00:33:24.810 00:33:25.340 Demilade Agboola: But

270 00:33:25.680 00:33:31.599 Demilade Agboola: yeah, we could just maybe do looms and just kind of like, see what’s happening and what what the change was, and

271 00:33:31.720 00:33:34.410 Demilade Agboola: all of that. So we could bring people up to speed.

272 00:33:40.260 00:33:44.357 Uttam Kumaran: Yeah, I think. I agree on looms. I also think,

273 00:33:45.220 00:33:47.389 Uttam Kumaran: there’s sort of a couple of like.

274 00:33:47.980 00:33:54.450 Uttam Kumaran: I think there’s a bunch of pieces around like code documentation, right? So like code. So there’s like

275 00:33:54.690 00:34:03.610 Uttam Kumaran: inline Docs, there’s like pr descriptions. There’s are like platform docs, right?

276 00:34:03.740 00:34:11.699 Uttam Kumaran: So for all of these like, we want to make sure these are all like pretty easy to maintain. So I think part of the Pr review process is to make sure that

277 00:34:12.010 00:34:15.850 Uttam Kumaran: there’s something in the code that explains that there’s key logic. So

278 00:34:16.179 00:34:19.209 Uttam Kumaran: I think as a team, we can probably get a I haven’t been in.

279 00:34:19.380 00:34:32.329 Uttam Kumaran: I haven’t been in like sort of being a stickler on those items, but maybe I can comment a little bit more like, Hey, this, this looks like a pretty complex piece of logic, just putting a 1 line comment in the code can actually help.

280 00:34:32.530 00:34:38.300 Uttam Kumaran: So having this as part of the like Pr checklist

281 00:34:38.500 00:34:44.439 Uttam Kumaran: pr descriptions. I actually think we’ll probably end up just like generating these with AI

282 00:34:46.210 00:34:50.679 Uttam Kumaran: But again, these way, these may lack some context.

283 00:34:51.010 00:34:56.059 Uttam Kumaran: So I I agree in that. I’m I’m a big fan of loom.

284 00:34:56.562 00:34:58.499 Uttam Kumaran: I know it’s a little bit annoying, but

285 00:34:59.000 00:35:06.500 Uttam Kumaran: I think in probably 30 seconds you can quickly record a loom walk through your changes, and what probably would have taken like 5 or 10 min to write.

286 00:35:08.630 00:35:11.919 Uttam Kumaran: So if we’re all good with that, then we can basically

287 00:35:12.520 00:35:18.210 Uttam Kumaran: require, okay, there’s there’s a loom for every Pr, you know. And

288 00:35:18.580 00:35:23.700 Uttam Kumaran: I’m I’m I’m okay. With that. There’s a loom, or like, the description is is good enough.

289 00:35:23.830 00:35:29.920 Uttam Kumaran: you know, to understand like what’s going on in the Pr. Of course, some Prs are very self explanatory, so

290 00:35:30.640 00:35:34.820 Uttam Kumaran: I think those don’t need to be full looms. But, for example, there’s something where it’s like

291 00:35:35.120 00:35:38.039 Uttam Kumaran: hundreds of lines of change, or there’s some complicated logic.

292 00:35:38.560 00:35:48.859 Uttam Kumaran: We may the the options for the pr reviewer either to like, try to think about it, or to just like approve it, or like to have a meeting right? So I think a loom saves some of that.

293 00:35:50.030 00:36:07.469 Demilade Agboola: Yeah, I also think there should be a minimum threshold, like, for instance, if all you’re doing is a call list, for instance, and you’re changing something to from just a regular column to a call list to some like, you know. I think if that’s all you’re doing, I don’t think the next screen needs, or might need a peer, a loom.

294 00:36:07.470 00:36:08.310 Uttam Kumaran: Yeah.

295 00:36:09.320 00:36:21.219 Demilade Agboola: But if you’re doing, maybe a certain number of changes like maybe over 10 lines, 20 lines, we could figure out what that number is. But like once we’re trying to integrate certain logic in there.

296 00:36:21.665 00:36:26.560 Demilade Agboola: It might be harder to rely on the description, and we might need more of blooms.

297 00:36:28.060 00:36:28.690 Uttam Kumaran: Okay.

298 00:36:37.640 00:36:44.249 Uttam Kumaran: awaish, or Annie like any thoughts on the Pr process.

299 00:36:44.990 00:36:50.149 Awaish Kumar: For the click, like I’m quite happy with what we have right now.

300 00:36:51.960 00:37:01.749 Awaish Kumar: In a sense that like we do, I do write description depending. And there’s and the size of description depends on the complexity of Pr.

301 00:37:01.870 00:37:09.349 Awaish Kumar: If it is a simple one, like few lines of change, there’s only 1, 2 liner, which explains the main

302 00:37:10.120 00:37:12.130 Awaish Kumar: purpose of the Pr.

303 00:37:12.610 00:37:17.820 Awaish Kumar: And if there is any complex model that we are adding, we can write the assumptions we made.

304 00:37:18.020 00:37:21.970 Awaish Kumar: and these and the like, the the

305 00:37:22.656 00:37:29.000 Awaish Kumar: the results and what we are expecting, and the maybe writing down the queries and the

306 00:37:29.610 00:37:33.190 Awaish Kumar: screenshots of data that. Okay, how it does it look.

307 00:37:33.500 00:37:38.770 Awaish Kumar: giving an example of like from raw data, how it went to the

308 00:37:39.000 00:37:42.710 Awaish Kumar: model, things like that is is enough.

309 00:37:43.250 00:37:49.766 Awaish Kumar: And and we might. Yeah, for some we might recall loom if if the text explanation is

310 00:37:50.850 00:37:55.019 Awaish Kumar: it’s somewhat like confusing, but but otherwise, like.

311 00:37:56.280 00:38:00.516 Awaish Kumar: I think we we are doing the reviews for every Pr. And

312 00:38:01.240 00:38:06.770 Awaish Kumar: we do merge after the checks are passed and the after Pr is approved.

313 00:38:08.650 00:38:09.480 Uttam Kumaran: I agree.

314 00:38:11.914 00:38:17.589 Annie Yu: I’m thinking, cause I I’m not de so I think my view is probably different than

315 00:38:18.418 00:38:31.259 Annie Yu: some of you I think honestly a loom, for like a Pr or like a modeling change for me, wouldn’t be always helpful rather than like description, because I,

316 00:38:31.420 00:38:33.049 Annie Yu: if I have to go through

317 00:38:33.330 00:38:46.820 Annie Yu: a video, and I probably end up having to take some notes on the site. Just so I can reference in the future. And also like don’t want, because I I don’t know like how

318 00:38:46.980 00:38:50.869 Annie Yu: like I don’t use loom a lot. So I think for me that

319 00:38:51.260 00:38:54.209 Annie Yu: and and I don’t want to wait on

320 00:38:55.523 00:39:01.680 Annie Yu: a change I need for my dashboard. Just because people have to

321 00:39:02.120 00:39:04.739 Annie Yu: record a loom. If that makes sense.

322 00:39:05.180 00:39:06.530 Uttam Kumaran: Yeah, I agree.

323 00:39:09.900 00:39:13.029 Uttam Kumaran: it’s just it’s like, you know, a part of it is like.

324 00:39:13.160 00:39:15.540 Uttam Kumaran: some things are so basic. We don’t need it.

325 00:39:16.370 00:39:21.340 Uttam Kumaran: But maybe it is like some things are pretty like, I. I review a lot of Pr sometimes I’m like.

326 00:39:21.670 00:39:23.659 Uttam Kumaran: where do I even begin, you know.

327 00:39:23.960 00:39:29.960 Uttam Kumaran: And but the problem is, I’m in a jam like I’m like, do I block this? And then like slow things down? Or

328 00:39:30.510 00:39:32.150 Uttam Kumaran: do I get just get this out.

329 00:39:32.590 00:39:36.010 Uttam Kumaran: and in the moment I would ping someone on slack. But

330 00:39:36.760 00:39:39.670 Uttam Kumaran: if people aren’t online, then it’s a little bit tougher.

331 00:39:42.890 00:39:45.520 Uttam Kumaran: I also think about like, for our Pm’s like

332 00:39:45.650 00:39:49.335 Uttam Kumaran: they may be able to look up back on those, and and

333 00:39:50.550 00:39:55.190 Uttam Kumaran: understand a little bit of like if those are what the change was. So

334 00:39:55.320 00:40:00.490 Uttam Kumaran: I think, maybe roughly, you know, if I can summarize basically like

335 00:40:00.760 00:40:06.039 Uttam Kumaran: some stuff requires loom, some stuff doesn’t. I think maybe it’s more up to the reviewer.

336 00:40:06.300 00:40:12.360 Uttam Kumaran: And right now I’m I review a lot of stuff. I think it’ll start to split up between some of our more senior folks.

337 00:40:12.730 00:40:21.970 Uttam Kumaran: and then, you know, I I really want us to see us trying to use some of these Async tools, but don’t want it to be a blocker. Certainly, however.

338 00:40:22.380 00:40:24.540 Uttam Kumaran: it may skip a meeting, so

339 00:40:25.252 00:40:31.070 Uttam Kumaran: I think we’ll have some requirements. That’s like, hey? At least you need. At least they have like a a couple of line description.

340 00:40:31.230 00:40:33.059 Uttam Kumaran: It needs to pass the checks.

341 00:40:33.240 00:40:35.190 Uttam Kumaran: and if it’s a larger Pr have a loom.

342 00:40:35.570 00:40:38.509 Uttam Kumaran: But for me, I know, like

343 00:40:38.790 00:40:47.399 Uttam Kumaran: if if I was to go write out like some of my changes. It’d take me like 1015 min. But if I just sit here and like, Hey, today, I made this change. Blah blah blah.

344 00:40:48.070 00:40:52.270 Uttam Kumaran: I can sort of like chip that copy, send the loom done like, move on.

345 00:41:02.410 00:41:08.869 Uttam Kumaran: Okay. So I think that makes sense about communication. Oops, let me.

346 00:41:10.040 00:41:15.970 Uttam Kumaran: So let’s think about okay, future planning documentation. We talk a little bit about

347 00:41:16.582 00:41:18.197 Uttam Kumaran: this is where I think

348 00:41:19.330 00:41:22.649 Uttam Kumaran: Kyle, probably me and you can spend some time thinking about this.

349 00:41:23.110 00:41:30.660 Uttam Kumaran: And then observability, data, quality and governance. I sort of package both of both of these together

350 00:41:33.080 00:41:38.020 Uttam Kumaran: data, infrastructure, vision and alignment communication. And Async, okay.

351 00:41:38.230 00:41:43.240 Uttam Kumaran: how do we feel about all of these categories? By the way, like, do we think this covers, like most of it.

352 00:41:45.050 00:41:50.359 Uttam Kumaran: And then we can sort of talk about how we we want to tackle any of these items this week.

353 00:42:09.180 00:42:11.250 Uttam Kumaran: I don’t know how to just write this. But

354 00:42:13.750 00:42:16.880 Uttam Kumaran: any thoughts we feel good with this like these sets.

355 00:42:23.660 00:42:29.880 Uttam Kumaran: So basically, what I’m thinking is like turning these into into like

356 00:42:30.420 00:42:33.290 Uttam Kumaran: into projects like in in linear

357 00:42:34.075 00:42:39.349 Uttam Kumaran: and thinking about how we can split split up work here.

358 00:42:41.330 00:42:46.590 Uttam Kumaran: Is anyone like, I think, kind of the way we can think about

359 00:42:47.250 00:42:50.600 Uttam Kumaran: this sort of work, which is like partly tech debt work. Partly.

360 00:42:52.090 00:42:55.730 Uttam Kumaran: Probably platform work is maybe like have owners of each area.

361 00:42:56.332 00:43:00.849 Uttam Kumaran: So basically, one of us becomes like the go to on.

362 00:43:00.970 00:43:05.090 Uttam Kumaran: How do we write great documentation here? And like what decisions which we should make?

363 00:43:07.150 00:43:12.699 Uttam Kumaran: Like I can. Does anyone have any strong feelings about wanting to own one area or another?

364 00:43:13.285 00:43:15.770 Uttam Kumaran: I know I’ve talked to some folks on this call, but

365 00:43:15.990 00:43:21.790 Uttam Kumaran: would love to to sort of split some of these items up. Yeah.

366 00:43:23.340 00:43:27.380 Awaish Kumar: So I would love to work on data, infrastructure.

367 00:43:30.230 00:43:31.080 Uttam Kumaran: This piece.

368 00:43:31.810 00:43:32.570 Awaish Kumar: Yeah, yeah.

369 00:43:33.520 00:43:34.160 Uttam Kumaran: Okay.

370 00:43:37.900 00:43:42.320 Demilade Agboola: Is there? Some of them seem kind of similar in a in a sense.

371 00:43:42.320 00:43:43.210 Uttam Kumaran: Yeah.

372 00:43:43.720 00:43:53.259 Demilade Agboola: How would observability and monitoring differ from infrastructure and architecture? It feels like a sub like observability is a subset of infrastructure architecture.

373 00:43:54.490 00:44:06.090 Uttam Kumaran: Good point. Maybe. Let’s maybe. Let’s let’s split up these questions into other subcategories

374 00:44:11.260 00:44:13.260 Uttam Kumaran: like this is all

375 00:44:17.440 00:44:24.399 Uttam Kumaran: I mean. Let’s think about it. So there’s almost like observer military and monitoring.

376 00:44:24.940 00:44:29.010 Uttam Kumaran: This is more of like ingestion.

377 00:44:29.670 00:44:36.030 Uttam Kumaran: And dbt, do we? Do we think about like that’s a better category.

378 00:44:41.040 00:44:45.509 Demilade Agboola: I mean, in a way. Yes, but you know you can also do some observability with Dbt.

379 00:44:47.250 00:44:55.079 Uttam Kumaran: Yeah, I guess more of what I’m thinking is not like that. They’re totally separate, but that someone is just an owner of like a category.

380 00:44:57.240 00:45:00.240 Uttam Kumaran: And then if, of course, if there are basically like.

381 00:45:00.460 00:45:07.060 Uttam Kumaran: if there’s observability goes beyond just dbt, so whatever tool we use to do observability across a stack.

382 00:45:07.310 00:45:12.840 Uttam Kumaran: we should verify that it works with Dbt, but of course there’s items in Dbt that are beyond observability.

383 00:45:13.170 00:45:17.990 Uttam Kumaran: So I think, having some separation there works. I sort of think of this as like everything around bringing data in

384 00:45:18.290 00:45:19.419 Uttam Kumaran: and modeling it.

385 00:45:26.630 00:45:31.290 Awaish Kumar: Like, there are 2 things. Number one is data. Infrastructure is

386 00:45:31.500 00:45:37.425 Awaish Kumar: like all the tools and things. Yeah, working smoothly.

387 00:45:38.720 00:45:47.179 Awaish Kumar: And we do have, like the reliability reliability on our tools, that data is on time and we process on time.

388 00:45:47.710 00:45:57.909 Awaish Kumar: But then the other thing is observability and monitoring like, maybe more about the data itself, like the

389 00:45:59.670 00:46:06.689 Awaish Kumar: if we are bringing in the complete data and the data quality things like that.

390 00:46:07.920 00:46:08.600 Uttam Kumaran: Yeah.

391 00:46:15.230 00:46:16.130 Uttam Kumaran: okay.

392 00:46:18.440 00:46:21.479 Demilade Agboola: I think I would like to handle like documentation and knowledge, sharing.

393 00:46:24.480 00:46:25.130 Uttam Kumaran: Okay.

394 00:46:28.190 00:46:33.219 Uttam Kumaran: And there can be multiple people working on things. So I I don’t. It doesn’t have to be one person on everything. I just

395 00:46:33.700 00:46:37.860 Uttam Kumaran: if everyone want is interested in one area, I want to put everyone because I’ll take the

396 00:46:38.170 00:46:40.128 Uttam Kumaran: I’ll take the other ones.

397 00:46:40.990 00:46:46.849 Uttam Kumaran: I don’t know, Kyle, if if documentation seems a knowledge sharing seems also like a little bit up your world. But

398 00:46:47.110 00:46:48.410 Uttam Kumaran: let me know what you think.

399 00:46:49.580 00:46:51.400 Caio Velasco: You guys could be with them. And.

400 00:46:52.410 00:46:53.300 Uttam Kumaran: Okay, cool.

401 00:46:54.690 00:47:00.029 Uttam Kumaran: This is a huge, this is like a huge one. So it’s like, there’s plenty of stuff to do here.

402 00:47:06.810 00:47:08.110 Uttam Kumaran: Annie anything

403 00:47:08.380 00:47:12.729 Uttam Kumaran: like catch your eye. I know we didn’t spend a lot of time talking about the

404 00:47:13.817 00:47:18.300 Uttam Kumaran: analysis area as well, but we can certainly carve that out.

405 00:47:19.870 00:47:25.070 Uttam Kumaran: But I don’t know anything in on this list, or if we want to create an some new stuff, would love to hear your feedback.

406 00:47:28.080 00:47:29.130 Annie Yu: I’m thinking.

407 00:47:29.650 00:47:30.890 Uttam Kumaran: Okay, no problem.

408 00:47:33.310 00:47:40.239 Uttam Kumaran: I, you know, I also think we. You know, part of one thing we haven’t talked about here is like fundamentals of great modeling. Great analysis.

409 00:47:40.650 00:47:43.770 Uttam Kumaran: That is something somewhere around like

410 00:47:45.316 00:47:54.470 Uttam Kumaran: like, I don’t know. Maybe that’s somewhere around, like technical capability leveling up.

411 00:47:54.880 00:47:55.990 Uttam Kumaran: I don’t know.

412 00:47:59.320 00:48:03.529 Uttam Kumaran: But communications is also like a really big one. So

413 00:48:45.820 00:48:49.600 Uttam Kumaran: yeah, customer success. That’s like, kind of this one

414 00:48:50.400 00:48:54.030 Uttam Kumaran: client delivers reporting and customer success.

415 00:48:55.690 00:49:01.369 Uttam Kumaran: This is sort of like thinking about the Eng Pm sort of relationship.

416 00:49:40.480 00:49:45.484 Uttam Kumaran: I guess I can take this one observability.

417 00:49:48.090 00:49:51.480 Uttam Kumaran: I feel like a wish you were also sort of thinking about this.

418 00:49:51.710 00:49:53.669 Uttam Kumaran: I don’t know, Annie. I think if you

419 00:49:53.870 00:50:00.659 Uttam Kumaran: one thing that I’m sure will be a lot in your world is like, is this dashboard up to date? Is this data accurate?

420 00:50:00.820 00:50:03.420 Uttam Kumaran: So I would love if like.

421 00:50:03.680 00:50:06.889 Uttam Kumaran: if I can make a suggestion, if you want to also be involved in

422 00:50:07.380 00:50:10.699 Uttam Kumaran: observability and monitoring, because ultimately I think

423 00:50:10.950 00:50:15.510 Uttam Kumaran: the analyst team will be on the hook for those questions first, st

424 00:50:15.620 00:50:18.770 Uttam Kumaran: most like, most likely, but if

425 00:50:18.890 00:50:23.450 Uttam Kumaran: it may or may not be your your you may not be the one resolving.

426 00:50:23.750 00:50:26.389 Uttam Kumaran: but I don’t know. I think your perspective could be really good. There.

427 00:50:28.930 00:50:29.710 Annie Yu: Sure.

428 00:50:32.330 00:50:33.770 Annie Yu: No, that’s a good point.

429 00:50:35.090 00:50:41.820 Uttam Kumaran: Yeah. Otherwise, the way it typically goes, you know, is like the engineering team will basically say this, how we do monitoring. But then none of that will work for, like

430 00:50:42.050 00:50:45.189 Uttam Kumaran: the people who are like meeting with clients and like dealing with the feedback.

431 00:50:45.770 00:50:49.420 Uttam Kumaran: So I think this is a good way to have like sort of some cross collaboration

432 00:50:53.260 00:50:55.590 Uttam Kumaran: I know waste. You’re also poking at that

433 00:50:58.200 00:51:03.779 Uttam Kumaran: and we won’t tackle all of these. But I just want to keep people’s names on things that that may seem important.

434 00:51:13.730 00:51:18.249 Demilade Agboola: I think client variables are important and customer success. I think I’d like to work on that as well.

435 00:51:20.470 00:51:23.880 Uttam Kumaran: Are there other? Are there other items here that we think we should add

436 00:51:25.070 00:51:27.310 Uttam Kumaran: interested, like, how you think about this.

437 00:51:33.570 00:51:44.829 Demilade Agboola: Yeah, I think the only like, if you think about Sla’s concerning data, deliverables will be things about like staffing and just understanding like that. We’re scoping properly.

438 00:51:46.340 00:51:49.740 Uttam Kumaran: Yeah, cool.

439 00:52:02.860 00:52:07.640 Uttam Kumaran: Okay, great. And then I think I’ll ask. I can send this to Luke and ask for

440 00:52:07.930 00:52:14.690 Uttam Kumaran: his feedback. I mean the only other things that are left here are like future planning, innovation, technical capabilities.

441 00:52:15.260 00:52:19.811 Uttam Kumaran: Nobody’s interested in how to do communications.

442 00:52:23.270 00:52:24.540 Uttam Kumaran: Anyone

443 00:52:24.650 00:52:30.370 Uttam Kumaran: like. I don’t know. I think, Kyle, in our after our conversation this morning, I feel like me, and you can talk a little bit about like

444 00:52:31.560 00:52:35.549 Uttam Kumaran: about these, but is, I don’t know, for is anyone interested in like

445 00:52:37.410 00:52:40.799 Uttam Kumaran: writing clean code? How do we do? Great push? Great

446 00:52:41.520 00:52:46.230 Uttam Kumaran: dashboards like sort of that sort of like? Well, I guess you would call this like technical excellence.

447 00:52:47.330 00:52:49.190 Uttam Kumaran: Yeah, techno box ones.

448 00:52:55.670 00:52:57.080 Caio Velasco: I can help for sure.

449 00:53:02.070 00:53:05.339 Uttam Kumaran: So I can lead and then maybe include anyone who wants to be here.

450 00:53:05.720 00:53:11.109 Awaish Kumar: But like, what is the purpose of that? Like certifications, like? Everybody has to do

451 00:53:12.206 00:53:14.589 Awaish Kumar: certifications, and they have to

452 00:53:14.840 00:53:20.880 Awaish Kumar: learn something. But, like the the what? What is the responsibility of lead here.

453 00:53:22.830 00:53:31.920 Uttam Kumaran: Yeah, I think it’s, I think this is mainly like, like improving improving the technical skills

454 00:53:33.500 00:53:35.270 Uttam Kumaran: like across our team.

455 00:53:36.130 00:53:36.870 Awaish Kumar: Okay.

456 00:53:37.180 00:53:42.120 Uttam Kumaran: Trainings like I think of, like office hours as part of this.

457 00:53:44.510 00:53:48.629 Awaish Kumar: Okay, kind of arranging trainings and things like that.

458 00:53:49.270 00:53:58.782 Uttam Kumaran: Yeah, it’s sort of just like, you know, and a lot of companies they have like standards, or like, basically like, how do we get, how do we raise everyone up? Technically, in whatever function they’re doing?

459 00:53:59.695 00:54:04.860 Uttam Kumaran: So just basically like thinking about like, okay for for new employees, do we need to have like sort of a

460 00:54:05.390 00:54:13.209 Uttam Kumaran: a learning process like, how are we? How are we all? How are we setting together like a like learning budget like training anything like that.

461 00:54:14.110 00:54:14.850 Awaish Kumar: Okay.

462 00:54:20.700 00:54:21.370 Uttam Kumaran: Okay.

463 00:54:22.340 00:54:28.019 Uttam Kumaran: so I’ll just I’ll put my name on the other ones. If there’s anyone else is curious about these, let me know. But I think.

464 00:54:28.576 00:54:32.169 Uttam Kumaran: Again. I I know this is like there’s a lot to do here.

465 00:54:32.280 00:54:39.139 Uttam Kumaran: So mainly, I just wanna make sure we’re like covering the core parts of what affects our day to day across clients.

466 00:54:39.712 00:54:41.890 Uttam Kumaran: So kind of the next steps.

467 00:54:43.193 00:54:44.739 Uttam Kumaran: Here are.

468 00:54:44.920 00:54:49.180 Uttam Kumaran: I’m gonna go ahead and add,

469 00:54:52.420 00:54:55.659 Uttam Kumaran: I’m gonna go ahead and have this

470 00:54:55.810 00:55:01.749 Uttam Kumaran: as part of our linear. So here we have our data platform. I’ll just add these as projects here.

471 00:55:02.419 00:55:05.211 Uttam Kumaran: and basically, we can start to think about

472 00:55:06.350 00:55:13.339 Uttam Kumaran: what we need to add again, I I we’re all data people. So I’m not gonna run this like, I’m not gonna run this super strict

473 00:55:13.510 00:55:17.120 Uttam Kumaran: like, I just wanna make sure that some of these items that we know

474 00:55:17.250 00:55:21.340 Uttam Kumaran: like we want to do that are helpful platform wide. We can get to

475 00:55:21.550 00:55:31.270 Uttam Kumaran: all of these. You know, for folks that are hourly like all of these hours are just book. It’s the brain forge client. This is how we’re us working on

476 00:55:31.540 00:55:32.660 Uttam Kumaran: the machine.

477 00:55:33.424 00:55:40.050 Uttam Kumaran: But consider it like 10% of your time or so so few hours a week.

478 00:55:40.661 00:55:43.280 Uttam Kumaran: If you have to dedicate to one of these items.

479 00:55:43.894 00:55:49.270 Uttam Kumaran: And then, yeah, I think we’ll we’ll we’ll maybe we’ll talk as a as this crew for 30 min once a week.

480 00:55:49.440 00:55:56.090 Uttam Kumaran: just to make sure we’re we’re moving things forward. But I’m really, really, I think the biggest priorities that I heard today are like

481 00:55:56.470 00:56:00.620 Uttam Kumaran: documentation is data like accurate

482 00:56:02.690 00:56:05.560 Uttam Kumaran: and communications. Right? So I think

483 00:56:05.830 00:56:12.155 Uttam Kumaran: as a crew, let’s focus on those 3 items, nail those, and then we can get

484 00:56:12.640 00:56:14.299 Uttam Kumaran: get sort of get to the rest.

485 00:56:16.550 00:56:21.389 Uttam Kumaran: but yeah, I’m very, very excited. You know, it’s been a long time coming for us to think about our platform.

486 00:56:21.520 00:56:27.240 Uttam Kumaran: But our platform is not just like tools. It’s process. It’s how we run, how we, as engineers.

487 00:56:27.740 00:56:29.130 Uttam Kumaran: are more effective.

488 00:56:29.460 00:56:34.540 Uttam Kumaran: So any like closing thoughts or ideas before we

489 00:56:34.870 00:56:42.839 Uttam Kumaran: we hop off, and I’m hoping to. I’ll I’ll go ahead and and create some tickets and give some guidance. And we can have conversations. Async, in slack about

490 00:56:42.950 00:56:45.040 Uttam Kumaran: like things we want to do.

491 00:56:45.687 00:56:49.290 Uttam Kumaran: And, you know, make sure things are in linear.

492 00:56:52.900 00:57:03.130 Annie Yu: I have kind of a separate question, and I just thought this might be a good ask, do we have any like er diagram or that kind of thing that can show

493 00:57:03.450 00:57:06.910 Annie Yu: the relationship of each table.

494 00:57:08.790 00:57:12.799 Uttam Kumaran: Right now, we just have our like larger architecture diagram.

495 00:57:15.350 00:57:19.210 Uttam Kumaran: But maybe good question for a wish or demo lade like.

496 00:57:19.750 00:57:24.980 Uttam Kumaran: yeah, I don’t know. I think probably Dbt is the best source for that right?

497 00:57:25.640 00:57:28.110 Uttam Kumaran: So we should maybe run dbt, docs.

498 00:57:30.950 00:57:41.329 Demilade Agboola: Yeah, right now, we we only have, like Dbt Docs, as a general like across projects. That’s probably the best way we can currently get our er diagram.

499 00:57:43.050 00:57:46.589 Awaish Kumar: But we do have some right for Eden, for example.

500 00:57:47.443 00:57:54.250 Awaish Kumar: We have these star schemas for each of our Mods. That’s basically the year diagram.

501 00:57:55.660 00:57:59.199 Uttam Kumaran: But there’s nothing visual. Right, Annie, you’re looking for something like more visual.

502 00:57:59.898 00:58:08.039 Annie Yu: No, I’m just curious, because and I think for me mainly I want to see. Sometimes

503 00:58:08.720 00:58:12.680 Annie Yu: I think for me it’s also like that primary key. If there’s like a

504 00:58:13.350 00:58:21.510 Annie Yu: composite key, I sometimes had to spend time to figure out. Okay, these 2 columns together are that distinct

505 00:58:22.630 00:58:25.109 Annie Yu: value for each record.

506 00:58:25.460 00:58:33.190 Annie Yu: So if there’s some type of form that can let

507 00:58:33.940 00:58:36.980 Annie Yu: like an an end user of a table, know that.

508 00:58:37.140 00:58:47.740 Annie Yu: And also like things like. Sometimes we have pre-aggregated table with different columns, but pre-aggregated on different level.

509 00:58:48.310 00:58:57.490 Annie Yu: and I feel like that, having that that like descript description, would probably also be helpful. In some cases.

510 00:59:04.870 00:59:10.580 Uttam Kumaran: Yeah. So for these, like, I feel like, some of these are probably just solvable through, like, like.

511 00:59:11.040 00:59:18.830 Uttam Kumaran: let’s say, a solution potential solution for this is in the table you go. And there’s like information at the top, like inline comments. Basically.

512 00:59:19.110 00:59:23.220 Uttam Kumaran: this is the pk, this table is at this granularity

513 00:59:24.140 00:59:26.469 Uttam Kumaran: that solve it because the er diagram

514 00:59:26.590 00:59:29.288 Uttam Kumaran: there’s also this big visual component right?

515 00:59:30.060 00:59:30.600 Annie Yu: Them.

516 00:59:30.820 00:59:33.599 Uttam Kumaran: But if that’s like less necessary than

517 00:59:33.860 00:59:41.629 Uttam Kumaran: my mind, for this points more towards just like each table needs better like inline docs.

518 00:59:43.070 00:59:48.840 Annie Yu: Yeah, yeah, I think that’s that’s the right focus and priority.

519 00:59:55.560 01:00:06.800 Uttam Kumaran: So I think this is a great one. I mean, this is probably a good 1st ticket for the documentation squad to think about how we can set a standard for model level documentation.

520 01:00:10.010 01:00:16.059 Uttam Kumaran: right? Like, I’m gonna just put the problems here. But if we think about like ideas, this could just be like

521 01:00:17.039 01:00:26.940 Uttam Kumaran: docs on pk, granularity things to watch for at the top of each file.

522 01:00:27.660 01:00:31.330 Uttam Kumaran: This is maybe like adding to Faqs.

523 01:00:34.660 01:00:40.310 Uttam Kumaran: but yeah, I don’t know any, I guess, for Demo Lade or Kyle. Any other things we want to add here.

524 01:00:44.550 01:00:49.809 Uttam Kumaran: I’ll just leave the problem. I’ll just leave it like this, because I want you guys to sort of think through

525 01:00:51.430 01:00:56.689 Uttam Kumaran: how to solve it. The but the other the other piece I’ll mention, too, is like

526 01:00:56.940 01:01:02.790 Uttam Kumaran: we should use AI like a ton in this process, I think.

527 01:01:04.440 01:01:09.200 Uttam Kumaran: both like, when we have this documentation, we can have AI go write some of this for us

528 01:01:09.400 01:01:14.680 Uttam Kumaran: updating documentation. So throughout this process, hopefully, I’ll start to suggest, like.

529 01:01:15.520 01:01:20.350 Uttam Kumaran: Hey, we should the AI team, we should just go help do these like help assist in this sort of way?

530 01:01:22.400 01:01:25.559 Uttam Kumaran: But like, yeah, hearing these problems that that makes a lot of sense.

531 01:01:26.760 01:01:29.180 Uttam Kumaran: Otherwise, you basically have to run a query to figure this out.

532 01:01:31.280 01:01:49.794 Amber Lin: Totally, and whoever owns that project will will be the person I go to to ask, because right now, even for for all the data projects we’ve been talking to a lot of mostly, and it’ll be great to have each person be responsible for each

533 01:01:50.360 01:02:06.100 Amber Lin: category, because then more problems will come up and we’ll have more, even more specific requirements. And for AI, we really require very, very detailed context and details instructions. Or else the AI will be very useful.

534 01:02:07.750 01:02:08.340 Uttam Kumaran: Yeah.

535 01:02:13.130 01:02:13.970 Uttam Kumaran: okay.

536 01:02:14.310 01:02:19.989 Uttam Kumaran: cool. I think anything else that comes up this week throw it in backlog or throw it in here again, like

537 01:02:20.540 01:02:26.429 Uttam Kumaran: we’re all data folks. So like, we don’t need to. DM, this one more than it needs to.

538 01:02:26.620 01:02:31.609 Uttam Kumaran: We all have like enough tickets and stuff. But I just wanna make sure that we can make some of these decisions as a squad

539 01:02:32.050 01:02:34.929 Uttam Kumaran: and then start to roll this out to each of our clients.

540 01:02:35.776 01:02:41.890 Uttam Kumaran: So yeah, super pumped. Any other anything else before we close?

541 01:02:47.650 01:02:48.400 Uttam Kumaran: Okay.

542 01:02:48.700 01:02:54.200 Uttam Kumaran: cool. Thanks. Guys, excited. Yeah, I’ll probably keep this time every week. And then we’ll just talk in slack.

543 01:02:56.770 01:02:57.410 Caio Velasco: Thank you.

544 01:02:57.410 01:02:58.860 Annie Yu: Sounds good. Thank you.

545 01:02:59.680 01:03:01.000 Demilade Agboola: Thank you. Bye.