2025-12-22_planning_de-ae-ai

Meeting Title: Planning: DE-AE-AI Date: 2025-12-22 Meeting participants: Awaish Kumar, Mustafa Raja, Rico Rejoso, Clarence Stone, Samuel Roberts, Elizah Joy, Demilade Agboola, Ashwini Sharma, Uttam Kumaran

WEBVTT

1 00:05:33.610 ⇒ 00:05:35.459 Awaish Kumar: Hello, everyone.

2 00:05:37.260 ⇒ 00:05:38.480 Awaish Kumar: How was the weekend?

3 00:05:41.630 ⇒ 00:05:43.530 Samuel Roberts: Pretty nice for me, yeah.

4 00:05:44.540 ⇒ 00:05:45.910 Samuel Roberts: A lot of family in town.

5 00:05:48.040 ⇒ 00:05:49.160 Awaish Kumar: Okay, go.

6 00:06:08.970 ⇒ 00:06:10.209 Awaish Kumar: There you go.

7 00:06:20.010 ⇒ 00:06:24.969 Awaish Kumar: Nico, do you know, like, if Damalade and Ashwini are…

8 00:06:26.320 ⇒ 00:06:29.240 Awaish Kumar: Are not all… not all out of office today?

9 00:07:01.050 ⇒ 00:07:02.950 Awaish Kumar: Okay, yeah, we can stop.

10 00:07:05.530 ⇒ 00:07:06.200 Awaish Kumar: Hmm.

11 00:07:13.010 ⇒ 00:07:14.789 Awaish Kumar: Let me share my screen.

12 00:07:44.170 ⇒ 00:07:49.120 Awaish Kumar: Okay, yeah, I will start with the data projects.

13 00:07:49.560 ⇒ 00:07:53.839 Awaish Kumar: Oh… So he can maybe…

14 00:07:58.420 ⇒ 00:08:01.039 Awaish Kumar: start with, Eden.

15 00:08:02.650 ⇒ 00:08:06.569 Awaish Kumar: Ashuni, do you want to give updates on your tickets?

16 00:08:07.570 ⇒ 00:08:11.870 Ashwini Sharma: On Aaron, I’m still working on the CICD for this thing.

17 00:08:12.970 ⇒ 00:08:14.359 Awaish Kumar: CICD for what?

18 00:08:14.360 ⇒ 00:08:20.260 Ashwini Sharma: Oh, sorry, that was for CTA, sorry for the confusion here. In this one.

19 00:08:20.460 ⇒ 00:08:24.339 Ashwini Sharma: I’m working on 1240, right? .

20 00:08:24.340 ⇒ 00:08:25.170 Awaish Kumar: Because…

21 00:08:25.880 ⇒ 00:08:27.370 Ashwini Sharma: Yeah, that was done.

22 00:08:34.400 ⇒ 00:08:40.489 Ashwini Sharma: So, yeah, on 12.40, I will sit together with, henry, and then…

23 00:08:40.740 ⇒ 00:08:43.230 Ashwini Sharma: decide, like, I have got the logic with me now.

24 00:08:43.640 ⇒ 00:08:47.039 Ashwini Sharma: But we’ll have to take a call on what all,

25 00:08:47.160 ⇒ 00:08:51.750 Ashwini Sharma: Oh, hold on a second, this is different, sorry, sorry, this is, something different.

26 00:08:52.630 ⇒ 00:08:55.220 Awaish Kumar: Okay, for this one, you can ask.

27 00:08:55.220 ⇒ 00:09:01.550 Ashwini Sharma: Yeah, I’ll get together with Henry only for this one, because, This should resolve,

28 00:09:01.930 ⇒ 00:09:09.829 Ashwini Sharma: The issue that he was talking about, mainly because, like, we have added that transaction number equals to 1 in order summary table, but that was never utilized.

29 00:09:10.090 ⇒ 00:09:15.279 Ashwini Sharma: And I saw all the dependencies on order summary as well, so nowhere…

30 00:09:15.550 ⇒ 00:09:27.999 Ashwini Sharma: we are utilizing any records per transaction number is anything other than one. So what I had done was, I kind of filtered all the other transactions number and just created one record per order ID.

31 00:09:29.640 ⇒ 00:09:36.389 Ashwini Sharma: not exactly per order, it has multiple events, but still, yeah, that solves the purpose that Henry is looking for.

32 00:09:36.580 ⇒ 00:09:44.289 Awaish Kumar: Okay, for order summary table, we have our data in a format that have multiple rows for an order.

33 00:09:45.110 ⇒ 00:09:45.770 Ashwini Sharma: Yes.

34 00:09:45.920 ⇒ 00:09:49.699 Awaish Kumar: And that row number was there just to get, like, if we want latest.

35 00:09:49.810 ⇒ 00:09:51.440 Ashwini Sharma: information.

36 00:09:53.390 ⇒ 00:09:54.340 Ashwini Sharma: Right.

37 00:09:54.550 ⇒ 00:09:55.349 Awaish Kumar: Fair enough. Yeah.

38 00:09:56.670 ⇒ 00:09:58.380 Awaish Kumar: So how you are solving that, right?

39 00:09:58.380 ⇒ 00:10:03.599 Ashwini Sharma: And, there is… Was there a requirement that we needed,

40 00:10:04.270 ⇒ 00:10:07.679 Ashwini Sharma: We needed older records for the order summary.

41 00:10:08.170 ⇒ 00:10:09.270 Awaish Kumar: Yes.

42 00:10:10.050 ⇒ 00:10:10.610 Ashwini Sharma: Oh.

43 00:10:11.960 ⇒ 00:10:16.269 Demilade Agboola: So, for order summary, the reason why we have it that way is for shipping.

44 00:10:16.510 ⇒ 00:10:24.420 Demilade Agboola: So, every shipping record creates a new role. A new role. Similar, Pulse. This pulse board, yes.

45 00:10:25.790 ⇒ 00:10:27.939 Ashwini Sharma: Sorry, man, I… I didn’t follow that.

46 00:10:27.940 ⇒ 00:10:37.300 Demilade Agboola: So every shipping record creates a new row, and so the idea is if we can select the latest row, that creates the…

47 00:10:38.090 ⇒ 00:10:43.450 Demilade Agboola: That creates a table where the green is the order ID.

48 00:10:44.040 ⇒ 00:10:47.120 Demilade Agboola: So every, every other idea is unique.

49 00:10:48.600 ⇒ 00:10:49.639 Ashwini Sharma: When do you use a honey?

50 00:10:49.640 ⇒ 00:10:50.440 Demilade Agboola: Every once.

51 00:10:51.060 ⇒ 00:10:53.360 Ashwini Sharma: Yeah, when do you use an older record?

52 00:10:55.060 ⇒ 00:10:57.970 Demilade Agboola: Like, the entire, like, all the data, when it is…

53 00:10:58.390 ⇒ 00:11:07.460 Ashwini Sharma: No, my question is, when do you… when is the situation where you would utilize… where transaction number equals to 2, or 3, or 4, other than 1?

54 00:11:07.460 ⇒ 00:11:08.130 Demilade Agboola: Ye…

55 00:11:08.630 ⇒ 00:11:18.110 Demilade Agboola: when we’re looking at shipping, shipping is the reason why it is expanded. So if we want to understand the progress that you… an order goes through, how long.

56 00:11:18.110 ⇒ 00:11:18.500 Ashwini Sharma: Got it.

57 00:11:18.970 ⇒ 00:11:21.010 Demilade Agboola: Yeah, that’s what we’ll look at.

58 00:11:21.010 ⇒ 00:11:24.510 Awaish Kumar: Ramadi, do we have a dashboard which is actually using that?

59 00:11:24.770 ⇒ 00:11:26.360 Awaish Kumar: Like, the auto journey?

60 00:11:27.600 ⇒ 00:11:33.909 Demilade Agboola: Yeah, the other journey uses that, and I guess maybe one or two more do.

61 00:11:36.310 ⇒ 00:11:52.739 Awaish Kumar: So, yeah, actually, we have two, three dashboards which basically are showing data related to shipping for an order, and they use all this data. Not exactly we select row number equal to 2, 3, but it just uses the journey of an order, like, when it moved from

62 00:11:52.810 ⇒ 00:11:59.730 Awaish Kumar: Like, the sent to pharmacy to deliver, how long it took to calculate these things, we need this, you know.

63 00:12:00.490 ⇒ 00:12:03.320 Ashwini Sharma: Okay, in that case, we can’t remove it, right?

64 00:12:03.960 ⇒ 00:12:09.529 Awaish Kumar: Yeah, so what was suggested in that case was that either we keep it like that.

65 00:12:09.700 ⇒ 00:12:17.579 Awaish Kumar: If it is… if, like… If it is okay, but if it is, like, making more, like.

66 00:12:18.200 ⇒ 00:12:37.019 Awaish Kumar: confusion for Henry or anyone else. What was suggested was that, like, create a, for example, order summary, just keep it… keep it like that, where you can create order summary latest or something, which only have the latest records for an order, like, single loop or order.

67 00:12:37.270 ⇒ 00:12:40.760 Ashwini Sharma: Okay, yeah, yeah, yeah, yeah. Okay, I’ll do that.

68 00:12:42.120 ⇒ 00:12:48.380 Awaish Kumar: Okay, thought, like… This should be…

69 00:12:49.360 ⇒ 00:12:51.430 Awaish Kumar: I can send it to the…

70 00:12:51.920 ⇒ 00:12:55.999 Awaish Kumar: past cycle. So, for this cycle, we have…

71 00:12:57.630 ⇒ 00:13:03.159 Awaish Kumar: three story points, and like, are you working on this Geminari? 1828? 1, 2,82?

72 00:13:04.310 ⇒ 00:13:14.610 Demilade Agboola: Yes, that’s for this cycle. I’m going to reach out to Sesame, or Henry, I’m not sure who’s online today, but the idea is…

73 00:13:14.830 ⇒ 00:13:24.019 Demilade Agboola: I do need some clarification, and I asked them, and no one responded, who was available for clarification. I’ll just follow up on that and get,

74 00:13:24.320 ⇒ 00:13:27.670 Demilade Agboola: Okay. A response, once I get a response, I will start the task.

75 00:13:28.330 ⇒ 00:13:33.289 Awaish Kumar: So, we have a short week, and we have almost, like, around 5 hours of work.

76 00:13:33.440 ⇒ 00:13:38.290 Awaish Kumar: what I… what I want to do is, like, we are almost at the…

77 00:13:38.550 ⇒ 00:13:43.260 Awaish Kumar: capacity for this client, so… it’s okay, like, we may get some requests

78 00:13:43.640 ⇒ 00:13:49.529 Awaish Kumar: in between for one or two story points, and we can do that. But for now, I think we are good for Eden.

79 00:13:53.830 ⇒ 00:13:59.249 Demilade Agboola: Yeah, I agree. It definitely is a short week, and there are a lot of things already on the plate.

80 00:14:00.750 ⇒ 00:14:03.859 Awaish Kumar: Then we can just move on to default.

81 00:14:04.350 ⇒ 00:14:10.949 Awaish Kumar: For this week, what are you planning to do? And what about… what is the status on 191?

82 00:14:13.940 ⇒ 00:14:23.550 Demilade Agboola: So for 191, I should close that today. I already built out most of the things in there. I will just show the client their review.

83 00:14:23.820 ⇒ 00:14:29.590 Demilade Agboola: Today, and then in terms of… Like, this week?

84 00:14:29.720 ⇒ 00:14:33.699 Demilade Agboola: Again, it’s a short week, but the idea is I’m just going to follow up on…

85 00:14:34.270 ⇒ 00:14:37.580 Demilade Agboola: The whole, like, the data, the architecture.

86 00:14:37.700 ⇒ 00:14:39.740 Demilade Agboola: And see if we can…

87 00:14:40.570 ⇒ 00:14:53.169 Demilade Agboola: like, what we have from Salesforce, like, see what we do have from Salesforce. I’m going to sync with Mustafa. I know, Thomas was supposed to send some Salesforce data. Yes. See what we have from Salesforce, how we can integrate that.

88 00:14:53.520 ⇒ 00:14:57.160 Demilade Agboola: This is no… Pardon?

89 00:14:57.160 ⇒ 00:14:58.299 Awaish Kumar: This is not Central, right?

90 00:14:58.300 ⇒ 00:15:01.050 Demilade Agboola: Don’t know that, that’s on Salesforce.

91 00:15:01.050 ⇒ 00:15:04.680 Awaish Kumar: Okay, so you are working on this and working We’ll close it maybe today.

92 00:15:04.680 ⇒ 00:15:05.210 Demilade Agboola: Excellent.

93 00:15:05.210 ⇒ 00:15:08.270 Awaish Kumar: Yeah, and then for Salesforce, I want to integrate.

94 00:15:08.270 ⇒ 00:15:09.730 Demilade Agboola: the Salesforce data.

95 00:15:09.900 ⇒ 00:15:12.870 Demilade Agboola: explore and then integrate Salesforce data into what we have.

96 00:15:13.310 ⇒ 00:15:17.669 Awaish Kumar: So, Salesforce data, where is it now? Like, right now, we… do we have access?

97 00:15:18.140 ⇒ 00:15:18.620 Mustafa Raja: No…

98 00:15:18.620 ⇒ 00:15:19.190 Demilade Agboola: Yes.

99 00:15:19.400 ⇒ 00:15:23.130 Mustafa Raja: Thomas was supposed to send the Salesforce data, over the.

100 00:15:23.130 ⇒ 00:15:24.070 Demilade Agboola: Love the weekend.

101 00:15:24.640 ⇒ 00:15:39.170 Mustafa Raja: I haven’t heard from him, so, today I… I will, catch up with him, see where it is right now. Hopefully we’ll have the Salesforce data today, and, me and Demiladi can take a look at that then.

102 00:15:40.180 ⇒ 00:15:44.400 Demilade Agboola: Okay, so… I am very…

103 00:15:44.400 ⇒ 00:15:47.029 Awaish Kumar: Save it in files, right?

104 00:15:47.030 ⇒ 00:15:48.490 Mustafa Raja: Yes, probably CSV.

105 00:15:48.490 ⇒ 00:15:50.740 Demilade Agboola: Yeah, it can be export.

106 00:15:50.740 ⇒ 00:15:53.540 Awaish Kumar: We are going to upload it to MotherDuck, right?

107 00:15:53.540 ⇒ 00:15:54.310 Mustafa Raja: Yes.

108 00:15:57.140 ⇒ 00:16:00.499 Awaish Kumar: Okay, and how many story points?

109 00:16:00.500 ⇒ 00:16:04.240 Mustafa Raja: Let’s do… yeah, two, let’s do two, yeah.

110 00:16:06.870 ⇒ 00:16:10.470 Awaish Kumar: So, it’s on you, and then this one, or this one.

111 00:16:11.520 ⇒ 00:16:19.179 Mustafa Raja: Yeah, this depends on, the Catalyst and S3 integration that we met with Thomas on.

112 00:16:19.180 ⇒ 00:16:20.049 Awaish Kumar: It’s blocked last week.

113 00:16:20.520 ⇒ 00:16:22.099 Mustafa Raja: Yeah, it’s long.

114 00:16:23.120 ⇒ 00:16:24.160 Awaish Kumar: This one…

115 00:16:24.340 ⇒ 00:16:26.629 Mustafa Raja: Yeah, this is in, client review.

116 00:16:31.960 ⇒ 00:16:37.670 Awaish Kumar: So basically, for this week, we only… for now, we only have this ticket, where you’re actively working on.

117 00:16:38.050 ⇒ 00:16:43.659 Mustafa Raja: Yes, I will, I will, catch up with Thomas and see where we are with it right now.

118 00:16:43.660 ⇒ 00:16:45.569 Awaish Kumar: 191 for Tamilade.

119 00:16:45.720 ⇒ 00:16:49.070 Awaish Kumar: We might do some Salesforce…

120 00:16:53.050 ⇒ 00:16:54.360 Awaish Kumar: Exploration…

121 00:16:59.110 ⇒ 00:17:02.760 Awaish Kumar: And… Portland.

122 00:17:08.230 ⇒ 00:17:11.680 Awaish Kumar: So, similarity, how… how many story points do you think.

123 00:17:12.510 ⇒ 00:17:16.990 Demilade Agboola: I haven’t seen it yet, but I think 2 to 3 should be fine. I should say 3.

124 00:17:17.150 ⇒ 00:17:19.969 Awaish Kumar: Yeah, let’s… then we can just keep it simple.

125 00:17:20.290 ⇒ 00:17:21.139 Awaish Kumar: Groups of one.

126 00:17:21.480 ⇒ 00:17:22.700 Awaish Kumar: Just explore it.

127 00:17:22.730 ⇒ 00:17:25.010 Demilade Agboola: And we’ll have separate modeling tickets.

128 00:17:28.860 ⇒ 00:17:32.530 Awaish Kumar: So, once you explore, you will know, like, what is to be done.

129 00:17:33.040 ⇒ 00:17:33.979 Demilade Agboola: Yeah, fair enough.

130 00:17:36.760 ⇒ 00:17:41.999 Awaish Kumar: Yeah, this… then we can… we can adjust this, but I think we can keep it at 2 for now.

131 00:17:48.220 ⇒ 00:17:54.560 Awaish Kumar: Okay, so we are on, like, 3 storylines, and… to hear.

132 00:17:56.010 ⇒ 00:18:01.000 Awaish Kumar: Five in total, and these are still kind of… blogged it.

133 00:18:01.210 ⇒ 00:18:06.530 Awaish Kumar: So is there anything, like, that we know we have to deliver?

134 00:18:06.690 ⇒ 00:18:07.759 Awaish Kumar: This week?

135 00:18:09.600 ⇒ 00:18:12.599 Demilade Agboola: Nothing I know of.

136 00:18:12.840 ⇒ 00:18:19.210 Demilade Agboola: I will say that we should start thinking about putting, like, the ETL in the, look.

137 00:18:19.670 ⇒ 00:18:26.400 Demilade Agboola: in the pipeline of what we want to do. Obviously not this week, but, like, something we should just put, like, subsequent cycles.

138 00:18:27.760 ⇒ 00:18:32.509 Awaish Kumar: Yeah, but, like, do you want to work on, like, kind of ETL plan?

139 00:18:33.360 ⇒ 00:18:34.550 Awaish Kumar: For the public.

140 00:18:34.890 ⇒ 00:18:35.700 Awaish Kumar: You can put…

141 00:18:35.700 ⇒ 00:18:36.509 Demilade Agboola: That’s what I’m thinking.

142 00:18:36.690 ⇒ 00:18:37.330 Awaish Kumar: Yeah.

143 00:18:37.460 ⇒ 00:18:43.960 Awaish Kumar: You can put up a document where you recommend what tools you are going to use, and why, and…

144 00:18:44.180 ⇒ 00:18:45.900 Awaish Kumar: Cost, and things like that.

145 00:18:47.660 ⇒ 00:18:51.629 Demilade Agboola: Yeah. Mustafa, have you done anything on that, so that we don’t, like.

146 00:18:51.800 ⇒ 00:18:52.400 Mustafa Raja: No.

147 00:18:52.400 ⇒ 00:18:54.330 Demilade Agboola: This is just in… okay, that’s fine then.

148 00:18:56.590 ⇒ 00:19:03.330 Awaish Kumar: ETL plan… Is there anything, Mustafa, like, from modeling side you need?

149 00:19:04.020 ⇒ 00:19:09.389 Mustafa Raja: Nope. For me, this week is only going to be the Salesforce data.

150 00:19:11.280 ⇒ 00:19:15.600 Awaish Kumar: Yeah, but that’s… we just get it, upload to MotherDuck, and that’s all.

151 00:19:15.740 ⇒ 00:19:16.380 Mustafa Raja: Yep.

152 00:19:18.500 ⇒ 00:19:20.519 Awaish Kumar: Okay, so do we want to…

153 00:19:20.870 ⇒ 00:19:24.859 Awaish Kumar: Do you want to get the scope from Khutam, or…

154 00:19:25.050 ⇒ 00:19:27.000 Awaish Kumar: A client, or something like that?

155 00:19:28.120 ⇒ 00:19:30.589 Awaish Kumar: Are we clear? Like, is that…

156 00:19:32.200 ⇒ 00:19:38.799 Mustafa Raja: Yeah, for me, it’s clear. I just, my task is to only get the Salesforce data, put it in Mother Duck for demo RD.

157 00:19:39.620 ⇒ 00:19:44.110 Demilade Agboola: Yeah, so that would be your task for the week?

158 00:19:44.600 ⇒ 00:19:44.980 Mustafa Raja: Yes.

159 00:19:44.980 ⇒ 00:19:55.309 Demilade Agboola: But, like, subsequent years, since the short week, this is fine, but I think subsequently, I will sync with Otam and just get more context on what we need going forward.

160 00:19:55.310 ⇒ 00:19:59.729 Awaish Kumar: analytics side, I think, okay, it seems okay that if we have Salesforce data.

161 00:19:59.860 ⇒ 00:20:02.679 Awaish Kumar: This week can easily be filled out.

162 00:20:03.280 ⇒ 00:20:10.840 Awaish Kumar: But on the Mustwa side, I don’t know how… how much you’re spending on this… like, ideally, we should be spending on this kind Muswa.

163 00:20:15.410 ⇒ 00:20:20.839 Mustafa Raja: Yeah, so my allocations for default are about 8 to 10 hours.

164 00:20:21.080 ⇒ 00:20:22.880 Awaish Kumar: Yeah, so…

165 00:20:22.880 ⇒ 00:20:27.240 Mustafa Raja: This is a… this is a small week. It’s only today and tomorrow, right?

166 00:20:27.810 ⇒ 00:20:29.210 Awaish Kumar: Wednesday also, right?

167 00:20:30.610 ⇒ 00:20:30.930 Mustafa Raja: Yeah.

168 00:20:30.930 ⇒ 00:20:32.929 Demilade Agboola: I think Wednesday as well, so it’s…

169 00:20:32.930 ⇒ 00:20:36.279 Awaish Kumar: We have, like, maybe we can spend 6 hours a day.

170 00:20:37.500 ⇒ 00:20:38.520 Mustafa Raja: Yep.

171 00:20:42.480 ⇒ 00:20:47.339 Awaish Kumar: So, is… are there any tickets in, kind of, backlog or somewhere?

172 00:20:58.690 ⇒ 00:20:59.780 Awaish Kumar: Increal.

173 00:21:01.620 ⇒ 00:21:05.579 Awaish Kumar: So… What about this mother duct 2S3? Is it…

174 00:21:05.910 ⇒ 00:21:08.040 Awaish Kumar: Does we got any reply from clients?

175 00:21:08.040 ⇒ 00:21:14.470 Mustafa Raja: No, and I think, I think they, they are not exploring that anymore, also.

176 00:21:17.890 ⇒ 00:21:21.980 Awaish Kumar: Brokers… she’s like, okay, nobody’s also working on.

177 00:21:26.740 ⇒ 00:21:34.430 Demilade Agboola: Also, part of what I’m doing, not necessarily only this week, this week I’m going to see a couple of people.

178 00:21:34.670 ⇒ 00:21:41.069 Demilade Agboola: But the idea is I’m going to meet people and get an idea of what they need, and get their requirements, analytical requirements.

179 00:21:41.370 ⇒ 00:21:44.619 Demilade Agboola: I start to use that to help with the scoping.

180 00:21:45.970 ⇒ 00:21:50.610 Awaish Kumar: Okay, so this… do we have a data platform document for this client?

181 00:21:51.020 ⇒ 00:21:53.560 Mustafa Raja: Mmm… no, I don’t think so.

182 00:21:54.190 ⇒ 00:21:56.780 Awaish Kumar: We have, architecture diagram.

183 00:21:58.220 ⇒ 00:21:59.620 Mustafa Raja: No, I didn’t think so.

184 00:22:00.660 ⇒ 00:22:05.159 Awaish Kumar: can… Can we include these if we are not working on anything else?

185 00:22:06.040 ⇒ 00:22:08.010 Mustafa Raja: Yeah, I guess we could… we could do that then.

186 00:22:11.800 ⇒ 00:22:15.169 Awaish Kumar: Yeah, but this is low priority. If you get anything else, we can…

187 00:22:15.420 ⇒ 00:22:16.130 Mustafa Raja: Okay.

188 00:22:16.730 ⇒ 00:22:17.810 Awaish Kumar: We can move it.

189 00:22:19.450 ⇒ 00:22:20.190 Awaish Kumar: Come back.

190 00:22:22.210 ⇒ 00:22:24.280 Awaish Kumar: Same for this.

191 00:22:31.570 ⇒ 00:22:34.320 Awaish Kumar: Okay, for Demolade, do you want me to…

192 00:22:36.580 ⇒ 00:22:40.400 Awaish Kumar: Add, like, tickets for your meetings, or something like that?

193 00:22:41.750 ⇒ 00:22:48.299 Demilade Agboola: Not necessarily. I have a meeting with Laura today, and so the idea is I just will get…

194 00:22:48.790 ⇒ 00:22:53.010 Awaish Kumar: Yeah, but… Like, kind of discovery, or…

195 00:22:53.440 ⇒ 00:22:57.239 Demilade Agboola: Yeah, Discovery Calls is just a basic idea.

196 00:22:58.880 ⇒ 00:23:00.269 Demilade Agboola: That’ll be the ticket for it.

197 00:23:01.940 ⇒ 00:23:03.169 Awaish Kumar: There is a ticket?

198 00:23:03.880 ⇒ 00:23:11.600 Demilade Agboola: No, no, so that would be the name of the ticket, basically, discovery call with Laura. Yeah, yeah, so that’s what I’m… just keeping it as a placeholder, so we know…

199 00:23:11.600 ⇒ 00:23:15.519 Awaish Kumar: What is the outcome of this meeting, and what you learn, things like that.

200 00:23:16.110 ⇒ 00:23:17.220 Demilade Agboola: Yeah, sounds good.

201 00:23:26.760 ⇒ 00:23:28.820 Awaish Kumar: Done.

202 00:23:29.060 ⇒ 00:23:35.969 Awaish Kumar: I don’t have anything on Alvin Sam’s README, you know… For element, I…

203 00:23:37.330 ⇒ 00:23:44.040 Awaish Kumar: Okay, basically, I will be working on… Setting up.

204 00:23:45.170 ⇒ 00:23:52.350 Awaish Kumar: Snowflake… And start, ingestion.

205 00:23:54.300 ⇒ 00:23:57.390 Awaish Kumar: If I… Recharge.

206 00:24:09.860 ⇒ 00:24:10.650 Awaish Kumar: Yeah.

207 00:24:13.550 ⇒ 00:24:16.220 Awaish Kumar: I think that’s what I would be doing.

208 00:24:19.230 ⇒ 00:24:23.420 Awaish Kumar: Yeah, set up… I can just split all of these.

209 00:24:33.830 ⇒ 00:24:43.120 Awaish Kumar: Basically, to write cycles… We have… congestion… For this…

210 00:24:46.600 ⇒ 00:24:49.510 Awaish Kumar: And then we have, DVD.

211 00:24:50.300 ⇒ 00:24:51.050 Awaish Kumar: Okay.

212 00:24:52.440 ⇒ 00:24:53.330 Awaish Kumar: picked up.

213 00:25:02.990 ⇒ 00:25:12.410 Awaish Kumar: And yeah, that’s… That’s it, but I’m… maybe I will work on also CICD stuff, oh…

214 00:25:13.630 ⇒ 00:25:17.940 Awaish Kumar: then start exploring Shopify Recharge data, so I’ll see if…

215 00:25:18.380 ⇒ 00:25:21.260 Awaish Kumar: I should add the tickets for those as well, maybe.

216 00:25:40.070 ⇒ 00:25:44.979 Awaish Kumar: Hmm… Yeah, so that’s it for Element.

217 00:25:45.530 ⇒ 00:25:50.939 Awaish Kumar: I will be off Wednesday, so for 2 days, I think that’s enough, and we can…

218 00:25:51.320 ⇒ 00:25:56.550 Awaish Kumar: deliver this by the… before I leave, and yeah, we’ll be good here.

219 00:25:59.900 ⇒ 00:26:04.980 Awaish Kumar: Apart from that, we can move on to… Heddraw?

220 00:26:05.890 ⇒ 00:26:10.570 Awaish Kumar: For Hydra, I’ve… Yeah, so…

221 00:26:12.170 ⇒ 00:26:22.850 Awaish Kumar: This is one of the tickets which I have to work on, but that depends on the syncing of the data. So for this week, we are mostly focusing on either rewriting

222 00:26:26.100 ⇒ 00:26:37.020 Awaish Kumar: Rewriting of… some models… revenue models, to lose…

223 00:26:38.830 ⇒ 00:26:46.720 Awaish Kumar: Invoice line items, or we maybe fix, Syncing issue.

224 00:26:47.840 ⇒ 00:26:48.510 Awaish Kumar: What?

225 00:26:49.290 ⇒ 00:26:50.770 Awaish Kumar: In my stable.

226 00:26:56.070 ⇒ 00:27:02.860 Awaish Kumar: I will work on that, and then this one, and yeah, that’s it for this week as well, for eDrop.

227 00:27:04.170 ⇒ 00:27:10.429 Awaish Kumar: through CDA, we… Yeah, actually, you can go on.

228 00:27:10.660 ⇒ 00:27:12.059 Awaish Kumar: I’ll give you updates.

229 00:27:12.660 ⇒ 00:27:19.420 Ashwini Sharma: Yeah, for CT, I’m working on the CICD thing. The other things are done, sort of, the training on dbt was done.

230 00:27:19.690 ⇒ 00:27:24.969 Ashwini Sharma: And what else? Yeah. Only CICT…

231 00:27:25.100 ⇒ 00:27:27.630 Awaish Kumar: Part is remaining. Yeah, that is also done.

232 00:27:29.490 ⇒ 00:27:30.760 Awaish Kumar: City of 14…

233 00:27:31.010 ⇒ 00:27:38.189 Ashwini Sharma: No, 15 is done, yeah, CICT is remaining.

234 00:27:38.840 ⇒ 00:27:39.300 Awaish Kumar: There we go.

235 00:27:39.300 ⇒ 00:27:41.970 Ashwini Sharma: I don’t know what is to be done in Gantt chart.

236 00:27:42.160 ⇒ 00:27:50.030 Awaish Kumar: Ganttar is that, like, the kind of… all the tickets which you have in pipeline that you want to do in the next few weeks.

237 00:27:50.230 ⇒ 00:27:58.980 Awaish Kumar: Or, like, as a team, as a whole, like, what you… I wonder, like, at least on DataWorks team, what we want to achieve in a month, or…

238 00:27:59.390 ⇒ 00:28:01.419 Awaish Kumar: Or do whatever you know.

239 00:28:01.600 ⇒ 00:28:02.430 Ashwini Sharma: Okay.

240 00:28:02.430 ⇒ 00:28:05.930 Awaish Kumar: Creating all those different milestones and tickets.

241 00:28:10.090 ⇒ 00:28:12.940 Ashwini Sharma: Alright, yeah, I’ll try to clean that up.

242 00:28:13.380 ⇒ 00:28:14.810 Awaish Kumar: Oh, yeah.

243 00:28:14.810 ⇒ 00:28:15.889 Ashwini Sharma: And.

244 00:28:15.890 ⇒ 00:28:20.750 Awaish Kumar: You, and, yeah, how many story points do you think?

245 00:28:21.500 ⇒ 00:28:24.000 Ashwini Sharma: Yeah, this is, this is a lot more,

246 00:28:28.110 ⇒ 00:28:31.059 Ashwini Sharma: Yeah, it’s taking me an entire day, and put it 7.

247 00:28:32.520 ⇒ 00:28:33.360 Awaish Kumar: Okay.

248 00:28:34.110 ⇒ 00:28:38.250 Awaish Kumar: Is it… is it Is it that complex, or…

249 00:28:38.250 ⇒ 00:28:41.540 Ashwini Sharma: It doesn’t work simply, with Snowflake.

250 00:28:42.140 ⇒ 00:28:44.869 Awaish Kumar: Okay, I might be doing the same thing today.

251 00:28:45.050 ⇒ 00:28:49.270 Ashwini Sharma: Yeah, we can utilize… Whatever I’ve done here.

252 00:28:52.190 ⇒ 00:28:53.900 Awaish Kumar: Okay, let’s see,

253 00:28:55.080 ⇒ 00:29:04.599 Awaish Kumar: how… how it goes. I have to do it for the element, so I will check how it goes, and maybe we can then share the learnings also. Sure.

254 00:29:04.970 ⇒ 00:29:09.589 Awaish Kumar: Apart from that, what else you’re working on?

255 00:29:10.240 ⇒ 00:29:10.890 Awaish Kumar: Thank you.

256 00:29:10.890 ⇒ 00:29:18.340 Ashwini Sharma: Nothing else on CTA right now. We have to figure the work out. Last time, I had a discussion with Catherine and Kyle.

257 00:29:19.030 ⇒ 00:29:23.330 Ashwini Sharma: They said they will, they will talk, on Monday regarding some new work.

258 00:29:27.500 ⇒ 00:29:31.300 Awaish Kumar: So right now, we only ingested re-remember data, and…

259 00:29:31.300 ⇒ 00:29:32.960 Ashwini Sharma: the remember data is there.

260 00:29:33.220 ⇒ 00:29:34.350 Ashwini Sharma: And.

261 00:29:35.370 ⇒ 00:29:37.579 Awaish Kumar: Does it require any modeling or anything?

262 00:29:37.950 ⇒ 00:29:43.470 Ashwini Sharma: Yeah, so initial modeling is already done, the basic modeling, right, just to create one report.

263 00:29:43.820 ⇒ 00:29:50.540 Ashwini Sharma: The other modeling will be done as we get more requirements, like, on exactly what.

264 00:29:50.540 ⇒ 00:29:52.980 Awaish Kumar: Kind of good ideas. Sorry?

265 00:29:53.540 ⇒ 00:30:00.549 Awaish Kumar: what kind of data we are getting from Remember? Is it, like, like, revenue, customers.

266 00:30:00.550 ⇒ 00:30:03.660 Ashwini Sharma: There is… there is CRM data.

267 00:30:03.660 ⇒ 00:30:04.310 Awaish Kumar: Okay.

268 00:30:05.290 ⇒ 00:30:09.820 Ashwini Sharma: Merely, it’s CRM, because that was the most important, this thing.

269 00:30:10.020 ⇒ 00:30:14.909 Ashwini Sharma: Otherwise, there is… there is some amount of accounting data It.

270 00:30:15.650 ⇒ 00:30:23.850 Awaish Kumar: Yeah, like, I can add a few tickets which might be useful, like, modeling for…

271 00:30:24.280 ⇒ 00:30:28.320 Awaish Kumar: like, customer mark, who’s from CRM, at least you can…

272 00:30:28.550 ⇒ 00:30:31.180 Awaish Kumar: Create some DIM customer kind of thing.

273 00:30:34.330 ⇒ 00:30:42.900 Awaish Kumar: Which have all the fee… all the trades related to a customer at one place. And similarly, we can have something

274 00:30:45.760 ⇒ 00:30:52.019 Awaish Kumar: Something related to… Finance mod, which is, like, accounting data, or…

275 00:30:52.200 ⇒ 00:30:56.019 Ashwini Sharma: Yeah, there is accounting data, there is.

276 00:30:56.180 ⇒ 00:30:57.079 Awaish Kumar: I don’t know how…

277 00:30:58.440 ⇒ 00:31:05.499 Awaish Kumar: how useful that is, what kind of data you have, but, we can think about it, like, if it makes sense, we can

278 00:31:05.960 ⇒ 00:31:08.310 Awaish Kumar: Create those, modeling.

279 00:31:09.520 ⇒ 00:31:15.650 Ashwini Sharma: No, I would wait and understand what KPIs they look into before doing this modeling.

280 00:31:16.320 ⇒ 00:31:29.119 Awaish Kumar: Yeah, there are two things, like, we… yeah, obviously, we look… look at what KPIs they are looking, and we are also free to recommend, like, if you, like, I’m not saying start modeling, but it’s like, you explore the data.

281 00:31:29.240 ⇒ 00:31:37.259 Awaish Kumar: And then you… you can recommend to the client, like, this is how you can measure ABC or XYZ,

282 00:31:37.680 ⇒ 00:31:41.649 Awaish Kumar: And then, if they say yes, we start modeling otherwise.

283 00:31:42.300 ⇒ 00:31:44.720 Awaish Kumar: We just put it in backlog.

284 00:31:45.380 ⇒ 00:31:46.670 Ashwini Sharma: Okay, alright.

285 00:31:51.780 ⇒ 00:31:55.269 Awaish Kumar: Okay, so let’s see whatever the result is.

286 00:31:55.550 ⇒ 00:32:00.740 Awaish Kumar: From your client meeting, and then we can put up more tickets Tomorrow?

287 00:32:01.740 ⇒ 00:32:08.559 Ashwini Sharma: There is a big project that is coming up, right? They have some kind of a reporting that is done on Power BI,

288 00:32:08.710 ⇒ 00:32:16.860 Ashwini Sharma: based on SQL Server data and several other data sources, and they want us to do the part that is doable using November’s data. So…

289 00:32:17.210 ⇒ 00:32:22.890 Ashwini Sharma: Yeah, we’ll be talking about that, and then maybe start modeling on that topic.

290 00:32:23.390 ⇒ 00:32:29.070 Awaish Kumar: Okay, so we know what model… what visualization tool we will be using for that?

291 00:32:29.070 ⇒ 00:32:31.149 Ashwini Sharma: No, not yet. Not yet.

292 00:32:32.200 ⇒ 00:32:32.930 Awaish Kumar: Okay.

293 00:32:34.050 ⇒ 00:32:37.709 Awaish Kumar: Will we be recommending anything, or they have already something?

294 00:32:39.120 ⇒ 00:32:47.650 Ashwini Sharma: I don’t have anything to recommend, like, they already have a Power BI or something, right? So, maybe they’ll utilize that, but I’ve heard

295 00:32:47.820 ⇒ 00:32:51.160 Ashwini Sharma: Catherine talked about Sigma also one or two times, so…

296 00:32:51.160 ⇒ 00:32:53.099 Uttam Kumaran: Yeah, so they, they’re…

297 00:32:53.210 ⇒ 00:32:59.790 Uttam Kumaran: Yeah, in January or February, they’re going to think about a potential… another solution for BI.

298 00:33:00.370 ⇒ 00:33:06.689 Uttam Kumaran: It’s not clear right now, like, what the requirements are, but they’re definitely interested in considering something.

299 00:33:07.520 ⇒ 00:33:13.930 Awaish Kumar: Okay, so do you want to start any kind of spikes or assessment of these tools, or…

300 00:33:14.700 ⇒ 00:33:20.100 Uttam Kumaran: No, I think for now, I think the… the priorities, Ashwini are just…

301 00:33:20.290 ⇒ 00:33:26.960 Uttam Kumaran: kind of, like, ramping up Kyle and the data, and basically the things that she mentioned on Friday.

302 00:33:27.080 ⇒ 00:33:30.050 Uttam Kumaran: Like, that new, requirement set, right?

303 00:33:30.270 ⇒ 00:33:31.210 Ashwini Sharma: Yeah.

304 00:33:32.880 ⇒ 00:33:35.620 Uttam Kumaran: Is what she mentioned on Friday pretty clear?

305 00:33:38.010 ⇒ 00:33:44.489 Ashwini Sharma: Yeah, at least I know what needs to be done, basically. Like, they have a couple of data sources, and we need to create that

306 00:33:44.890 ⇒ 00:33:45.959 Ashwini Sharma: Report that.

307 00:33:45.960 ⇒ 00:33:46.510 Uttam Kumaran: diagram.

308 00:33:46.510 ⇒ 00:33:48.569 Ashwini Sharma: are utilizing, yeah, the diagram, yeah.

309 00:33:48.840 ⇒ 00:33:52.220 Ashwini Sharma: And, yeah, so I’ll have a chat with…

310 00:33:52.220 ⇒ 00:34:01.849 Uttam Kumaran: Pushing on that, I’ll end up pushing the BI tool when the time is right, and then let you know, so… I feel like we’re… yeah, go ahead.

311 00:34:02.220 ⇒ 00:34:07.709 Awaish Kumar: But we are working right now, like, we are just working on CICD stuff. Is that all we need to do this week?

312 00:34:07.710 ⇒ 00:34:09.920 Uttam Kumaran: No, we’re working on modeling, also.

313 00:34:09.920 ⇒ 00:34:17.339 Ashwini Sharma: Yeah, we’ll do modeling, yeah. I’ll have a chat with this guy, Kyle, today, and get more information on what to model.

314 00:34:19.820 ⇒ 00:34:24.310 Awaish Kumar: what modeling, kind of, you’re working… you are… you will be… you might be working?

315 00:34:26.420 ⇒ 00:34:27.320 Ashwini Sharma: Hmm.

316 00:34:29.090 ⇒ 00:34:32.500 Ashwini Sharma: Let me… I’m just trying to recollect the name.

317 00:34:33.370 ⇒ 00:34:34.620 Awaish Kumar: It… okay.

318 00:34:38.629 ⇒ 00:34:44.219 Ashwini Sharma: Yeah, can you just create a dummy ticket over there, and then placeholder ticket, I’ll add the details, what needs to be done.

319 00:34:44.249 ⇒ 00:34:46.149 Awaish Kumar: Like, what should be the title?

320 00:34:46.900 ⇒ 00:34:49.510 Ashwini Sharma: Modeling for new report, or something.

321 00:34:52.590 ⇒ 00:34:53.999 Awaish Kumar: Okay, please add.

322 00:34:55.239 ⇒ 00:34:56.709 Ashwini Sharma: Yeah, I’ll add the details.

323 00:34:56.710 ⇒ 00:34:57.640 Awaish Kumar: Okay.

324 00:35:01.260 ⇒ 00:35:05.849 Awaish Kumar: Okay, then… yeah, we almost discussed all the data clients.

325 00:35:08.130 ⇒ 00:35:10.649 Awaish Kumar: Tom, like, did you want to go over again, or…

326 00:35:11.000 ⇒ 00:35:20.470 Uttam Kumaran: No, I think the biggest thing is Element and, default are going to be out. So, as much as we can continue on our own, let’s continue.

327 00:35:20.650 ⇒ 00:35:24.400 Uttam Kumaran: Did you guys… its tasks there seem pretty clear?

328 00:35:24.990 ⇒ 00:35:37.449 Awaish Kumar: So these are the ones I created for Element. So, we want to set up, like, Snowflake, set up the dbt project, start ingestion for Shopify and recharge, and maybe export some data for…

329 00:35:37.570 ⇒ 00:35:39.030 Awaish Kumar: For future modeling.

330 00:35:39.670 ⇒ 00:35:40.370 Uttam Kumaran: Perfect.

331 00:35:40.630 ⇒ 00:35:44.339 Uttam Kumaran: And yeah, setting up CICD along with the dbt. So these are kind of… Perfect.

332 00:35:44.340 ⇒ 00:35:47.330 Awaish Kumar: Okay, so I will be hoping to close this.

333 00:35:47.560 ⇒ 00:35:48.310 Awaish Kumar: Okay.

334 00:35:48.490 ⇒ 00:35:49.270 Uttam Kumaran: Okay, great.

335 00:35:50.030 ⇒ 00:35:54.540 Mustafa Raja: I have a question, so if default is out, is it okay if I follow up with Thomas?

336 00:35:55.930 ⇒ 00:36:00.059 Uttam Kumaran: Yeah, you could follow up, I just… I don’t know if he’s gonna respond, though.

337 00:36:00.060 ⇒ 00:36:01.789 Mustafa Raja: Okay, I just did.

338 00:36:02.500 ⇒ 00:36:06.099 Uttam Kumaran: Okay, yeah. I feel like they may… Some people.

339 00:36:06.100 ⇒ 00:36:06.440 Demilade Agboola: Maybe.

340 00:36:06.440 ⇒ 00:36:07.870 Uttam Kumaran: John, I don’t know, but…

341 00:36:07.870 ⇒ 00:36:12.290 Demilade Agboola: Yeah, a number of people are off, because even when I tagged,

342 00:36:12.900 ⇒ 00:36:22.259 Demilade Agboola: Ryan, Laura, Stan, and Lev, to try and get a meeting this week. Basically all of them are out except Laura, so I’m supposed to meet Laura today.

343 00:36:22.600 ⇒ 00:36:22.940 Uttam Kumaran: Okay.

344 00:36:22.940 ⇒ 00:36:29.429 Demilade Agboola: the… yeah, every other person seems to be out, so I guess most of the team is out, but a couple of people here and there will be on, so…

345 00:36:29.660 ⇒ 00:36:31.160 Demilade Agboola: Yeah, I think…

346 00:36:31.620 ⇒ 00:36:38.139 Uttam Kumaran: Yeah, I feel exactly the same, and so whatever progress you guys feel like we can make in getting Omni into a good place.

347 00:36:38.300 ⇒ 00:36:40.320 Uttam Kumaran: Or getting, like.

348 00:36:40.580 ⇒ 00:36:55.379 Uttam Kumaran: you know, just getting, like, infrastructure, and then also, yeah, totally the lower category, Caitlin was very explicit in that she’s, like, a major stakeholder, so if that’s all we get to, guys, like, that’s fine.

349 00:36:55.380 ⇒ 00:37:06.649 Awaish Kumar: So, this is… Want to close this integration dashboard, then this discovery call with Laura by Demlade, then we maybe have put up some ETL plan.

350 00:37:06.970 ⇒ 00:37:09.040 Awaish Kumar: For Mustafa, we have, like.

351 00:37:09.990 ⇒ 00:37:16.710 Awaish Kumar: We’ll be focusing on getting the Salesforce data, and maybe setting up an architecture or data platform document.

352 00:37:17.340 ⇒ 00:37:17.930 Uttam Kumaran: Okay.

353 00:37:18.910 ⇒ 00:37:19.790 Uttam Kumaran: Perfect.

354 00:37:20.040 ⇒ 00:37:20.630 Awaish Kumar: Okay.

355 00:37:21.990 ⇒ 00:37:36.790 Uttam Kumaran: And then… yeah, so for… for Element, I’m also fine. I don’t know if there’s, like, any other clients, like, I… I assume you guys chatted about Eden, but other than that, like, that’s all that’s top of mind for me, so…

356 00:37:36.840 ⇒ 00:37:47.569 Uttam Kumaran: I think we have these, you know, two days or so, and then, yeah, I feel okay. Anything else, like, we… anything, like, we want to do as, like, a crew, like, in terms of…

357 00:37:48.250 ⇒ 00:37:50.440 Uttam Kumaran: Tech debt, or, like…

358 00:37:50.720 ⇒ 00:38:01.660 Uttam Kumaran: broader discussions that maybe we should have about process, like, maybe it’s also a good time this next two weeks, like, to plan out Oish, like, a couple things to chat about, like, if we wanted to…

359 00:38:01.950 ⇒ 00:38:07.020 Uttam Kumaran: if we wanted to confirm, like, the new dbt structure, we wanted to confirm, like.

360 00:38:07.570 ⇒ 00:38:11.450 Uttam Kumaran: Anything broader, I’m open to those types of calls as well.

361 00:38:12.560 ⇒ 00:38:19.700 Awaish Kumar: Okay, yeah, there, like, there is one discussion, on liberty structure started by Ashwani. We can…

362 00:38:20.010 ⇒ 00:38:21.879 Awaish Kumar: Have a discussion on that part.

363 00:38:22.520 ⇒ 00:38:25.299 Awaish Kumar: Like, that’s, like, including one more layer.

364 00:38:26.160 ⇒ 00:38:28.640 Awaish Kumar: Of, like, staging that for queuing.

365 00:38:28.920 ⇒ 00:38:36.380 Awaish Kumar: So right now, we have it for some clients, and we don’t for others. We can think about standardizing that piece.

366 00:38:36.380 ⇒ 00:38:41.600 Uttam Kumaran: Yeah, do we… do we want to do that? We can do that now, or we can host a specific meeting for that.

367 00:38:43.130 ⇒ 00:38:45.650 Awaish Kumar: Yeah, we can do that now,

368 00:38:46.100 ⇒ 00:38:52.930 Uttam Kumaran: Yeah, maybe if you… if there’s no… are there any other… other client topics to discuss? Otherwise, like, yeah, I think it’s a good use of time.

369 00:38:56.940 ⇒ 00:38:59.190 Awaish Kumar: Yeah, I don’t think there’s any time fleets.

370 00:38:59.820 ⇒ 00:39:02.509 Uttam Kumaran: Okay, cool, yeah, maybe let’s do that. Yeah.

371 00:39:02.790 ⇒ 00:39:17.009 Uttam Kumaran: maybe Ashwini, or if you guys want to send the new proposed structure, I’m not sure if Demolade has seen it, but I’d also like Mustafa and Sam to also have a quick read, and then let’s discuss. I’m familiar with what the proposal is.

372 00:39:18.790 ⇒ 00:39:22.930 Ashwini Sharma: Yeah, so, let me know if you’re able to see my screen read.

373 00:39:24.800 ⇒ 00:39:26.500 Demilade Agboola: Yeah, personal.

374 00:39:27.670 ⇒ 00:39:33.639 Ashwini Sharma: Right, yeah, so this is what I was suggesting right here, if you see the models.

375 00:39:33.890 ⇒ 00:39:34.690 Ashwini Sharma: Pardon?

376 00:39:35.100 ⇒ 00:39:36.479 Awaish Kumar: I can’t see your screen.

377 00:39:37.450 ⇒ 00:39:38.600 Ashwini Sharma: You can see my screen?

378 00:39:41.030 ⇒ 00:39:41.750 Ashwini Sharma: It’s…

379 00:39:41.750 ⇒ 00:39:43.090 Uttam Kumaran: Oh, I see, I see it.

380 00:39:43.740 ⇒ 00:39:46.600 Mustafa Raja: Yeah, it could be in the tabs, or at the top.

381 00:39:46.920 ⇒ 00:39:48.789 Demilade Agboola: Yeah, it’s probably in the tabs at the top.

382 00:39:49.200 ⇒ 00:39:50.749 Demilade Agboola: But I could see his scream.

383 00:39:52.940 ⇒ 00:39:57.100 Ashwini Sharma: Let me share only specific screen, I don’t know, it’s weird.

384 00:40:00.900 ⇒ 00:40:03.210 Awaish Kumar: Okay. How about now? Now I can see.

385 00:40:04.940 ⇒ 00:40:06.730 Ashwini Sharma: This left side panel, you can see.

386 00:40:07.280 ⇒ 00:40:07.740 Awaish Kumar: Yep.

387 00:40:09.180 ⇒ 00:40:15.599 Ashwini Sharma: All right, yeah. So, if you look into the model, right, maybe I can,

388 00:40:16.410 ⇒ 00:40:22.320 Ashwini Sharma: One second, I’ll have to share the screen, because I’ll be switching between multiple Windows,

389 00:40:24.830 ⇒ 00:40:26.820 Ashwini Sharma: Alright, let me know if it is.

390 00:40:27.660 ⇒ 00:40:32.880 Ashwini Sharma: If you can still see that, cursor screen.

391 00:40:33.480 ⇒ 00:40:34.240 Awaish Kumar: Yes.

392 00:40:36.680 ⇒ 00:40:44.450 Ashwini Sharma: Alright, go over here, right? So, you see, right now, these guys have this remembers data source, right?

393 00:40:44.550 ⇒ 00:40:55.619 Ashwini Sharma: And if you see, the remembers data source is sort of broken down into functional areas, right? There is a set of tables, or views for accounting, and then there are certain other tables for CRM.

394 00:40:55.890 ⇒ 00:40:59.499 Ashwini Sharma: And then, similarly, there is something for shopping, and so on, right?

395 00:41:00.150 ⇒ 00:41:01.420 Ashwini Sharma: Now,

396 00:41:02.250 ⇒ 00:41:11.609 Ashwini Sharma: Tomorrow, there may be another data source, Shopify. You know, Shopify may have its own customer-based data, right? Its own orders or purchase-related data.

397 00:41:12.060 ⇒ 00:41:22.030 Ashwini Sharma: procurement data, and so on, right? And then there might be thirds, or C events, or something like that, which may have event-related data. And again, it will have its own customer base, and so on, right?

398 00:41:22.230 ⇒ 00:41:25.050 Ashwini Sharma: So, what happens is, like.

399 00:41:25.780 ⇒ 00:41:35.929 Ashwini Sharma: you know, our models get… keep on getting complicated, right? It doesn’t have to be that way. So, what I’m suggesting is, right now, we just have intermediate and mart layer.

400 00:41:36.590 ⇒ 00:41:40.300 Ashwini Sharma: For each of the 3 layers of schemas.

401 00:41:41.140 ⇒ 00:41:50.830 Ashwini Sharma: What I was suggesting is, like, having one more, right, which is the staging layer. And what I’ve done in staging is, like, I’ve tried to mimic whatever’s there in the database.

402 00:41:50.980 ⇒ 00:41:54.989 Ashwini Sharma: And if you see any one of these right here, let’s look at CRM,

403 00:41:55.820 ⇒ 00:42:01.280 Ashwini Sharma: For example, like, let’s look at what I had worked on something over here, customer, right?

404 00:42:01.880 ⇒ 00:42:06.320 Ashwini Sharma: So, what I’m trying to do in the staging layer is ensuring that

405 00:42:07.190 ⇒ 00:42:13.030 Ashwini Sharma: the different, names follow a certain standard. For example, it’s all snake case.

406 00:42:13.500 ⇒ 00:42:22.710 Ashwini Sharma: Right now. And we also have, like, as we move up in the hierarchy of these models, or down, whatever way you think of it.

407 00:42:23.350 ⇒ 00:42:30.619 Ashwini Sharma: we need to be sure that a certain column is of a certain type, right? Let’s not rely on Snowflake to interpret.

408 00:42:30.870 ⇒ 00:42:34.250 Ashwini Sharma: It’s data type, and then project it,

409 00:42:34.580 ⇒ 00:42:39.479 Ashwini Sharma: in the downstream model. So, basically, the typecasting is done right over here.

410 00:42:41.390 ⇒ 00:42:46.219 Ashwini Sharma: So the staging does this, and what we can do in the intermediate is

411 00:42:46.350 ⇒ 00:43:00.680 Ashwini Sharma: basically group together the staging models, and then create unified entities, right? Or maybe, like, if we have to do some kind of a joins to create a unified entity, we can do that. And ultimately, in the Mart layer, it’ll all be facts and dimensions.

412 00:43:01.340 ⇒ 00:43:13.130 Ashwini Sharma: Right? And in this particular case, there was, reports also, right? Reports do not fall into the category of facts or dimensions, right? But they utilize both of them. So I created a different schema called reports.

413 00:43:13.250 ⇒ 00:43:18.539 Ashwini Sharma: And… It’s just dependent on these two, right, dimensions and facts.

414 00:43:20.020 ⇒ 00:43:24.280 Ashwini Sharma: So, yeah, I mean, all I’m suggesting is having one more layer, which…

415 00:43:24.740 ⇒ 00:43:30.160 Ashwini Sharma: Will be used for some sort of cleaning, renaming, aliasing purposes.

416 00:43:30.440 ⇒ 00:43:36.620 Ashwini Sharma: Typecasting purposes, and then the rest of the model follows more or less the same structure.

417 00:43:43.030 ⇒ 00:43:44.070 Demilade Agboola: I mean…

418 00:43:44.880 ⇒ 00:43:53.880 Demilade Agboola: to be fair, like, this is kind of how I, like, mentally think of dbt and how I tend to write my models. I will say, though, that, like, it does…

419 00:43:54.010 ⇒ 00:44:00.399 Demilade Agboola: Some of the, like, the things can be done in the raw models.

420 00:44:00.920 ⇒ 00:44:09.380 Demilade Agboola: Obviously, the problem is that, like, you want your URL models to be one-to-one with the database, so you can make… you can easily compare.

421 00:44:09.580 ⇒ 00:44:12.719 Demilade Agboola: And see, if there’s anything going on.

422 00:44:12.890 ⇒ 00:44:14.170 Demilade Agboola: as well.

423 00:44:14.490 ⇒ 00:44:26.309 Demilade Agboola: I think I like this, personally. I think it works, but to be fair, like, I can also use the current way, where people, like, just go raw into intermediate, and then mods.

424 00:44:26.580 ⇒ 00:44:31.939 Demilade Agboola: I do feel like staging does help make things a bit cleaner.

425 00:44:32.190 ⇒ 00:44:41.350 Demilade Agboola: And it’d be easier to figure out, certain parts of it. As well as also, in certain models, you can start to…

426 00:44:41.550 ⇒ 00:44:43.279 Demilade Agboola: You can actually make

427 00:44:43.730 ⇒ 00:44:56.650 Demilade Agboola: mini joins, I guess, and have, like, more complete staging. Like, you can have a base layer within it as well. So there are, like, certain advantages to handling certain complexities with

428 00:44:56.860 ⇒ 00:45:03.370 Demilade Agboola: Models, that this allows us that, you know, just having… going from raw to intermediate doesn’t always give us

429 00:45:05.930 ⇒ 00:45:14.029 Demilade Agboola: It also depends on the time, though, but if you’re trying to factor time in, I guess that could also be the downside of using it this way.

430 00:45:16.260 ⇒ 00:45:21.220 Uttam Kumaran: In this case, do we have… is there still a raw… Area, or no?

431 00:45:23.390 ⇒ 00:45:35.060 Ashwini Sharma: Yeah, there is no raw area. So, raw is where you land the data, right? Where the, let’s say, polyatomic, where polyatomic will create tables. That’s the raw area, right?

432 00:45:35.060 ⇒ 00:45:36.730 Uttam Kumaran: It’s truly external.

433 00:45:38.180 ⇒ 00:45:41.799 Uttam Kumaran: So, raw never shows up in the codebase.

434 00:45:42.500 ⇒ 00:45:47.870 Demilade Agboola: Yeah, so the… the way… if you look at some of the models that we have.

435 00:45:48.030 ⇒ 00:45:51.959 Demilade Agboola: the way Raw is named is literally just,

436 00:45:52.120 ⇒ 00:45:55.559 Demilade Agboola: It’s technically a staging, it’s just a different name.

437 00:45:55.560 ⇒ 00:45:56.140 Awaish Kumar: Yep.

438 00:45:56.810 ⇒ 00:45:57.710 Ashwini Sharma: Okay.

439 00:45:58.370 ⇒ 00:46:00.079 Demilade Agboola: You know, it’s just a different name.

440 00:46:00.970 ⇒ 00:46:06.949 Awaish Kumar: Yeah, in raw, we normally do these basic cleanups that you are trying to do, but we are just calling it raw.

441 00:46:09.330 ⇒ 00:46:17.739 Ashwini Sharma: Because when you call something as raw, it literally should be raw, right? I agree. Without any manipulations.

442 00:46:19.410 ⇒ 00:46:22.100 Uttam Kumaran: And then, so, can you go… yeah, go ahead.

443 00:46:22.540 ⇒ 00:46:33.119 Demilade Agboola: I was gonna say, yeah, I totally agree with that, because, like, the idea of a raw is that, like, there should be a one-to-one map into, like, how things are in the database, or wherever the source is.

444 00:46:38.060 ⇒ 00:46:39.619 Awaish Kumar: It was something, Tom?

445 00:46:40.860 ⇒ 00:46:44.840 Uttam Kumaran: Yeah, can I see the staging, in cursor again?

446 00:46:45.030 ⇒ 00:46:45.760 Ashwini Sharma: Sure.

447 00:46:50.630 ⇒ 00:46:55.160 Uttam Kumaran: So this is a placeholder when we have Salesforce Marketing Cloud data.

448 00:46:55.420 ⇒ 00:46:57.280 Ashwini Sharma: Yeah. But, yeah.

449 00:46:58.030 ⇒ 00:47:06.329 Uttam Kumaran: So let’s just click on… okay, so this is one of them. So, for example, here you have Stage, Remember, CRM, and…

450 00:47:08.260 ⇒ 00:47:14.109 Uttam Kumaran: it’s pulling this from C source, so I guess where is C… where is this, like, C source?

451 00:47:15.380 ⇒ 00:47:18.170 Ashwini Sharma: Okay, this is a CTE, it’s defined.

452 00:47:18.170 ⇒ 00:47:20.450 Uttam Kumaran: Oh, sorry, sorry, I didn’t see this. Okay, okay, never mind, never mind.

453 00:47:21.010 ⇒ 00:47:27.240 Uttam Kumaran: So, CRM raw… I see, okay, so… Okay, so stay, so, so…

454 00:47:27.250 ⇒ 00:47:29.740 Ashwini Sharma: Yeah. Yeah, so staging becomes…

455 00:47:30.180 ⇒ 00:47:36.449 Uttam Kumaran: raw. Like, if I was to… if I was to just explain it, because me and Awash…

456 00:47:36.810 ⇒ 00:47:43.260 Uttam Kumaran: And Devilan, I’ve just been seeing it the other day. So, right, so basically what happens is staging becomes raw in dbt.

457 00:47:43.730 ⇒ 00:47:50.750 Uttam Kumaran: Raw, like, basically raw and Snowflake remains the same.

458 00:47:51.020 ⇒ 00:47:55.869 Uttam Kumaran: And then we, we replaced staging with intermediate.

459 00:47:57.110 ⇒ 00:47:58.260 Uttam Kumaran: in DVT.

460 00:47:58.390 ⇒ 00:48:00.009 Uttam Kumaran: Okay, okay, alright.

461 00:48:00.150 ⇒ 00:48:00.789 Awaish Kumar: But the only…

462 00:48:00.790 ⇒ 00:48:02.339 Uttam Kumaran: That, that makes… yeah, go ahead.

463 00:48:02.520 ⇒ 00:48:04.720 Awaish Kumar: I’m only confused with the naming.

464 00:48:04.830 ⇒ 00:48:11.280 Awaish Kumar: So we have… now we will have staging as a stage, and then we also have staging as environment.

465 00:48:11.280 ⇒ 00:48:14.690 Uttam Kumaran: I agree. I also would like to discuss that.

466 00:48:14.690 ⇒ 00:48:15.080 Ashwini Sharma: Yes.

467 00:48:15.690 ⇒ 00:48:23.029 Ashwini Sharma: That was the other thing that I wanted to discuss, right? Because it’s not staging exactly, it’s sort of a QC environment, right?

468 00:48:23.160 ⇒ 00:48:25.620 Ashwini Sharma: Maybe all…

469 00:48:25.910 ⇒ 00:48:35.380 Ashwini Sharma: need that QC environment, but generally, like, what will happen is, right now we are consultants, maybe one or two developers from our team is going to work for the client, right?

470 00:48:35.620 ⇒ 00:48:40.760 Ashwini Sharma: But eventually, at one point of time, client will involve their own data team to work on it.

471 00:48:41.560 ⇒ 00:48:58.620 Ashwini Sharma: And whenever that happens, like, the way it works is, like, for example, I’m working on a PR, you are working on a PR, and then 3 other developers are all working on the same PR, right? When they… not in the same PR, but with the same codebase, and when all of them raise different, different PRs.

472 00:48:58.920 ⇒ 00:49:05.259 Ashwini Sharma: It kind of, you know, executes the job, On the same database.

473 00:49:05.540 ⇒ 00:49:06.789 Ashwini Sharma: Nothing pinks up.

474 00:49:09.860 ⇒ 00:49:10.520 Uttam Kumaran: Yeah.

475 00:49:10.680 ⇒ 00:49:19.339 Ashwini Sharma: Right? So, we need a mechanism where, you know, things can be separated, right? Like, my changes should be.

476 00:49:19.340 ⇒ 00:49:21.110 Uttam Kumaran: I mean, people call it PrEP.

477 00:49:21.360 ⇒ 00:49:22.550 Uttam Kumaran: Also, I feel like.

478 00:49:22.550 ⇒ 00:49:32.470 Ashwini Sharma: And the way that I wanted to do was more or less like this, right? I wanted to create a database on the fly, right, with a GitHub PR number.

479 00:49:32.630 ⇒ 00:49:35.420 Ashwini Sharma: And when we create a database over there.

480 00:49:35.630 ⇒ 00:49:39.270 Ashwini Sharma: what I do is, like, you know, run the dbt.

481 00:49:39.650 ⇒ 00:49:46.900 Ashwini Sharma: on that database, create tables, or whatever it is, for the PR testing purpose, right? And then destroy it once we are done.

482 00:49:48.820 ⇒ 00:50:02.559 Ashwini Sharma: And for the QC-related thing, right, when PRs are merged to a certain state where customer wants to look at or analyze the data and then say, okay, things are good or not good.

483 00:50:02.820 ⇒ 00:50:06.829 Ashwini Sharma: That’s… Where we create a different database.

484 00:50:06.980 ⇒ 00:50:09.869 Ashwini Sharma: And then, we let the DVD run.

485 00:50:10.220 ⇒ 00:50:14.260 Ashwini Sharma: The models on top of it, and the customer would analyze that data.

486 00:50:14.400 ⇒ 00:50:17.030 Ashwini Sharma: And that should be kept separate from production, right?

487 00:50:20.470 ⇒ 00:50:29.930 Uttam Kumaran: So, check out the, links I sent, Ashwini, if you want to… can you open that up for everyone to see? So this is, so I took inspiration when we initially

488 00:50:30.080 ⇒ 00:50:32.819 Uttam Kumaran: created our dbt.

489 00:50:33.460 ⇒ 00:50:40.580 Uttam Kumaran: I took inspiration from both GitLab and dbt itself. Like, the company dbt has a dbt project.

490 00:50:40.840 ⇒ 00:50:43.310 Uttam Kumaran: And so…

491 00:50:43.510 ⇒ 00:50:48.530 Uttam Kumaran: Yeah, we can’t… yeah, exactly. So, if you go into this, like, take a look at how they do it.

492 00:50:48.770 ⇒ 00:50:51.469 Uttam Kumaran: If you scroll up to the top, there’s, like, a little summary.

493 00:50:54.980 ⇒ 00:50:56.879 Uttam Kumaran: So, see, they call it PrEP.

494 00:51:02.350 ⇒ 00:51:04.150 Uttam Kumaran: But they also call it staging.

495 00:51:04.150 ⇒ 00:51:08.289 Samuel Roberts: I was just gonna ask, I wasn’t sure if I misreading this table or not.

496 00:51:08.290 ⇒ 00:51:12.669 Uttam Kumaran: No, they also call it staging, but I think it’s, like,

497 00:51:13.210 ⇒ 00:51:19.840 Uttam Kumaran: Yeah, they also call it staging. There’s this, but there’s also, like, development environments. So yeah, I mean.

498 00:51:20.370 ⇒ 00:51:24.049 Uttam Kumaran: This is one way of… Of doing it, you know?

499 00:51:26.620 ⇒ 00:51:35.060 Samuel Roberts: as a, like, somewhat external, like, take on this, if we could avoid staging and use prep, and it’s all the same in the way Ashwini talked about it.

500 00:51:35.060 ⇒ 00:51:35.430 Uttam Kumaran: one.

501 00:51:35.430 ⇒ 00:51:38.180 Samuel Roberts: that makes the most sense. Like, the…

502 00:51:39.050 ⇒ 00:51:46.200 Samuel Roberts: Keeping the name separate makes sense, even if it’s doing exactly the same thing, and if there is another term and we’re not using all of these layers already.

503 00:51:46.420 ⇒ 00:51:49.079 Samuel Roberts: I would say that makes sense to me, but…

504 00:51:49.190 ⇒ 00:51:51.960 Samuel Roberts: That’s a little external, so don’t worry about it too much.

505 00:51:53.940 ⇒ 00:51:58.329 Ashwini Sharma: And there is one more thing that I wanted to highlight, right, which is…

506 00:51:59.050 ⇒ 00:52:02.309 Ashwini Sharma: Let me see if… do we have that?

507 00:52:04.470 ⇒ 00:52:05.390 Ashwini Sharma: Yeah.

508 00:52:10.370 ⇒ 00:52:11.670 Ashwini Sharma: Oh, it’s not.

509 00:52:23.950 ⇒ 00:52:25.429 Ashwini Sharma: Maybe I can shoot.

510 00:52:25.560 ⇒ 00:52:31.870 Ashwini Sharma: Over here, right? So this is the Eden pipeline, right? And if we… oh, sorry, this is not Eden.

511 00:53:02.220 ⇒ 00:53:02.970 Ashwini Sharma: Okay.

512 00:53:03.170 ⇒ 00:53:09.490 Ashwini Sharma: You see this one, right? So this is what runs every time somebody pushes a PR in the

513 00:53:10.260 ⇒ 00:53:11.960 Ashwini Sharma: in the Eden codebase.

514 00:53:12.860 ⇒ 00:53:17.700 Ashwini Sharma: And what it is doing is… See this one right?

515 00:53:18.320 ⇒ 00:53:22.969 Ashwini Sharma: It’s running the models, and… It runs the entire dbt, right?

516 00:53:23.130 ⇒ 00:53:29.409 Ashwini Sharma: And when we do this on Snowflake, right, over a period of time, the bill goes really high.

517 00:53:29.980 ⇒ 00:53:32.890 Ashwini Sharma: And what, ideally, we should be doing is…

518 00:53:33.140 ⇒ 00:53:35.430 Ashwini Sharma: We should only be running models which

519 00:53:35.540 ⇒ 00:53:40.880 Ashwini Sharma: have changed, and they’re downstream dependencies, not… not anything else.

520 00:53:40.880 ⇒ 00:53:41.470 Awaish Kumar: Yes.

521 00:53:42.780 ⇒ 00:53:47.670 Ashwini Sharma: So, there is something called SlimCI in dbt.

522 00:53:48.040 ⇒ 00:53:50.579 Ashwini Sharma: We need to explore that and then see

523 00:53:50.700 ⇒ 00:53:56.649 Ashwini Sharma: what we can do. But only for PR purpose, I would say, you know, if we can just validate

524 00:53:57.030 ⇒ 00:54:06.249 Ashwini Sharma: If the schema generation from dbt works fine, then I think that should be good enough. And maybe run some basic tests out of static data.

525 00:54:13.890 ⇒ 00:54:16.849 Uttam Kumaran: I guess I’m not following this as much.

526 00:54:20.150 ⇒ 00:54:22.869 Ashwini Sharma: Okay, so, let me, no, of course.

527 00:54:23.280 ⇒ 00:54:42.289 Ashwini Sharma: it in a more… better way, right? So, think about this, right? PRs are generated, like, a lot more than what customers would like to run dbt on their models, right? It would be maybe once a day, right? Eden does it once every hour, so Eden might be an exception, but

528 00:54:42.410 ⇒ 00:54:46.409 Ashwini Sharma: Other customers might not do it at that frequency.

529 00:54:48.690 ⇒ 00:54:53.849 Uttam Kumaran: Correct. I don’t know what you have seen, but most of the big customers that I’ve seen who are running dbt.

530 00:54:53.900 ⇒ 00:54:59.460 Ashwini Sharma: they run it, like, once a day. Or maybe for selected business… selected models, it might be.

531 00:54:59.460 ⇒ 00:54:59.910 Uttam Kumaran: But you.

532 00:54:59.910 ⇒ 00:55:00.230 Ashwini Sharma: Meaning…

533 00:55:00.230 ⇒ 00:55:04.019 Uttam Kumaran: In terms of PR, or in terms of running the jobs themselves?

534 00:55:04.230 ⇒ 00:55:06.579 Ashwini Sharma: I’m trying to compare both of them, right?

535 00:55:06.800 ⇒ 00:55:12.789 Uttam Kumaran: So, for jobs, I don’t think… I think we’re gonna… we’re gonna start to see 4-hour minimum SLAs, probably.

536 00:55:13.690 ⇒ 00:55:23.620 Uttam Kumaran: Because we’re going to start working with bigger and bigger customers. So, for the most part, you can assume, like, let’s just assume the default is every 4 hours during

537 00:55:24.250 ⇒ 00:55:25.600 Uttam Kumaran: business hours.

538 00:55:28.740 ⇒ 00:55:31.860 Ashwini Sharma: Okay, and, and, let’s see…

539 00:55:32.300 ⇒ 00:55:39.199 Demilade Agboola: My question is, is it every 4 hours because they need it to be 4 hours, or is it every 4 hours because we want to give them every 4 hours?

540 00:55:39.930 ⇒ 00:55:42.479 Uttam Kumaran: I think it’s gonna be because they need it.

541 00:55:43.610 ⇒ 00:55:50.210 Uttam Kumaran: What 4 hours grants you is at least one intraday…

542 00:55:50.550 ⇒ 00:55:55.579 Uttam Kumaran: point where data is updated. My point being is that I think

543 00:55:56.030 ⇒ 00:56:01.030 Uttam Kumaran: As we get to more and more clients, a requirement of the data is going to be that

544 00:56:01.290 ⇒ 00:56:05.659 Uttam Kumaran: The… it’s both gonna be on the previous day close, and…

545 00:56:05.780 ⇒ 00:56:08.729 Uttam Kumaran: If you can get same-day information in.

546 00:56:09.040 ⇒ 00:56:15.650 Uttam Kumaran: people are going to want to report on that. Across all of our clients, I think Eden right now is the highest SLA.

547 00:56:15.760 ⇒ 00:56:18.760 Awaish Kumar: But I, I expect default and others.

548 00:56:18.760 ⇒ 00:56:21.119 Uttam Kumaran: to want to drive. Of course, as a data team.

549 00:56:21.970 ⇒ 00:56:23.759 Uttam Kumaran: We have to balance two things.

550 00:56:24.370 ⇒ 00:56:28.060 Uttam Kumaran: We always want to get the most fresh, up-to-date data.

551 00:56:28.300 ⇒ 00:56:34.959 Uttam Kumaran: We also will explain the cost and the limitations of doing that. For example, if a source only updates every 12 hours.

552 00:56:35.080 ⇒ 00:56:41.070 Uttam Kumaran: You’re only as limited as your… as your longest… SLA source, right?

553 00:56:41.300 ⇒ 00:56:52.549 Uttam Kumaran: But I don’t… I don’t find it… I feel like 4 hours is a nice round number for us to, like, try to drive towards. The problem is, though, like, if… if the…

554 00:56:52.820 ⇒ 00:56:58.720 Uttam Kumaran: If ultimately, like, data takes 12 hours to refresh from the source, then that’s our fastest.

555 00:56:59.980 ⇒ 00:57:08.169 Uttam Kumaran: refresh, right? There’s no point to rerun stuff. So, similarly, Demolati on Urban Stems is a good example.

556 00:57:08.780 ⇒ 00:57:16.730 Uttam Kumaran: We are running models there basically as slow as the slowest data refresh.

557 00:57:17.320 ⇒ 00:57:22.420 Uttam Kumaran: But for them, they wanted, like, every hour, or every, like, 2 hours, you know?

558 00:57:24.910 ⇒ 00:57:29.300 Demilade Agboola: Yeah, I mean, definitely. I do wonder, like.

559 00:57:29.690 ⇒ 00:57:37.199 Demilade Agboola: how much of the… what eat versus how much they want. And, like, for instance, like, we eat in…

560 00:57:37.400 ⇒ 00:57:42.820 Demilade Agboola: Yeah, we might actually run, say, GitHub every hour.

561 00:57:43.130 ⇒ 00:57:49.130 Demilade Agboola: Well, Tableau refreshes are, like, once a day, apart from a few extracts.

562 00:57:49.920 ⇒ 00:58:05.820 Demilade Agboola: So, ultimately, like, it does help us, and to be fair, it has been helpful in situations where, like, things break in the middle of the night. We still have some, like, data that, you know, ran maybe 2 hours prior that was fine, and we can, you know, still use that.

563 00:58:06.270 ⇒ 00:58:17.070 Demilade Agboola: But, like, in terms of, like, the speed that people want the data, like, the… it doesn’t always necessarily translate, and I know Eden hasn’t necessarily had an issue with…

564 00:58:17.360 ⇒ 00:58:21.189 Demilade Agboola: speed of data. They just need it the next morning, and everything is there.

565 00:58:21.580 ⇒ 00:58:27.670 Demilade Agboola: I don’t know about default, haven’t spoken to any of the stakeholders, I don’t know the velocity of decisions they make.

566 00:58:27.940 ⇒ 00:58:32.930 Demilade Agboola: It’s possible, yeah, it’s possible that they really need it every hour.

567 00:58:33.430 ⇒ 00:58:35.720 Demilade Agboola: I don’t know that. The thing is…

568 00:58:35.720 ⇒ 00:58:39.980 Awaish Kumar: It’s okay, like, running it twice a day, you know, in business hours.

569 00:58:40.680 ⇒ 00:58:45.840 Awaish Kumar: Like, like, some data might refresh from, like, 12.

570 00:58:46.280 ⇒ 00:58:55.950 Awaish Kumar: PM, so we don’t want to run everything at 12 p.m. We might… we can run some at beginning of the day, and then some in the middle of the day, and then we’re good, right?

571 00:58:57.250 ⇒ 00:59:08.629 Ashwini Sharma: Okay, maybe, like, I kind of, you know, didn’t put the right words, right? Let me pose a question, right? When somebody, when a developer raises a PR,

572 00:59:08.800 ⇒ 00:59:14.230 Ashwini Sharma: what do we want to do with that PR before we allow it to merge with the branch?

573 00:59:19.670 ⇒ 00:59:23.919 Demilade Agboola: Just validate the logic that is being introduced to the system.

574 00:59:25.650 ⇒ 00:59:28.909 Ashwini Sharma: And, and that is using the data, or…

575 00:59:29.100 ⇒ 00:59:30.400 Ashwini Sharma: Only the schema.

576 00:59:33.040 ⇒ 00:59:35.720 Demilade Agboola: I think both, ideally.

577 00:59:36.380 ⇒ 00:59:39.170 Demilade Agboola: You would want to be able to see any, like.

578 00:59:39.990 ⇒ 00:59:44.390 Demilade Agboola: Bad schema decisions, and if there’s bad data, you also won’t see that as well.

579 00:59:51.180 ⇒ 00:59:51.930 Uttam Kumaran: Yeah.

580 00:59:53.650 ⇒ 00:59:54.630 Uttam Kumaran: I agree.

581 00:59:55.970 ⇒ 01:00:01.580 Demilade Agboola: I mean, obviously, the first priority is schema. You don’t want anything that breaks the schemas and breaks what’s going on there.

582 01:00:01.870 ⇒ 01:00:06.489 Demilade Agboola: but also, yeah, data, because, I mean, there are… there are times when

583 01:00:06.680 ⇒ 01:00:18.939 Demilade Agboola: Especially for… on Eden, I remember there was, a PR that… actually, I sent the PR in… that had a cross… not… that had a join.

584 01:00:19.840 ⇒ 01:00:26.969 Demilade Agboola: that caused multiple rows to appear. It just caused dupes across the, the table.

585 01:00:27.480 ⇒ 01:00:29.850 Demilade Agboola: And then inflated the revenue.

586 01:00:30.700 ⇒ 01:00:38.209 Demilade Agboola: So, yeah, it’s… I guess, like, those kind of things sometimes. Like, yeah, the schema was fine, but the data was off.

587 01:00:39.450 ⇒ 01:00:42.990 Ashwini Sharma: Yeah, and do we want to test that data in the

588 01:00:43.260 ⇒ 01:00:47.199 Ashwini Sharma: In the PR schema that we create, or should we test that thing?

589 01:00:47.860 ⇒ 01:00:50.549 Ashwini Sharma: In a different schema, where we retain, because…

590 01:00:50.550 ⇒ 01:00:55.910 Uttam Kumaran: You know, ideally you should do it in a PR schema, but again, like,

591 01:00:56.580 ⇒ 01:01:04.380 Uttam Kumaran: it’s… it’s sort of up to how complicated you want to do it. Like, if you want to create a PR schema on the fly, run everything, and then drop it.

592 01:01:05.060 ⇒ 01:01:07.839 Uttam Kumaran: I feel like that is, like, the ideal.

593 01:01:08.060 ⇒ 01:01:16.040 Uttam Kumaran: But the other thing is, like, if you look at the way GitLab has it, they have, like, almost… they have also have… they end up doing, like, a pre…

594 01:01:16.230 ⇒ 01:01:18.419 Uttam Kumaran: Like, part of what they end up doing is they have, like.

595 01:01:18.840 ⇒ 01:01:26.430 Uttam Kumaran: Dev, staging, and then they do, like, basically, like, Free, pre-prod, prod.

596 01:01:26.940 ⇒ 01:01:27.680 Awaish Kumar: Yeah, for…

597 01:01:29.190 ⇒ 01:01:35.179 Awaish Kumar: Like, those… it makes sense when we have, like, pick a team of people where, like.

598 01:01:35.260 ⇒ 01:01:52.430 Awaish Kumar: So we verified the PR, we looked at the data at a developer level, pushed it, it got merged, then we have someone sitting at, like, we have Tableau reports from that, like, QA environment, then we have Tableau reports connecting to that QA environment, someone is queuing that.

599 01:01:52.500 ⇒ 01:01:56.420 Awaish Kumar: And verifying all the data looks good, and then it gets merged.

600 01:01:56.550 ⇒ 01:02:02.979 Awaish Kumar: So, like, it adds… Little bit of complexity and extra effort.

601 01:02:08.580 ⇒ 01:02:13.869 Awaish Kumar: like, kind of… to create kind of a QA environment separately, which basically having

602 01:02:14.090 ⇒ 01:02:20.010 Awaish Kumar: We have our broad data and our reports, and then we also have same QA…

603 01:02:21.000 ⇒ 01:02:23.310 Awaish Kumar: Database and, and the reports.

604 01:02:23.430 ⇒ 01:02:31.020 Awaish Kumar: Somebody has to verify and approve, then it goes to production, and then we might have some of this deployment

605 01:02:31.320 ⇒ 01:02:32.060 Awaish Kumar: Policy.

606 01:02:32.060 ⇒ 01:02:39.869 Uttam Kumaran: So here’s the big picture, is that, Ashwini, as bugs are entering the system, think about the cost

607 01:02:40.400 ⇒ 01:02:50.900 Uttam Kumaran: to the client for which it takes for us to resolve those bugs, right? If a bug enters the system and it’s a few hours, that’s potentially several hundred dollars of cost.

608 01:02:51.430 ⇒ 01:02:58.620 Uttam Kumaran: So, I would love for us to have a much more robust way of preventing bugs from entering the system.

609 01:02:58.860 ⇒ 01:03:13.900 Uttam Kumaran: And I… the cost would… it’s… it’s gonna totally make sense for them, in terms of, like, potential slowdown or complexity. For example, ideally, we have, like, basically a one… we try to have, like, a one-for-one identical staging environment to production.

610 01:03:13.940 ⇒ 01:03:23.819 Uttam Kumaran: Where if we really wanted to, we could create… we could basically have dashboards that are one-to-one comparison of staging to production. We could run

611 01:03:24.230 ⇒ 01:03:28.580 Uttam Kumaran: We can run all of our checks that we run in production on staging.

612 01:03:28.750 ⇒ 01:03:34.130 Uttam Kumaran: And we want to just really prevent bugs from entering.

613 01:03:34.250 ⇒ 01:03:35.250 Uttam Kumaran: Yeah.

614 01:03:35.360 ⇒ 01:03:38.600 Ashwini Sharma: Right, right. You know, and because the risk is so… the risk…

615 01:03:38.610 ⇒ 01:03:41.779 Uttam Kumaran: For the high, in terms of cost, is so high, and…

616 01:03:41.940 ⇒ 01:03:56.980 Uttam Kumaran: by just, like, based on experience, like, it’s the largest trust-defeating thing, is when, like, data is wrong, you know? And so, I would rather over-allocate to making sure that that doesn’t happen.

617 01:03:57.910 ⇒ 01:04:02.550 Ashwini Sharma: Yeah, maybe I can… Oh, something right.

618 01:04:03.080 ⇒ 01:04:10.799 Awaish Kumar: Yeah, but what I think, like, Ashwini is trying to say is that we have maybe harvest staging, and then…

619 01:04:11.610 ⇒ 01:04:17.549 Awaish Kumar: But then we have this pre-production thing as well.

620 01:04:19.330 ⇒ 01:04:22.840 Ashwini Sharma: So, we… okay, you’re able to see this note, right?

621 01:04:26.800 ⇒ 01:04:31.439 Ashwini Sharma: Okay, so this is what I feel works best,

622 01:04:31.720 ⇒ 01:04:40.039 Ashwini Sharma: let me know your thoughts, right? So, dev is kind of broken down into multiple components. One is, like, Ashwini underscore.

623 01:04:40.350 ⇒ 01:04:49.059 Ashwini Sharma: Right? And this is going to have all the schemas that… that… I create from laptop.

624 01:04:53.030 ⇒ 01:04:57.340 Ashwini Sharma: And then there will be Demi underscore, right, and then…

625 01:04:57.510 ⇒ 01:04:59.360 Ashwini Sharma: The same thing over here, right?

626 01:05:10.440 ⇒ 01:05:13.160 Ashwini Sharma: And then maybe, default?

627 01:05:17.580 ⇒ 01:05:24.040 Ashwini Sharma: And then we have the QC, right? QC has… Basically, default score.

628 01:05:33.100 ⇒ 01:05:34.530 Ashwini Sharma: And then broad.

629 01:05:35.990 ⇒ 01:05:37.699 Ashwini Sharma: We have all schemas.

630 01:05:41.450 ⇒ 01:05:46.310 Ashwini Sharma: Maybe QC can mimic fraud, so maybe I’ll… I’ll just remove this.

631 01:05:48.520 ⇒ 01:05:53.919 Ashwini Sharma: And… and finally, we have… Not finally, but somewhere over here.

632 01:05:54.900 ⇒ 01:05:56.890 Ashwini Sharma: We have PR branches.

633 01:05:58.170 ⇒ 01:05:58.820 Uttam Kumaran: Okay.

634 01:05:58.820 ⇒ 01:05:59.470 Ashwini Sharma: DB.

635 01:05:59.470 ⇒ 01:06:00.250 Uttam Kumaran: Okay.

636 01:06:00.250 ⇒ 01:06:04.600 Ashwini Sharma: And this is sort of like, maybe some default.

637 01:06:05.180 ⇒ 01:06:07.290 Uttam Kumaran: So you want to do a release, basically.

638 01:06:07.690 ⇒ 01:06:08.380 Ashwini Sharma: Yes.

639 01:06:15.390 ⇒ 01:06:31.199 Uttam Kumaran: Yeah, I mean, this is where, like, I think… I think it’s up to… it’s, like, up to us to decide. It adds… it adds another layer of complexity, but again, like, I’m telling you, one of the big areas where our team has lost a lot of faith is when

640 01:06:31.370 ⇒ 01:06:36.430 Uttam Kumaran: issues get entered into the system. I don’t think there’s a single other

641 01:06:36.700 ⇒ 01:06:40.610 Uttam Kumaran: Like, cause for lack for trust loss.

642 01:06:40.810 ⇒ 01:06:48.129 Uttam Kumaran: for our team so far that I’ve seen. For the most part, we’re okay, but if a bug enters and numbers are wrong.

643 01:06:48.570 ⇒ 01:07:02.879 Uttam Kumaran: man, like, it’s really, really… it just, like… and I know it’s… it’s, like, on our side, I’m like, okay, it’s just, like, every day we’ll figure this out, but for the client, I don’t know, it’s just, like, always a big risk, so… but yeah, go ahead, Demolade.

644 01:07:05.120 ⇒ 01:07:11.559 Demilade Agboola: I was going to clarify if the QC will mimic Prod, so basically it will be one-to-one with prod.

645 01:07:12.850 ⇒ 01:07:23.319 Ashwini Sharma: Yes, it, it, it, okay, so, yes and no, both. In terms of structure, it will mimic broad. It will have one-to-one, right? But…

646 01:07:23.410 ⇒ 01:07:41.010 Ashwini Sharma: Basically, in the release process, right, if somebody has deployed a PR, and the developer thinks that this PR is good to go, developer has done their testing, now it is there for customer to test, we’re going to push this, release this to QC. So QC might have additional objects, or it might have

647 01:07:41.420 ⇒ 01:07:42.889 Ashwini Sharma: modified objects.

648 01:07:43.250 ⇒ 01:07:44.590 Ashwini Sharma: Compared to broader.

649 01:07:45.630 ⇒ 01:07:48.899 Ashwini Sharma: But structure-wise, prod and QC are exactly the same.

650 01:07:49.970 ⇒ 01:07:52.000 Demilade Agboola: Okay, gotcha.

651 01:07:57.260 ⇒ 01:08:03.280 Demilade Agboola: Alright, as long as… as long as the PR branches will also, like, it gets cleaned, so, like, every single time…

652 01:08:03.280 ⇒ 01:08:09.360 Ashwini Sharma: We have to ensure that, yeah, we need to ensure that. And for Snowflake, there is a feature called Clone.

653 01:08:09.900 ⇒ 01:08:14.579 Ashwini Sharma: Which is, you know, it’s very smart, right? And what you can do is…

654 01:08:14.790 ⇒ 01:08:22.780 Ashwini Sharma: You can directly, issue a clone command, which clones a database within a fraction of seconds, so if your prod has, maybe.

655 01:08:23.060 ⇒ 01:08:29.440 Ashwini Sharma: thousands of objects. Within a few seconds, you have all those objects replicated in a PR branch.

656 01:08:29.830 ⇒ 01:08:38.399 Ashwini Sharma: And then when you run your PR test, or SlimCI, or whatever it is, right? It only works on things that you have modified.

657 01:08:38.850 ⇒ 01:08:46.090 Ashwini Sharma: it won’t touch the other objects. So, that way, like, the compute consumption that happens in Snowflake reduces significantly.

658 01:08:48.560 ⇒ 01:08:58.009 Demilade Agboola: Also, I think another thing to just be wary of is, like, Snowflake-specific… Concepts versus, like.

659 01:08:58.920 ⇒ 01:09:11.530 Demilade Agboola: reputable concepts that we want to apply across boards to every customer that comes in. So if they come in and they’re on Redshift, if they come in and they’re on BigQuery, or they come in and they’re on whatever, like, warehouse.

660 01:09:11.649 ⇒ 01:09:19.809 Demilade Agboola: Just being able to… As close as closely as possible, replicate that logic in a way that,

661 01:09:20.790 ⇒ 01:09:22.900 Demilade Agboola: Transfers over, you know.

662 01:09:24.830 ⇒ 01:09:28.699 Ashwini Sharma: Yeah, there should be something that’s common across all the different warehouses, right?

663 01:09:28.700 ⇒ 01:09:29.390 Demilade Agboola: That’s what you said.

664 01:09:29.390 ⇒ 01:09:32.669 Ashwini Sharma: There will be something that differs across each of them.

665 01:09:32.979 ⇒ 01:09:49.379 Demilade Agboola: Oh, definitely, definitely. All I’m just trying to say is, like, I don’t want us to build, like, logic that is, like, based off what Snowflake does, and Snowflake does well, which is great, but then we… when we move to another client on a different infrastructure…

666 01:09:49.380 ⇒ 01:09:55.260 Uttam Kumaran: To run a full, like, Drop, select all, create table.

667 01:09:55.640 ⇒ 01:09:56.169 Awaish Kumar: No, we got.

668 01:09:56.170 ⇒ 01:10:00.459 Demilade Agboola: Yeah, yeah, but it was just like… yeah, just like we, for instance…

669 01:10:00.770 ⇒ 01:10:02.820 Awaish Kumar: filter, right? For latest days.

670 01:10:03.080 ⇒ 01:10:05.049 Awaish Kumar: Only run for the last 7 days.

671 01:10:05.160 ⇒ 01:10:11.090 Awaish Kumar: And because at the PR level, now we are not testing the data, we are just testing that it works.

672 01:10:13.840 ⇒ 01:10:21.130 Demilade Agboola: Oh yeah, but just… yes, for that, but I’m actually just speaking to the entire concept of what we’re trying to build now.

673 01:10:21.720 ⇒ 01:10:25.069 Demilade Agboola: like, not just PRs, but, like, just everything.

674 01:10:25.440 ⇒ 01:10:36.380 Demilade Agboola: and at least if we know, like, we should also be aware of, like, where the slight variations come in. So, for, you know, Snowflake, this is how things go for BigQuery.

675 01:10:36.920 ⇒ 01:10:41.720 Demilade Agboola: We might make a slight tweak, because, you know, they don’t… like, for instance.

676 01:10:41.800 ⇒ 01:10:57.960 Demilade Agboola: Snowflake allows us to have multiple schemas, so you have your schema, and then you have things in between. Like, you have your schemas in your higher, like, your database and their schemas, but BigQuery just gives you… you just go straight to, like, datasets, and there’s no, like, nesting again. It just goes straight to the tables.

677 01:10:58.040 ⇒ 01:11:12.710 Demilade Agboola: So just, like, those, like, little… it’s not big deals, not big things, but, like, those modifications to be aware of, like, in a BigQuery environment, this is how we handle it, versus a Snowflake environment where it’s handled this way. That’s all I’m speaking to.

678 01:11:13.880 ⇒ 01:11:17.100 Awaish Kumar: Yeah, but, like, if we are going with this structure.

679 01:11:17.330 ⇒ 01:11:21.870 Awaish Kumar: I’m… like, I wouldn’t say we should run anything

680 01:11:22.020 ⇒ 01:11:27.289 Awaish Kumar: like, with full data in dev and PR branches, because

681 01:11:27.540 ⇒ 01:11:30.870 Awaish Kumar: Then it will be, like, just a lot of copies of data.

682 01:11:30.970 ⇒ 01:11:34.179 Awaish Kumar: then I would recommend, like, for dev and PR branches.

683 01:11:34.380 ⇒ 01:11:39.870 Awaish Kumar: environments, it will only run for, like, maybe last 7 days, so you can just test

684 01:11:40.150 ⇒ 01:11:44.180 Awaish Kumar: your code, and it runs, and that’s all. And maybe we just…

685 01:11:44.630 ⇒ 01:11:53.539 Awaish Kumar: queue a little bit on later states, like, your data looks good. Then merge it, and then the full data quality checks will be happening at QC level.

686 01:11:58.620 ⇒ 01:12:00.850 Demilade Agboola: Yeah, that’s fine. That’s definitely fine.

687 01:12:01.350 ⇒ 01:12:09.070 Awaish Kumar: Otherwise, yeah, and that we can restrict for any environment, like, blow pig, curry, snowflake, whatever. You can just say, like, if the…

688 01:12:09.700 ⇒ 01:12:11.910 Awaish Kumar: In a custom, like,

689 01:12:12.250 ⇒ 01:12:20.409 Awaish Kumar: macro that… if the… if the schema is dev or PR branches, then just run it only for last… last few days.

690 01:12:22.320 ⇒ 01:12:24.709 Awaish Kumar: Yeah, that’s how we can do that.

691 01:12:25.240 ⇒ 01:12:31.079 Ashwini Sharma: The only difference over here is this thing, right? Because you already follow this structure right now.

692 01:12:32.580 ⇒ 01:12:34.999 Ashwini Sharma: This structure is already being followed right now.

693 01:12:35.450 ⇒ 01:12:37.660 Ashwini Sharma: Right? We have 3 different layers.

694 01:12:38.740 ⇒ 01:12:40.849 Ashwini Sharma: And this is named as SDG.

695 01:12:42.560 ⇒ 01:12:55.690 Awaish Kumar: But it’s not exactly like that. We have Stagey, we have prod, we have dev, which is common across different, users. Like, one dev environment, for me, Demi, Utam, everyone, like…

696 01:12:55.820 ⇒ 01:12:58.920 Awaish Kumar: But now we are talking about having a different environment.

697 01:12:58.920 ⇒ 01:13:08.219 Ashwini Sharma: Okay, okay, okay, that’s… okay, this is what you’re saying, right? A segregation of dev environment into user-specific objects. Yeah, that is extra, right?

698 01:13:08.580 ⇒ 01:13:27.079 Awaish Kumar: So yeah, that’s what I’m saying. If I run the datasets fully for each individual user, and then also PR branches run something like that, and just have a lot of copies of data here and there, that’s what I want to avoid. At least we can… we can just stick it to… for a few days, so it does not…

699 01:13:27.260 ⇒ 01:13:29.219 Awaish Kumar: Incur the cost.

700 01:13:30.400 ⇒ 01:13:31.949 Ashwini Sharma: Yeah, yeah, we could do that.

701 01:13:41.210 ⇒ 01:13:42.729 Uttam Kumaran: So, final decision?

702 01:13:52.870 ⇒ 01:13:56.639 Demilade Agboola: At least for me, I think, like, this, this is fine, this works.

703 01:14:00.340 ⇒ 01:14:00.910 Awaish Kumar: No.

704 01:14:01.090 ⇒ 01:14:01.740 Awaish Kumar: That’s wrong.

705 01:14:01.740 ⇒ 01:14:08.919 Demilade Agboola: I think, yeah, I think we’ll, like, we won’t run as many models or run as much data, like.

706 01:14:09.810 ⇒ 01:14:10.800 Demilade Agboola: Greet this.

707 01:14:13.670 ⇒ 01:14:20.829 Ashwini Sharma: Even if we don’t go with this structure, right, I strictly suggest that we have this release cycle concept.

708 01:14:21.090 ⇒ 01:14:23.530 Ashwini Sharma: In… in the changes that we do, because…

709 01:14:23.800 ⇒ 01:14:31.100 Ashwini Sharma: Yeah, as Utam said, right, the cost of having a bug go up to prod is… It’s quite expensive.

710 01:14:33.760 ⇒ 01:14:41.349 Demilade Agboola: I will also say, though, that one of the big issues is catching An error in…

711 01:14:42.170 ⇒ 01:14:48.090 Demilade Agboola: like, a data error before it hits prod. Like, that is a huge issue in the sense that…

712 01:14:48.820 ⇒ 01:14:58.949 Demilade Agboola: Ultimately, sometimes, especially when we’re on a new project, and we’re getting used to the project, we don’t always have the full business knowledge, and so…

713 01:14:59.070 ⇒ 01:15:07.339 Demilade Agboola: sometimes… If you’re working on things in part, or in that you’re working on things in your dev environment.

714 01:15:07.650 ⇒ 01:15:21.759 Demilade Agboola: you might not always be aware that something has gone wrong. The schema is fine, everything is fine, like, the changes work, but you might have done a one-to-many join, where you expect it to be a one-to-one join.

715 01:15:21.940 ⇒ 01:15:34.350 Demilade Agboola: And so now you have, like, for instance, an explosion of data, or, like, you might have added a filter. Like, there are cases where we’ve had filters where, it doesn’t filter out null, or it’s filtering out null when it should.

716 01:15:34.470 ⇒ 01:15:37.059 Demilade Agboola: Like, those kind of, like, little things…

717 01:15:37.260 ⇒ 01:15:39.810 Demilade Agboola: But they make a change to the data.

718 01:15:40.030 ⇒ 01:15:43.830 Demilade Agboola: And they changed the number that the client sees.

719 01:15:43.830 ⇒ 01:15:44.490 Awaish Kumar: Okay.

720 01:15:44.940 ⇒ 01:15:50.580 Demilade Agboola: that can be the issue. So, like, I’m just thinking of, like, how can we get ahead of that as well, is sort of what I’.

721 01:15:50.580 ⇒ 01:16:00.220 Awaish Kumar: Can we… can we just, like, go back and write, like, come up with a proposal in writing? Say, Ashwini, if you can start a Notion Dog, and we can just all…

722 01:16:01.390 ⇒ 01:16:06.850 Awaish Kumar: Get together there and start writing with thoughts and… and… And then finalize there.

723 01:16:07.220 ⇒ 01:16:09.610 Awaish Kumar: Otherwise, we are just gonna keep talking.

724 01:16:10.950 ⇒ 01:16:12.030 Ashwini Sharma: Sure, yeah.

725 01:16:13.030 ⇒ 01:16:14.680 Ashwini Sharma: I’ll create one document.

726 01:16:16.720 ⇒ 01:16:17.460 Uttam Kumaran: Okay.

727 01:16:20.240 ⇒ 01:16:21.080 Uttam Kumaran: Cool.

728 01:16:21.180 ⇒ 01:16:23.840 Uttam Kumaran: Anything else?

729 01:16:24.620 ⇒ 01:16:26.440 Uttam Kumaran: We wanted to chat about today?

730 01:16:41.220 ⇒ 01:16:49.090 Uttam Kumaran: One thing I can also plan on doing tomorrow, guys, is maybe I can share my, like, cursor workflow tomorrow.

731 01:16:49.440 ⇒ 01:16:54.160 Uttam Kumaran: I think I would like to do with this crew, maybe, like, all of engineering I can have there.

732 01:16:54.420 ⇒ 01:16:59.270 Uttam Kumaran: maybe I’ll… I’ll send a note in the channel to maybe plan times.

733 01:16:59.410 ⇒ 01:17:02.120 Uttam Kumaran: But my schedule is pretty free.

734 01:17:02.430 ⇒ 01:17:07.709 Uttam Kumaran: I think maybe just to talk a little bit about today, we have a Magic Spoon call…

735 01:17:07.810 ⇒ 01:17:10.620 Uttam Kumaran: Sweeney and Demolade at noon.

736 01:17:11.080 ⇒ 01:17:15.109 Uttam Kumaran: Awash, I’m happy to include you there if you want to join.

737 01:17:16.430 ⇒ 01:17:17.090 Awaish Kumar: Okay.

738 01:17:17.830 ⇒ 01:17:19.109 Uttam Kumaran: Just to listen in.

739 01:17:20.850 ⇒ 01:17:21.730 Awaish Kumar: Okay, sure.

740 01:17:22.360 ⇒ 01:17:29.250 Uttam Kumaran: And… We have,

741 01:17:29.820 ⇒ 01:17:36.010 Uttam Kumaran: Yeah, I have a call with ABC later today. And then maybe, I think.

742 01:17:36.200 ⇒ 01:17:41.849 Uttam Kumaran: It would be good to try to do a delivery planning meeting Maybe later this afternoon?

743 01:17:42.000 ⇒ 01:17:45.910 Uttam Kumaran: I think as things are slowing down.

744 01:17:46.130 ⇒ 01:17:54.569 Uttam Kumaran: I want… and then I guess, Clarence, maybe it’ll be based on, like, our conversation this morning. I would like us to start to plan January.

745 01:17:54.730 ⇒ 01:17:56.150 Uttam Kumaran: In, like, Q1.

746 01:17:56.410 ⇒ 01:17:58.570 Uttam Kumaran: Or at least, like, talk about it today.

747 01:17:58.930 ⇒ 01:18:08.200 Uttam Kumaran: So that I can plan it out over the next two weeks. I have, like, a few other side projects I want to finish out, which includes Q1 planning.

748 01:18:08.410 ⇒ 01:18:15.950 Uttam Kumaran: So, maybe we can all hop on the phone, just, like, a couple of the leads here, and talk about it later today. Just, like, what Q1 is gonna look like.

749 01:18:17.100 ⇒ 01:18:21.940 Awaish Kumar: Magic Spoon, do we have… any, like, For kickoff.

750 01:18:22.110 ⇒ 01:18:24.360 Awaish Kumar: Like, do we have anything to prepare, or…

751 01:18:24.980 ⇒ 01:18:28.989 Uttam Kumaran: Yeah, I guess, Demi and Ashwini, you guys have, like, clear…

752 01:18:29.240 ⇒ 01:18:37.409 Uttam Kumaran: asks on the access side. I mean, this is mainly gonna be just, like, getting everyone, like, do another round of introductions.

753 01:18:37.590 ⇒ 01:18:39.380 Uttam Kumaran: Rehash, like, the goal.

754 01:18:40.040 ⇒ 01:18:48.770 Uttam Kumaran: And then I was really gonna hand it to you guys, you know, to drive, like, any questions. I’m happy to take note of those down and… and…

755 01:18:48.950 ⇒ 01:18:50.670 Uttam Kumaran: So that we can have an agenda.

756 01:18:55.490 ⇒ 01:18:59.740 Ashwini Sharma: But this is the one creation of pipelines using Prefect, right?

757 01:19:00.490 ⇒ 01:19:01.190 Uttam Kumaran: Yes.

758 01:19:01.190 ⇒ 01:19:07.199 Ashwini Sharma: Yeah, there are two… one work item that was assigned to me, creation of something using Perfect.

759 01:19:07.310 ⇒ 01:19:09.030 Ashwini Sharma: Let me follow that up.

760 01:19:11.130 ⇒ 01:19:14.609 Awaish Kumar: Yeah, it’s about discovery of… discovery of Sprint’s API.

761 01:19:16.670 ⇒ 01:19:19.419 Awaish Kumar: Yeah, it was pins, yes.

762 01:19:19.420 ⇒ 01:19:21.759 Ashwini Sharma: Just trying to get to where was that?

763 01:19:31.580 ⇒ 01:19:32.539 Awaish Kumar: So… Yeah.

764 01:19:32.540 ⇒ 01:19:34.659 Ashwini Sharma: Do you have a ticket for that?

765 01:19:34.660 ⇒ 01:19:38.650 Awaish Kumar: So, yeah, you can see my screen. This is basically what we had for last week.

766 01:19:38.950 ⇒ 01:19:42.880 Ashwini Sharma: Can you add me to the… Magic Spoon project, and…

767 01:19:42.880 ⇒ 01:19:44.030 Awaish Kumar: Pretty early.

768 01:19:44.030 ⇒ 01:19:44.880 Ashwini Sharma: Linear?

769 01:19:54.330 ⇒ 01:19:56.119 Ashwini Sharma: Responsive, yeah, yeah.

770 01:19:57.280 ⇒ 01:20:01.500 Ashwini Sharma: It’s not showing in my… Do I need to refresh.

771 01:20:07.530 ⇒ 01:20:09.120 Ashwini Sharma: Okay, now it is coming up.

772 01:20:19.310 ⇒ 01:20:24.100 Ashwini Sharma: Do we have some pipelines that have already been created in Prefect?

773 01:20:24.800 ⇒ 01:20:27.070 Ashwini Sharma: Within the team, for any other client?

774 01:20:31.080 ⇒ 01:20:33.020 Ashwini Sharma: No? Nobody uses this?

775 01:20:34.100 ⇒ 01:20:35.150 Awaish Kumar: Not now.

776 01:20:35.370 ⇒ 01:20:39.129 Awaish Kumar: I have worked with Prefect, but yeah, nobody uses it before.

777 01:20:43.850 ⇒ 01:20:45.950 Uttam Kumaran: Yeah, I’ve used Prefect before. I mean…

778 01:20:46.270 ⇒ 01:20:55.530 Uttam Kumaran: Again, let me… let me know if it’s something that we want, like, Mira OH to do, we can take it, but yeah, we have to basically just build the prefect pipe pipeline.

779 01:21:00.460 ⇒ 01:21:08.479 Ashwini Sharma: Okay, I’ll just log in into their system and then see their existing pipelines, how they have done it. Maybe that will be an easier way to…

780 01:21:09.160 ⇒ 01:21:09.870 Awaish Kumar: Okay.

781 01:21:09.870 ⇒ 01:21:10.820 Ashwini Sharma: You know,

782 01:21:11.880 ⇒ 01:21:17.920 Ashwini Sharma: But yeah, creation of… I don’t know, like, if it is a simple API, hit get data, write it to warehouse.

783 01:21:19.450 ⇒ 01:21:23.219 Ashwini Sharma: Might be, you know, pretty easy to do it, but if it…

784 01:21:23.530 ⇒ 01:21:25.390 Ashwini Sharma: If it gets more complex, then…

785 01:21:25.740 ⇒ 01:21:36.449 Ashwini Sharma: No, I’ve created pipelines in Fivetran, right? I know what all issues come up when we are creating these pipelines. Okay, we can’t use a tool, right? They want us to create

786 01:21:36.550 ⇒ 01:21:37.829 Ashwini Sharma: It didn’t perfect only.

787 01:21:37.830 ⇒ 01:21:52.450 Awaish Kumar: We… we don’t know about Sprint’s API yet, what is there, like, you have to discover that. We don’t… we can’t use any tool, we have to just… just figure out what endpoints to hit, and how to do pagination and stuff, and that’s all.

788 01:21:52.740 ⇒ 01:21:57.480 Awaish Kumar: And you have to write Python script, as per prefix structure.

789 01:22:00.820 ⇒ 01:22:06.030 Ashwini Sharma: Yeah, they have a GraphQL API for Atlassian, right? Atlassian spins.

790 01:22:08.180 ⇒ 01:22:10.669 Awaish Kumar: And then, yeah, it’s similar to Dexter.

791 01:22:11.000 ⇒ 01:22:14.950 Awaish Kumar: A little bit different structure, but that’s… that’s all.

792 01:22:18.680 ⇒ 01:22:21.059 Ashwini Sharma: Spins, Spence, S-P-I-N-S.

793 01:22:21.500 ⇒ 01:22:29.799 Awaish Kumar: And Demolade, like, we… We have, like, data platform documentation or anything? Like, list of sources, or some…

794 01:22:35.380 ⇒ 01:22:38.560 Demilade Agboola: For… Logic swings?

795 01:22:38.560 ⇒ 01:22:39.330 Awaish Kumar: Yep.

796 01:22:40.020 ⇒ 01:22:41.650 Demilade Agboola: Right now, not yet.

797 01:22:42.260 ⇒ 01:22:43.730 Demilade Agboola: But…

798 01:22:44.110 ⇒ 01:22:49.030 Awaish Kumar: Do you have a list of sources or something that you want to discuss in the kickoff meeting?

799 01:22:50.000 ⇒ 01:22:56.850 Demilade Agboola: I do have some questions I want to get through, just usually about, like, the…

800 01:22:58.120 ⇒ 01:23:12.719 Demilade Agboola: I had looked into the Omni, but, like, right now, I can’t get access again, because it appears every single time I need to log in, and I get logged out, I need to get the new code. So that would be an issue if we haven’t…

801 01:23:13.930 ⇒ 01:23:22.439 Demilade Agboola: That would just basically be an issue, because now it means I will be dependent on their team to send us the code to log in every single time.

802 01:23:22.810 ⇒ 01:23:38.169 Demilade Agboola: So I’m trying to see if I can get that going with Michael. But yeah, just basically, understanding the different, topics, and just understanding, like, the different dashboards and collections, and who uses what.

803 01:23:38.480 ⇒ 01:23:46.970 Demilade Agboola: Just have an idea of what’s going on with that, and where the new thing that we’re putting in will fit into everything as well.

804 01:23:48.670 ⇒ 01:23:49.330 Awaish Kumar: Okay.

805 01:23:51.010 ⇒ 01:23:57.179 Awaish Kumar: Yeah, we have a meeting in two and a half hour, so you can be… prepare for it.

806 01:23:59.990 ⇒ 01:24:00.670 Demilade Agboola: Sure.

807 01:24:04.540 ⇒ 01:24:06.240 Awaish Kumar: Okay, yeah, that’s all that.

808 01:24:07.520 ⇒ 01:24:08.060 Uttam Kumaran: Okay.

809 01:24:09.210 ⇒ 01:24:10.050 Uttam Kumaran: Perfect.

810 01:24:10.590 ⇒ 01:24:15.440 Uttam Kumaran: Clarence, you want to stay on for, like, 10 minutes? We can plan out today.

811 01:24:15.760 ⇒ 01:24:16.960 Clarence Stone: Yeah, sure.

812 01:24:17.200 ⇒ 01:24:19.360 Uttam Kumaran: Okay, okay, great. Alright, thank you, everyone.

813 01:24:19.960 ⇒ 01:24:21.190 Awaish Kumar: Thank you, bye.

814 01:24:21.690 ⇒ 01:24:23.030 Demilade Agboola: Thank you. Bye.

815 01:24:25.590 ⇒ 01:24:27.070 Clarence Stone: Dude, you’re back already?

816 01:24:27.730 ⇒ 01:24:35.089 Uttam Kumaran: I’m not, I wasn’t spending, Christmas there or anything. It’s just, it’s just me.

817 01:24:35.360 ⇒ 01:24:37.309 Uttam Kumaran: It’s just, just going to see Robert.

Brainforge Knowledge

Explorer

2025-12-22_planning_de-ae-ai_8590673e

Graph View