Meeting Title: Brainforge x Magic Spoon: Kickoff! Date: 2025-12-22 Meeting participants: Awaish Kumar, IT Provision, Ashwini Sharma, Uttam Kumaran, Michael Thorson, Cinnamon Toast, Demilade Agboola, Mary Burke


WEBVTT

1 00:01:43.320 00:01:44.120 IT Provision: Sweet.

2 00:01:47.650 00:01:48.250 Awaish Kumar: Hello?

3 00:01:49.150 00:01:51.030 IT Provision: Nice to meet you, Josh.

4 00:01:53.330 00:01:55.119 Awaish Kumar: Yeah, nice to meet you.

5 00:01:55.800 00:01:57.719 Awaish Kumar: My name is Avish Kumar.

6 00:02:00.940 00:02:04.020 Awaish Kumar: I’m working as a data engineer.

7 00:02:05.680 00:02:08.850 Awaish Kumar: And yeah, other team members will join soon.

8 00:02:11.280 00:02:14.219 IT Provision: How long have you been with, Brainforge for?

9 00:02:15.340 00:02:16.990 Awaish Kumar: Oh, fine, eho.

10 00:02:17.540 00:02:18.920 IT Provision: A year? Oh, nice.

11 00:02:21.190 00:02:22.820 IT Provision: I love your background.

12 00:02:23.670 00:02:25.550 Awaish Kumar: Colors.

13 00:02:37.580 00:02:38.350 Ashwini Sharma: Hello.

14 00:02:40.360 00:02:40.990 IT Provision: Thank you.

15 00:02:44.160 00:02:45.950 Uttam Kumaran: Hey! Good morning.

16 00:02:46.740 00:02:48.760 IT Provision: Hey, good morning. Good to see you.

17 00:02:49.110 00:02:50.229 Uttam Kumaran: Good to see you, too.

18 00:02:52.110 00:02:53.759 Uttam Kumaran: You look very bundled up.

19 00:02:54.720 00:03:01.739 IT Provision: My office doesn’t have heat, so I have a space heater. It’s not that bad.

20 00:03:02.170 00:03:03.350 Uttam Kumaran: Okay, okay.

21 00:03:04.770 00:03:09.299 Uttam Kumaran: It’s actually surprisingly warm here in Austin this week, it’s like 60 or 78.

22 00:03:09.870 00:03:10.340 IT Provision: Oh, nice.

23 00:03:10.340 00:03:12.459 Uttam Kumaran: But it was, like, 30 last week, so…

24 00:03:13.510 00:03:15.839 IT Provision: Yeah, we’re in that 30 stage right now.

25 00:03:20.800 00:03:23.110 IT Provision: You guys ready for the, holidays?

26 00:03:24.270 00:03:33.040 Uttam Kumaran: Yes, very ready. But just use… use some breaks, spend some time, go to the gym, hang out, yeah.

27 00:03:35.170 00:03:36.200 Uttam Kumaran: How about you?

28 00:03:37.420 00:03:42.090 IT Provision: Yeah, just can’t wait for some, quiet time with family.

29 00:03:47.750 00:03:48.769 Uttam Kumaran: Hey, Michael.

30 00:03:49.560 00:03:50.400 Michael Thorson: my ass.

31 00:03:50.800 00:03:51.580 Michael Thorson: Mine.

32 00:03:52.300 00:03:53.040 Uttam Kumaran: Morning.

33 00:03:58.050 00:04:05.420 IT Provision: So I guess we’re just waiting for… Merry to start… From Ireland.

34 00:04:06.130 00:04:10.200 Uttam Kumaran: Cool, yeah, and then I have a… I can… we can do, again, maybe a brief round of…

35 00:04:10.340 00:04:20.010 Uttam Kumaran: of intros, and then… yeah, just wanted to… I’ll just rehash, sort of, like, kind of short-term goals, and then we’ll kind of pass it to Ashwini and Demolade to talk about

36 00:04:20.510 00:04:23.809 Uttam Kumaran: This week, next week, so… Cool, hey, Mary.

37 00:04:24.240 00:04:24.960 Mary Burke: Hi guys!

38 00:04:26.010 00:04:26.829 Uttam Kumaran: Good morning.

39 00:04:27.730 00:04:28.480 Mary Burke: Morning!

40 00:04:29.320 00:04:43.669 Uttam Kumaran: Great, so, I’ll kind of kick off. So, really, really excited to get started. We have started on some stuff last week, so sort of just wanted to get everybody in a room, especially kind of before holidays, and just,

41 00:04:43.680 00:04:56.099 Uttam Kumaran: highlight again goals, but maybe I’ll just… we’ll do a little bit of a round of brief introductions again. So you have myself, you know, I run Brainforge,

42 00:04:56.100 00:05:09.379 Uttam Kumaran: And you have Awash, who is a senior, data architect and sort of leads a lot on the data engineering, analytics engineering side, on our team. I have him pulled in. We’ll… me and him will probably end up working a little bit on, like, that light stack audit.

43 00:05:09.380 00:05:23.900 Uttam Kumaran: But he sort of looks over anything that’s around, on the… on the data side of things. You have Ashwini and Demolade, you met before. Ashwini will be taking care of a lot of the initial spins, ingestion work, and then Demolade will be working on

44 00:05:25.240 00:05:42.679 Uttam Kumaran: the modeling for the data marts. So kind of that’s… that is sort of our team. As we need sort of people that are experts in certain areas, like, we have other folks, but this is sort of the core crew, that will sort of be helping, you know, for these initial first phases.

45 00:05:42.680 00:06:01.330 Uttam Kumaran: again, just to rehash, like, as I mentioned, those are our core priorities. So one is landing the SPINS data, second is building the, you know, sort of core data marts out for that, and then, as we sort of are going through the end-to-end of, like, developing that phase into modeling, we’re just gonna look around and sort of find out if there’s any

46 00:06:01.330 00:06:04.530 Uttam Kumaran: You know, improvements or optimizations that we’d recommend.

47 00:06:04.530 00:06:07.470 Uttam Kumaran: Unlike the whole system.

48 00:06:07.790 00:06:16.929 Uttam Kumaran: So maybe with that, maybe I’ll just pause there. Is there anything else, like, high level we want to cover? Otherwise, maybe, Ashwini, I could pass it to you to talk about the…

49 00:06:17.060 00:06:20.180 Uttam Kumaran: You know, your sort of path on the ingestion side.

50 00:06:23.700 00:06:27.100 Uttam Kumaran: Cool. Okay, maybe go for it, Sweeney. Yeah.

51 00:06:28.220 00:06:36.100 Ashwini Sharma: Sure, hi. Hey. So, yeah, I just wanted, some more access, to some of the existing pipelines that you have.

52 00:06:36.320 00:06:40.190 Ashwini Sharma: That are extracting data from various different data sources.

53 00:06:40.310 00:06:43.620 Ashwini Sharma: And the other question that I have is, like.

54 00:06:44.180 00:06:51.729 Ashwini Sharma: The extraction… are you following, incremental extraction from sources, or is it… Full extraction every time.

55 00:06:52.150 00:06:56.370 Ashwini Sharma: And if it is incremental, like, what are you doing to maintain,

56 00:06:57.420 00:06:59.559 Ashwini Sharma: You know, a reference to a cursor.

57 00:07:00.430 00:07:01.820 Ashwini Sharma: timestamp-based.

58 00:07:02.850 00:07:12.920 Michael Thorson: I think it varies by source. You can… I think you could probably, like, review the prefix code to see which ones are set up to be incrementally,

59 00:07:13.190 00:07:17.600 Michael Thorson: But I don’t have that top head, I don’t know if Mary or JT, you have that in mind.

60 00:07:18.970 00:07:35.529 Mary Burke: No, I think, I’m probably close to the Business Central, and I think that varies, table by table, but, I mean, Shopify, DCL, and a few of the large ledger tables of our ERP are all incremental, but to Michael’s point, that would be best located in Prefect.

61 00:07:37.780 00:07:38.240 Ashwini Sharma: Okay.

62 00:07:38.240 00:07:44.600 Uttam Kumaran: Ashwini, do you have all access to the other connectors and… or other pipelines in Prefect now?

63 00:07:44.850 00:07:51.599 Ashwini Sharma: I could see the pipelines, but I could not really see the code that That runs on those pipelines.

64 00:07:51.600 00:07:54.910 Awaish Kumar: Yeah. It might be connected with some GitHub repository.

65 00:07:55.790 00:07:56.400 Michael Thorson: Yeah.

66 00:07:56.400 00:07:57.719 Awaish Kumar: Mattress to that.

67 00:07:59.940 00:08:03.740 Michael Thorson: Yeah, I… it looks like, we had some, like, two-factor…

68 00:08:03.900 00:08:16.150 Michael Thorson: blocking today for the GitHub login, so I was just working to turn that off this morning. So, yeah, thanks for the patience there. So you can hopefully log into GitHub, take a look at that repo locally, and then just, like, study up there.

69 00:08:17.710 00:08:22.019 Uttam Kumaran: And so we’ll log into GitHub using the, the magic spoons at Brainforge email.

70 00:08:23.250 00:08:27.799 Michael Thorson: Log into GitHub using IT provision. I sent… that’s all in the.

71 00:08:27.800 00:08:28.350 Uttam Kumaran: And then the…

72 00:08:28.350 00:08:28.960 Michael Thorson: red.

73 00:08:29.260 00:08:30.220 Uttam Kumaran: Okay, great.

74 00:08:30.780 00:08:31.480 Michael Thorson: Yep.

75 00:08:32.049 00:08:32.559 Uttam Kumaran: Cool.

76 00:08:35.069 00:08:36.329 Uttam Kumaran: And, okay.

77 00:08:36.330 00:08:40.769 Ashwini Sharma: How long do we have to go back in time to extract the data?

78 00:08:41.090 00:08:42.600 Ashwini Sharma: One year, two years?

79 00:08:45.570 00:08:48.849 Michael Thorson: For spins specifically, or what are you referring to?

80 00:08:48.850 00:08:50.170 Ashwini Sharma: Fospence, yes.

81 00:08:51.250 00:08:54.090 Michael Thorson: For, yeah, general…

82 00:08:54.950 00:09:04.450 Michael Thorson: I mean, it’s… like, the data comes out aggregated, like, 52 weeks or so, so, like, I think we can just start with, like, a… like, today’s, like, latest

83 00:09:04.830 00:09:12.919 Michael Thorson: time frame dump, and then we just extract periods for… I think it’s 4 weeks up to 52 weeks, which is all outlined in the…

84 00:09:13.180 00:09:14.980 Michael Thorson: the onboarding dog.

85 00:09:14.980 00:09:16.830 Speaker 1 (Cinnamon Toast): 3 years of data, though.

86 00:09:17.110 00:09:19.139 Speaker 1 (Cinnamon Toast): I think that’s the max we can pull.

87 00:09:19.310 00:09:21.840 Speaker 1 (Cinnamon Toast): So 3 years would be the… would be the number.

88 00:09:23.020 00:09:24.649 Speaker 1 (Cinnamon Toast): That I’d like to go for.

89 00:09:25.580 00:09:26.090 Michael Thorson: Cool.

90 00:09:29.750 00:09:30.840 Ashwini Sharma: Got it. Okay.

91 00:09:32.280 00:09:37.359 Ashwini Sharma: And can you also share the client ID and client secret? It mentioned somewhere that

92 00:09:37.630 00:09:39.899 Ashwini Sharma: It was provided during onboarding.

93 00:09:41.590 00:09:48.690 Michael Thorson: That should all be in the Brainforge onboarding packet, I’ll send that in the chat as well. Like, all information’s in there.

94 00:09:48.710 00:09:49.660 Ashwini Sharma: Alright.

95 00:09:54.890 00:09:58.039 Michael Thorson: Oh yeah, Ashwini, it’s in the onboarding packet.

96 00:09:58.040 00:09:59.369 Uttam Kumaran: Doc. Yeah.

97 00:09:59.370 00:10:00.649 Michael Thorson: I’ll highlight it, though.

98 00:10:07.330 00:10:07.860 Uttam Kumaran: Cool.

99 00:10:09.760 00:10:11.730 Ashwini Sharma: Demolata, you’re on.

100 00:10:11.730 00:10:12.609 Uttam Kumaran: You’re on mute.

101 00:10:13.510 00:10:21.300 Demilade Agboola: Actually, I forgot to turn on my mic. So the… I’ve been looking around, and it appears that the…

102 00:10:22.110 00:10:28.459 Demilade Agboola: Omni is pretty well organized, I gotta give it to you, but a lot of things, topics are really well put together.

103 00:10:28.630 00:10:34.599 Demilade Agboola: And the, dashboards are well organized into the appropriate folders.

104 00:10:34.800 00:10:54.399 Demilade Agboola: I mean, there are some things where, like, there’s a… I’ve seen, like, a folder or two where there’s nothing in there, so, like, those kind of things will be handled. I just want to ask, like, the Spins data, when it comes in, would it go into the retail forward slash Spins extract dashboard folder, or would that be a different place internally?

105 00:10:56.170 00:10:59.700 Michael Thorson: We can, we can, like, you’re just talking about the Omni folder itself?

106 00:10:59.830 00:11:00.610 Michael Thorson: Like, the label.

107 00:11:00.610 00:11:07.000 Demilade Agboola: Oh, yeah, for the dashboards, like, yeah, because there is already, like, a Spins Extract collection of dashboards.

108 00:11:07.250 00:11:11.299 Demilade Agboola: Will that… will that fit in here, or will that be an entirely different location?

109 00:11:11.920 00:11:27.850 Michael Thorson: Those dashboards are just, like, a one-off while we were QAing the API access, so, like, those are pretty much deprecated when we get the actual pipeline set up. We can just, like, delete those and replace them at some point, but yeah, can use the same folder structure if we want to. Not super important.

110 00:11:28.610 00:11:34.390 Demilade Agboola: Okay, gotcha. I just wanted to know, like, where people expect things to be in,

111 00:11:34.670 00:11:37.179 Demilade Agboola: So that’s not to, like, duplicate, basically.

112 00:11:37.310 00:11:38.820 Demilade Agboola: Nope.

113 00:11:39.120 00:11:40.709 Uttam Kumaran: In terms of…

114 00:11:42.050 00:11:46.509 Demilade Agboola: the actual… sorry, I was saying… Okay, I think…

115 00:11:46.700 00:11:53.000 Demilade Agboola: I shouldn’t something. In terms of the actual, like, data.

116 00:11:53.320 00:11:56.189 Demilade Agboola: It does appear that,

117 00:11:59.930 00:12:10.690 Demilade Agboola: it does appear that there’s some dashboards that are, like, one-offs? Because I can see, like, it was created some months ago, and it’s maybe been reviewed, like, two times or three times or something. Do we, like…

118 00:12:11.060 00:12:15.820 Demilade Agboola: Have a process to, like, get rid of, like, unused or less-used dashboards?

119 00:12:17.630 00:12:38.039 Mary Burke: I wouldn’t say we have a process right now for that. When we migrated from Looker over to Omni in August of this year, we were only bringing over those dashboards that, at the time were high priority and viewed often, so that’s, this is kind of our pared-down list a little bit, so I don’t think we’re going to be removing any of the

120 00:12:38.100 00:12:40.730 Mary Burke: current dashboards, but I think in terms of, like.

121 00:12:41.390 00:12:45.039 Mary Burke: Usability in the, in the, like, light…

122 00:12:45.260 00:12:51.939 Mary Burke: tech stack audit. It’s more the data that’s in the warehouse versus in the… in the platform.

123 00:12:52.530 00:13:00.369 Demilade Agboola: Okay. Alright, so I will start to poke around that aspect of it, because, like, I had issues getting dbt access, as well as,

124 00:13:00.370 00:13:01.420 Mary Burke: Gotcha.

125 00:13:01.420 00:13:08.529 Demilade Agboola: So, getting to see the full schema and how things are put together in that part of things, as well as the warehouse.

126 00:13:09.480 00:13:22.000 Demilade Agboola: is still a work in progress, but, like, in terms of, like, looking through Omni, I’ve been poking around and just, like, looking at topics, and how things have put together there, and I think, like, to be fair, a lot of things are really well done, in Omni.

127 00:13:24.410 00:13:26.000 Demilade Agboola: I see Ashwini has his hand up.

128 00:13:27.200 00:13:35.019 Ashwini Sharma: Yeah, sorry. Yeah, regarding the GitHub access, I was just wondering, like, would it be possible for you to add

129 00:13:35.330 00:13:43.470 Ashwini Sharma: our GitHub IDs to your repo, instead of having a common GitHub account and then using that to log in.

130 00:13:48.100 00:13:48.800 Mary Burke: I think…

131 00:13:49.870 00:13:57.800 Mary Burke: Eventually, we would be… be open to it, but just as we’re working through this, this partner transition now, we,

132 00:13:58.820 00:14:03.510 Mary Burke: Yeah, I think not at this time, if you guys are okay using me.

133 00:14:03.710 00:14:04.889 Mary Burke: the alias.

134 00:14:06.340 00:14:06.930 Ashwini Sharma: Sure.

135 00:14:08.050 00:14:11.090 Michael Thorson: Ashwin, I think it’ll… Yeah, I apologize for that. I know it’s kind of a hassle.

136 00:14:12.130 00:14:15.640 Uttam Kumaran: No, that’s okay. Ashwini, maybe it’ll probably just be primarily you.

137 00:14:15.850 00:14:26.390 Uttam Kumaran: I guess until… is… is… I don’t know if, again, I can log in, but if Omni is also hooked into there, but if mostly Omni development will do through the ID, through the primary…

138 00:14:26.600 00:14:31.260 Uttam Kumaran: UI, then we won’t need access…

139 00:14:32.250 00:14:33.899 Speaker 1 (Cinnamon Toast): I don’t know who the CBT.

140 00:14:34.170 00:14:44.049 Uttam Kumaran: Yeah, I mean, I… Demolody will be doing that as well. I guess it just depends. If we’re accessing it through… if we’re accessing DBT through the Omni UI, then we won’t really need the codebase anyways.

141 00:14:44.220 00:14:47.589 Uttam Kumaran: Because we’ll be pushing PRs and everything just through that.

142 00:14:47.730 00:14:50.460 Uttam Kumaran: Otherwise…

143 00:14:50.910 00:14:56.880 Demilade Agboola: I probably will be hopping… I will be hopping to dbt select model, directly.

144 00:14:57.160 00:15:01.489 Demilade Agboola: So that we can get the altitude that feeds on, so…

145 00:15:01.720 00:15:14.539 Demilade Agboola: I’ll kind of look through, see what the structure is like right now, see rooms for improvement, or see, like, what’s going on well. I will say that I do like the fact that there are, from Omni’s perspective, I can see that there are, like.

146 00:15:14.980 00:15:27.669 Demilade Agboola: built-out smart models, report models, that you can just use directly into that, and I think that’s beautiful. So I will look around, see opportunities for improvements in terms of performance, in terms of, readability, in terms of…

147 00:15:27.780 00:15:38.579 Demilade Agboola: Just, like, everything that you will want to have, and if we’re, like, repeating ourselves, like, if we don’t have dry models, things like that, looking around and trying to see how we can improve all of that.

148 00:15:40.650 00:15:41.290 Demilade Agboola: Ma’am.

149 00:15:42.880 00:15:43.360 Uttam Kumaran: Yeah.

150 00:15:43.360 00:15:44.230 Michael Thorson: So yeah, I think… That’s a great…

151 00:15:45.220 00:16:03.020 Michael Thorson: Like, I feel like that’s a pretty good starting place for, like, these initial… this initial, like, week or two, and then we can, like, always revisit, like, if you are seeing, like, areas of opportunity where, like, I really wanna… we really want to push PRs, like, let’s just discuss that ahead of time, maybe, just since we do have a relationship that we want to honor in the meantime.

152 00:16:03.600 00:16:13.089 Uttam Kumaran: Okay, cool. So I guess, Demi, maybe you and Ashwini try to log into stuff and just get your environment set up, and then let’s just let Michael and team know if, like.

153 00:16:13.230 00:16:15.320 Uttam Kumaran: the GitHub thing is gonna be a blocker.

154 00:16:16.770 00:16:23.889 Awaish Kumar: Yeah, and this onboarding packet document also says about, like, understanding of the AWS cost.

155 00:16:24.180 00:16:28.180 Awaish Kumar: Although there is a dashboard on the…

156 00:16:28.630 00:16:43.310 Awaish Kumar: like, AWS console, which shows the cost, like, I just want to understand, like, how you would like to see… do we want to see by user, how each user is using that shift, and how much it is costing? Are we more interested in query patterns? Are we…

157 00:16:43.500 00:16:50.800 Awaish Kumar: Just overall want to see the scores for each service, and how granular you want to look at those things.

158 00:16:53.730 00:16:57.600 Speaker 1 (Cinnamon Toast): So… when I think about that, I think probably the…

159 00:16:57.700 00:17:04.349 Speaker 1 (Cinnamon Toast): we’re not as focused on the kind of user basis, the efficiencies there. I think we’re more just looking for

160 00:17:04.880 00:17:09.919 Speaker 1 (Cinnamon Toast): Areas of focus to try to see if maybe we can turn this model

161 00:17:10.020 00:17:23.600 Speaker 1 (Cinnamon Toast): incremental, if there’s anything we can do to, like, make it a lot less intensive in our, like, dbt code specifically, or if there’s an upload process that we’re doing that’s really inefficient. So it’s more about what in our codebase can we fix.

162 00:17:23.599 00:17:33.419 Speaker 1 (Cinnamon Toast): And so it doesn’t need to be that granular, as long as we know, like, hey, this is a target opportunity, and then maybe we can, you know, we can be like, this looks like it’s very expensive.

163 00:17:33.440 00:17:40.209 Speaker 1 (Cinnamon Toast): maybe in the future, we can dive in and see how we can make it cheaper. We’re not looking at… We are looking for… Can I give enough clarification?

164 00:17:41.030 00:17:58.200 Awaish Kumar: Yeah, so, like, we are looking for cost and performance optimizations there, but yeah, for that, we need to do audit, like, how a user is… what query patterns are being used, which models are being utilized, and how they are built, and then we can propose some strategies to optimize those.

165 00:18:00.450 00:18:16.519 Uttam Kumaran: Is you guys seeing any, like, symptoms of, like, stuff? Like, are you… like, a couple examples would be, okay, queries are ending up slow, like, during these time periods, or, like, we just saw a huge uptick in cost, like, recently, like.

166 00:18:16.700 00:18:22.100 Uttam Kumaran: Anecdotally, do you have any of those, or it’s just sort of… things are running… Just check it out.

167 00:18:23.570 00:18:25.080 Demilade Agboola: Or are things breaking?

168 00:18:25.520 00:18:26.350 Demilade Agboola: At setting period.

169 00:18:26.350 00:18:27.250 Speaker 1 (Cinnamon Toast): I would.

170 00:18:27.250 00:18:27.780 Demilade Agboola: meetings.

171 00:18:27.780 00:18:30.759 Speaker 1 (Cinnamon Toast): aren’t breaking, but I’ll call out Shopify orders.

172 00:18:30.980 00:18:32.959 Michael Thorson: is, like, our… that…

173 00:18:33.790 00:18:35.460 Speaker 1 (Cinnamon Toast): Mart, it’s kind of slow.

174 00:18:35.800 00:18:39.319 Speaker 1 (Cinnamon Toast): Okay. We have, although, yeah, I don’t know if that’s maybe because we’re joining.

175 00:18:39.320 00:18:39.690 Uttam Kumaran: That…

176 00:18:39.690 00:18:40.139 Speaker 1 (Cinnamon Toast): a bunch.

177 00:18:40.140 00:18:47.590 Uttam Kumaran: That’s exactly, like, the type of stuff… yeah, that’s exactly, like, the anecdotal thing I’m, like, looking for, basically, is, like, where to kind of poke in first.

178 00:18:48.120 00:18:55.109 Michael Thorson: Oh, yeah. If you look at the bottom of the onboarding doc as well, I called out, like, the core models that might have…

179 00:18:55.250 00:19:11.260 Michael Thorson: some good drill sites. Shopify orders by far being, like, the number one area to look into, since that was just also refactored. So, there’s some opportunity to just, like, understand that, unpack it, understand, like, are there better ways to go about this model?

180 00:19:11.660 00:19:13.620 Speaker 2 (Cinnamon Toast): And another, like, quick cut.

181 00:19:13.620 00:19:18.599 Speaker 1 (Cinnamon Toast): that would probably be helpful is understanding, like, hey, on a single.

182 00:19:18.600 00:19:24.199 Speaker 2 (Cinnamon Toast): day to build this one mart, like, to build a mart, it’s costing us roughly this.

183 00:19:24.550 00:19:37.139 Speaker 2 (Cinnamon Toast): And then, like, comp that to number of people who query it by day, or maybe we can average it by week, like, whatever it is, but just getting some sort of ratio of, like, hey, this is actually, like, one of our top 3 most expensive one and bottom three used.

184 00:19:37.550 00:19:41.200 Speaker 2 (Cinnamon Toast): let’s run this every 3 days. Like, stuff like that could be easy.

185 00:19:41.690 00:19:48.670 Speaker 2 (Cinnamon Toast): But again, just, like, general vibes, if they’re all looking pretty cheap, then… We’re not worried about it.

186 00:19:48.670 00:19:50.140 Uttam Kumaran: Cool. Alright, great.

187 00:19:54.150 00:20:00.309 Demilade Agboola: Also, I’m seeing Michael put stuff about the manual feed from Google Sheets.

188 00:20:00.470 00:20:10.149 Demilade Agboola: And, like, obviously, that has the issues with, like, you know, people changing stuff and all of that. I’m curious if…

189 00:20:11.560 00:20:19.680 Demilade Agboola: there is an opportunity to switch tools with that, or is it just a function of how can we best optimize Google Sheets to utilize…

190 00:20:20.550 00:20:26.880 Demilade Agboola: basically, I’m wondering, like, how much of the audit is about optimization versus potential, like, tool changes?

191 00:20:30.970 00:20:37.559 Michael Thorson: Yeah, and the call-out there is specifically in reference to the daily marketing performance,

192 00:20:37.830 00:20:42.199 Michael Thorson: Gsheet that we’ve been managing, we’ve been discussing, like, JT and Mary, so…

193 00:20:42.660 00:20:47.929 Michael Thorson: If we want to investigate, like, better ways to manage that messy input data, so…

194 00:20:49.080 00:20:51.650 Speaker 2 (Cinnamon Toast): Yeah, open… open to discussing.

195 00:20:51.860 00:20:56.819 Speaker 2 (Cinnamon Toast): there’s not… like, I wouldn’t be opposed to the recommendation. That being said, like.

196 00:20:56.970 00:21:03.470 Speaker 2 (Cinnamon Toast): the way this data comes in, it’s from multiple sources, agencies, given to us in Gsheet format.

197 00:21:03.470 00:21:03.790 Michael Thorson: Yep.

198 00:21:03.790 00:21:08.950 Speaker 2 (Cinnamon Toast): So passing it back in G… like, passing it back in GSheets, Actually, the most efficient?

199 00:21:09.120 00:21:12.400 Speaker 2 (Cinnamon Toast): For, like, from a processing perspective, so…

200 00:21:12.820 00:21:19.859 Speaker 2 (Cinnamon Toast): Like, unless you have, like, a really big solve, it is a manual process that requires manual input.

201 00:21:20.330 00:21:24.339 Speaker 2 (Cinnamon Toast): So, I don’t think a… So unless there’s, like, a… yeah.

202 00:21:24.780 00:21:34.879 Mary Burke: Yeah, and we understand that your team won’t have some of that context, too, so we’re happy to provide some of that… those little details and nuances of our ways of working, too.

203 00:21:36.590 00:21:37.110 Uttam Kumaran: football.

204 00:21:38.500 00:21:41.580 Demilade Agboola: Definitely. I… I definitely see the…

205 00:21:43.010 00:21:57.879 Demilade Agboola: like, I’ll go through things, like, from end to end, get an idea of how things work, from what I can see, and definitely I’ll also hop on the call and be able to, like, ask questions about certain things, so that we can sort of get full business context.

206 00:21:57.990 00:22:04.149 Demilade Agboola: And then use that to be able to either recommend better solutions, or, like, just the…

207 00:22:04.640 00:22:08.130 Demilade Agboola: Exact, appropriate solutions to your use cases.

208 00:22:09.160 00:22:26.450 Speaker 2 (Cinnamon Toast): Yeah, and specifically, just calling out, like, if… if you’re going through stuff, and you see, like, you’ve hit a couple miles, like, I just would like to know more context, feel free to slack, and we can hop on, like, a huddle. Like, we can do, like, we don’t need to boil the earth just to figure out what the heck’s going on, like, we have context, specifically dbt.

209 00:22:26.820 00:22:30.209 Speaker 2 (Cinnamon Toast): Prefect, we do not. Sorry, you’re running the Wild West.

210 00:22:30.930 00:22:33.449 Speaker 1 (Cinnamon Toast): But we poop.

211 00:22:33.930 00:22:36.890 Speaker 1 (Cinnamon Toast): But that’s all DVD. Cool.

212 00:22:40.890 00:22:49.389 Awaish Kumar: like, and, like… like, can you, like, briefly tell me about, like, prefect and stitch choices? Like…

213 00:22:50.060 00:22:56.269 Awaish Kumar: we have a stretch for ingestion, then, like, we also have, like, prefect for running the scripts.

214 00:22:56.680 00:23:05.200 Awaish Kumar: Oh… Like, is that a… we have some specific use cases, like some API…

215 00:23:05.360 00:23:09.040 Awaish Kumar: Or, like, some script, Python scripts which are running on Prefect, or is it…

216 00:23:09.140 00:23:11.349 Awaish Kumar: We are just using it for spins.

217 00:23:12.860 00:23:18.260 Speaker 2 (Cinnamon Toast): So, prefix we use for everything. Basically, we use… Stitch.

218 00:23:18.500 00:23:33.589 Speaker 2 (Cinnamon Toast): for any ones that they have existing connectors to make it simple. We’re currently on an old stitch contract, where we totally go over the Roman pretty much every month, and they just don’t charge us for it. So, like, a tool switch is not gonna likely be…

219 00:23:33.750 00:23:51.639 Speaker 2 (Cinnamon Toast): feasible, because we’re definitely getting too much value, despite Stitch not being as good as other tools, so we’re gonna take it. But then on the prefix side, that’s pretty much any API call that we’re making that’s… that we have, we use.

220 00:23:51.880 00:23:54.569 Speaker 2 (Cinnamon Toast): Prefect to write and structure that.

221 00:23:56.050 00:24:00.290 Speaker 2 (Cinnamon Toast): And that was, that’s historically been used by our current partner.

222 00:24:00.530 00:24:03.869 Speaker 2 (Cinnamon Toast): as a better version of Airflow.

223 00:24:04.550 00:24:05.480 Awaish Kumar: Okay.

224 00:24:05.480 00:24:06.130 Cinnamon Toast: Okay.

225 00:24:07.250 00:24:09.410 Demilade Agboola: Also, I do have a question.

226 00:24:09.620 00:24:20.280 Demilade Agboola: I was curious if there is any desired cadence to, like, reports that has proctored into, like, the DBT jobs and how things are scheduled?

227 00:24:20.430 00:24:21.510 Demilade Agboola: Or…

228 00:24:21.680 00:24:30.229 Demilade Agboola: because, like, if we say, like, Shopify, for instance, takes too long, Shopify does take too long to run, is it a question, again, I have to look through and just

229 00:24:30.310 00:24:41.909 Demilade Agboola: you know, verifiable. It’s a question of, like, we want this to, like, run at a certain cadence, and it’s just too heavy to run at that cadence? Or is it just, like, oh, whenever it runs, which is, like, maybe once a day?

230 00:24:42.120 00:24:45.540 Demilade Agboola: Takes 2 hours to run, or 3 hours to run, for instance.

231 00:24:47.380 00:24:51.310 Speaker 2 (Cinnamon Toast): I would say from a timing perspective, we don’t have any problems with our

232 00:24:52.420 00:24:58.480 Speaker 2 (Cinnamon Toast): how long dbt takes? Nothing we need is… Quicker than yesterday’s data.

233 00:24:58.660 00:25:03.189 Speaker 2 (Cinnamon Toast): So we run it overnight, and it doesn’t hurt us. There’s probably cost efficiencies we could be.

234 00:25:03.195 00:25:03.835 Speaker 1 (Cinnamon Toast): dealing with.

235 00:25:03.840 00:25:21.090 Speaker 2 (Cinnamon Toast): Because we don’t really… we don’t care about how long it’s taking, so we’re probably also eating costs with that association. When I talk about the using Shopify orders, it’s actually in Omni using it. And I’m guessing it’s just more about the size and the way we’re dealing with the data. Maybe we need to index it, like…

236 00:25:21.220 00:25:23.460 Speaker 2 (Cinnamon Toast): I don’t know. There’s things we might need to be doing.

237 00:25:23.460 00:25:26.029 Demilade Agboola: It’s more of, like, responsiveness in the dashboards.

238 00:25:27.670 00:25:33.000 Speaker 2 (Cinnamon Toast): yeah, responsiveness is the only, like, actual barrier we have. There’s probably cost areas, but, like.

239 00:25:34.260 00:25:39.069 Speaker 2 (Cinnamon Toast): Everything, everything’s pretty much every… Looking at yesterday, running overnight.

240 00:25:39.230 00:25:42.290 Speaker 2 (Cinnamon Toast): I don’t think we have anything more important than that.

241 00:25:42.970 00:25:43.510 Cinnamon Toast: correct me if I’m.

242 00:25:43.510 00:25:48.729 Uttam Kumaran: And is that… is that just… is that based on limitation, or are there intraday, like, use cases?

243 00:25:51.110 00:25:52.310 Uttam Kumaran: Or not really.

244 00:25:54.400 00:25:57.520 Speaker 2 (Cinnamon Toast): Not really, because we have in-platform.

245 00:25:57.860 00:25:58.210 Uttam Kumaran: I see.

246 00:25:58.210 00:26:07.069 Speaker 2 (Cinnamon Toast): At least on the growth side, we have in-platform metrics, so the team’s pretty used to, like, if they want something for that day, they look in platform, anything else, they’re an Omni.

247 00:26:07.540 00:26:08.270 Uttam Kumaran: Okay, okay.

248 00:26:14.140 00:26:15.919 Uttam Kumaran: Cool. Anything else, guys?

249 00:26:16.800 00:26:19.810 Awaish Kumar: Yes, I’m just curious to learn if…

250 00:26:20.090 00:26:33.130 Awaish Kumar: switching, Redshift to maybe serverless? Is that an option? Can we assess that if… if switching can be cheaper, or it’s just, like, we just want to use the cluster-based.

251 00:26:36.340 00:26:47.070 Mary Burke: I think we’d be open to… to hearing it if we think the cost savings are there, but not a high priority project for us, if everything’s…

252 00:26:47.400 00:26:49.309 Mary Burke: Working relatively efficiently.

253 00:26:50.510 00:26:52.220 Speaker 2 (Cinnamon Toast): pretty significant.

254 00:26:53.380 00:26:58.450 Speaker 2 (Cinnamon Toast): in cost savings. I think that would be… the requirement.

255 00:26:59.020 00:27:01.049 Awaish Kumar: Okay. Not super interested, but…

256 00:27:03.210 00:27:04.680 Cinnamon Toast: large.

257 00:27:05.290 00:27:06.090 Cinnamon Toast: Sure.

258 00:27:09.790 00:27:10.470 Uttam Kumaran: Okay.

259 00:27:11.380 00:27:18.529 Uttam Kumaran: I feel like we’re… we have a good path forward. I think, yeah, maybe Ashwini and Demolade today, if you guys can confirm GitHub?

260 00:27:18.650 00:27:25.150 Uttam Kumaran: And that we can start to pull down the codebase and see everything. I think also, probably over the next two weeks, we’ll…

261 00:27:25.190 00:27:40.649 Uttam Kumaran: we’ll sort of start from the redshift side and sort of move down the stack in terms of, like, looking through everything, but I think it’s pretty clear, Demi, is, like, those marts we can take a look at first. I mean, of course, like, we’re looking for big joins, big tables.

262 00:27:40.750 00:27:54.819 Uttam Kumaran: And then once we get into Omni as well, one thing I recommended the team is, like, when we look through things, we’ll try to basically try to set up an Omni dashboard that looks at dbt runtime, dbt costs, Looker runtime, Looker costs, so that, like.

263 00:27:54.900 00:28:01.270 Uttam Kumaran: that is something that lives as an artifact, which is basically, like, an infrahealth dash, and we can build that right into…

264 00:28:01.650 00:28:06.530 Uttam Kumaran: Right into Omni, so that’ll be what we’re gonna use to basically look through everything, so…

265 00:28:07.760 00:28:08.320 Cinnamon Toast: sick.

266 00:28:09.320 00:28:09.730 Uttam Kumaran: Cool.

267 00:28:09.730 00:28:10.610 Michael Thorson: I like that.

268 00:28:11.170 00:28:13.559 Uttam Kumaran: And then, in terms of, like… Yeah, go ahead.

269 00:28:13.560 00:28:25.740 Michael Thorson: Oh. Yeah, I was gonna say, since we have some extra time in this meeting, too, before we all log off, I’m happy to walk through all the two-factor authentication, just so we can get into GitHub and into prefix. Make sure everyone’s, like, in. I know it’s…

270 00:28:25.740 00:28:26.110 Uttam Kumaran: Okay.

271 00:28:26.110 00:28:27.559 Michael Thorson: It’s going to be back and forth.

272 00:28:28.070 00:28:43.869 Uttam Kumaran: Okay, let’s… let’s do that, and maybe, just a couple of us that are needed can stay on. And then, in terms of a next meeting, maybe I’ll try to grab some time, like, next, Monday or Tuesday. Maybe, yeah, I’ll… Mary, I can… I can message you.

273 00:28:43.900 00:28:51.310 Uttam Kumaran: And maybe arrange that. That way, we have, like, one touchpoint next week. I know some of our team is, like, in and out,

274 00:28:51.430 00:28:56.359 Uttam Kumaran: But I feel like that, and then after… after next week, we can get into a more regular cadence.

275 00:28:57.950 00:29:09.090 Mary Burke: Okay, that sounds great. Yeah, and then, like Michael and JD said, too, we’re happy to, if there are any quick questions, we’re always happy to jump on a huddle, too. So, we’re very responsive via Slack.

276 00:29:09.680 00:29:16.439 Uttam Kumaran: Yeah, I… this… this meeting is mainly just want to get everybody to say hi. We’ll do it… we’ll do… we’ll be on Slack a bunch, so, yeah.

277 00:29:16.440 00:29:16.800 Michael Thorson: Sure.

278 00:29:18.190 00:29:24.700 Uttam Kumaran: Cool, and then, yeah, I think maybe, Ashwini, Demi, if you want to stay on with Michael, we can just stick around.

279 00:29:25.880 00:29:26.620 Demilade Agboola: Yeah.

280 00:29:26.970 00:29:34.000 Demilade Agboola: Also, in terms of optimization, just a final question, in terms of optimization, what’s the higher priority, performance or cost?

281 00:29:37.180 00:29:38.829 Speaker 2 (Cinnamon Toast): cost, I would say.

282 00:29:39.370 00:29:41.750 Speaker 2 (Cinnamon Toast): I don’t think we really have any performance issues.

283 00:29:42.170 00:29:42.570 Michael Thorson: Yes.

284 00:29:42.570 00:29:44.369 Speaker 2 (Cinnamon Toast): It’d be nice if it was quicker.

285 00:29:44.650 00:29:46.050 Speaker 2 (Cinnamon Toast): But no one’s that, huh?

286 00:29:46.240 00:29:46.910 Speaker 2 (Cinnamon Toast): But someday.

287 00:29:46.910 00:29:50.740 Mary Burke: Business Central, but that’s another can of worms.

288 00:29:52.290 00:29:52.830 Uttam Kumaran: Okay.

289 00:29:55.470 00:29:56.850 Uttam Kumaran: Okay, great.

290 00:29:57.540 00:29:58.519 Demilade Agboola: Thank you. Yeah.

291 00:29:58.520 00:30:05.240 Michael Thorson: Maybe… maybe… oh, and just to add to that, it’s just reliability in general. I think that’s the…

292 00:30:05.880 00:30:09.540 Michael Thorson: That’s the… that’s the bigger, kind of, like, low-hanging fruit, just like ours.

293 00:30:09.540 00:30:16.350 Uttam Kumaran: Has there been a moment recently where, like, a bunch of stuff failed, and, like, what hap… what was the situation? Like, what happened?

294 00:30:17.540 00:30:19.879 Mary Burke: Or, like, a pipeline broke and we weren’t notified.

295 00:30:20.290 00:30:20.750 Mary Burke: and…

296 00:30:20.750 00:30:21.590 Uttam Kumaran: Then we’ve…

297 00:30:21.590 00:30:25.590 Mary Burke: I’ve been operating with incomplete data for… Too long.

298 00:30:26.190 00:30:27.170 Michael Thorson: Heck, yeah.

299 00:30:27.170 00:30:27.730 Demilade Agboola: air.

300 00:30:28.120 00:30:28.690 Uttam Kumaran: Okay.

301 00:30:29.220 00:30:29.610 Demilade Agboola: Gotcha.

302 00:30:29.610 00:30:43.720 Michael Thorson: Yeah, we use Metaplan for just, like, reactive, quality assurance, so we’re, like, kind of leaning into that as, is this pipeline running, and is row count kind of moving up, and then occasionally we will miss a couple.

303 00:30:43.720 00:30:46.020 Mary Burke: Yeah, and that doesn’t always capture everything.

304 00:30:46.600 00:30:53.140 Uttam Kumaran: Do you guys have, like, a Slack channel for data info alerts, or how is alerting… are broad, like.

305 00:30:53.140 00:30:53.720 Mary Burke: We get…

306 00:30:53.720 00:30:54.550 Uttam Kumaran: happening.

307 00:30:54.980 00:30:57.270 Mary Burke: We get emails from Metaplane.

308 00:30:57.840 00:30:58.490 Uttam Kumaran: Okay.

309 00:30:58.490 00:30:59.270 Demilade Agboola: Hmm.

310 00:30:59.270 00:31:13.259 Uttam Kumaran: I think one thing, we need to take a look at when you’re in Prefect is take a look at retries and, like, the job failures, and see if we can hook up something so that, like, at least our Slack channel gets hit if there’s a… if there’s a prefect issue.

311 00:31:13.470 00:31:22.249 Uttam Kumaran: And then, we do a lot of work with Metaplane, by the way, so… and that’s, like, a lot of stuff that Ashwini owns, so we can also take a look there. I mean, it’s…

312 00:31:22.320 00:31:39.799 Uttam Kumaran: it’s a tough software, like, it can be very, very noisy, and there’s this, like, annoying, like, calibration period that, like, we basically message them a lot about, like, this is kind of stupid. But it’s something we work with, a lot, so I think across the whole system.

313 00:31:39.870 00:31:53.809 Uttam Kumaran: we’ll think about, like, hey, how do we get alerted if there is issues, either on the data ingestion side? And so you’ll see as we build this pipeline, we’ll build this one pipeline with sort of a lot of that in mind, and then we can see what can we apply across all prefect pipelines, basically.

314 00:31:55.400 00:32:04.849 Uttam Kumaran: You know, to make sure that if there is stale data, one place or another, we’re alerting. For example, a prefect pipeline may go down, and

315 00:32:05.050 00:32:17.069 Uttam Kumaran: if this doesn’t hit you, then Metaplane at some point should check refresh time and hit us, right? And so, like, how do we think about that? Similarly for dbt jobs. So, yeah, okay.

316 00:32:17.470 00:32:18.539 Uttam Kumaran: That makes sense.

317 00:32:19.050 00:32:29.710 Ashwini Sharma: I have one more question regarding prefect pipelines. In terms of, you know, capturing deletes, do you capture deletes, or do I just ignore them?

318 00:32:33.150 00:32:35.179 Speaker 2 (Cinnamon Toast): I think it’s gonna be…

319 00:32:35.180 00:32:38.940 Uttam Kumaran: Yeah, it’s gonna be depending on the sources of Sweeney, so we’ll have to check all of…

320 00:32:39.160 00:32:43.409 Uttam Kumaran: The team here, I think, isn’t gonna have much context on source by source.

321 00:32:43.820 00:32:44.260 Ashwini Sharma: Okay.

322 00:32:44.260 00:32:47.549 Uttam Kumaran: So we have to look into all the configs and prefix directly.

323 00:32:48.170 00:32:48.860 Ashwini Sharma: Alright.

324 00:32:49.750 00:32:50.260 Uttam Kumaran: Yeah.

325 00:32:52.180 00:32:56.099 Speaker 2 (Cinnamon Toast): if you have a… I know there’s probably a lot of stuff, like,

326 00:32:56.660 00:32:58.459 Speaker 2 (Cinnamon Toast): If we want to pull up the…

327 00:32:58.640 00:33:06.849 Speaker 2 (Cinnamon Toast): the different pipelines, we can kind of, like… I don’t know what was in the sheet, I don’t know if this had this, but just, like, I think Michael T had the priority of…

328 00:33:07.320 00:33:09.450 Speaker 2 (Cinnamon Toast): the, marts, correct?

329 00:33:09.720 00:33:11.539 Speaker 2 (Cinnamon Toast): I don’t know if we can call out.

330 00:33:11.790 00:33:14.480 Speaker 2 (Cinnamon Toast): Did we call it pipelines that were important?

331 00:33:16.470 00:33:23.799 Michael Thorson: I just put one in there, like, the general ledger. Do you want to, like, just talk through some other key ones together?

332 00:33:24.630 00:33:29.979 Speaker 2 (Cinnamon Toast): Just to give… I don’t know if you want, like, we could… just to give you, like, a list of what’s mostly important.

333 00:33:30.180 00:33:30.830 Michael Thorson: Yeah.

334 00:33:30.830 00:33:31.450 Speaker 1 (Cinnamon Toast): Like, I think…

335 00:33:31.450 00:33:40.209 Uttam Kumaran: I mean, again, we’ll arrive at the same conclusions, I’m sure, but if we can speed up and just get, yeah, what the core ones are, like, that’ll… that’s just it.

336 00:33:43.100 00:33:44.649 Mary Burke: I just threw a few more in there.

337 00:33:45.460 00:33:46.130 Michael Thorson: as well.

338 00:33:48.360 00:33:56.039 Michael Thorson: Yeah, we’re… JT, obviously, we’re focusing mostly on, like, Business Central, so if there’s any that you specifically want some investigation into…

339 00:33:57.780 00:34:01.230 Speaker 2 (Cinnamon Toast): I’ll go take a look, but I’ll add it there, just so that you know.

340 00:34:02.000 00:34:02.740 Michael Thorson: Yes.

341 00:34:16.080 00:34:20.330 Speaker 2 (Cinnamon Toast): Also calling out, we don’t use dbt tests right now.

342 00:34:21.080 00:34:21.420 Michael Thorson: Yeah.

343 00:34:21.429 00:34:23.879 Speaker 1 (Cinnamon Toast): Probably has a couple good wins in there.

344 00:34:24.510 00:34:28.210 Speaker 2 (Cinnamon Toast): Just on, like… reactionary stuff.

345 00:34:32.739 00:34:33.339 Uttam Kumaran: Okay.

346 00:34:34.579 00:34:40.169 Uttam Kumaran: Yeah, so we’ll look at jobs, we’ll look at the number of jobs, Kind of like…

347 00:34:40.579 00:34:45.429 Uttam Kumaran: the VIN diagram of, like, what models are running at what point.

348 00:34:45.559 00:34:49.979 Uttam Kumaran: And then job… like, job timings.

349 00:34:50.409 00:34:55.549 Uttam Kumaran: And yeah, so, like, orchestration of dbt will be, like, a piece of the sort of audit thing that we’ll look through.

350 00:34:59.310 00:34:59.910 Cinnamon Toast: Sick.

351 00:35:00.620 00:35:01.330 Uttam Kumaran: Cool.

352 00:35:02.070 00:35:11.849 Uttam Kumaran: Okay, well, maybe a couple of us, and Michael, we can stay on, and then maybe Ashwini, if you want to start with just looking through getting access to prefect stuff, we can nail that first.

353 00:35:12.270 00:35:14.810 Ashwini Sharma: Sure, yeah, I got access to Prefect.

354 00:35:15.090 00:35:16.910 Michael Thorson: That would be okay.

355 00:35:17.160 00:35:18.660 Ashwini Sharma: We would need, access to get.

356 00:35:18.660 00:35:21.360 Uttam Kumaran: Can you check… yeah, I mean, we can go through GitHub.

357 00:35:21.360 00:35:21.900 Ashwini Sharma: Yeah.

358 00:35:22.330 00:35:29.370 Michael Thorson: Yeah, just go ahead and try to log in, and then send, two-factor via text, and I’ll send over that,

359 00:35:30.020 00:35:31.220 Michael Thorson: That code, dude.

360 00:35:31.660 00:35:34.810 Michael Thorson: It’ll be good for 30 days, so it should just get us over the hump.

361 00:35:35.310 00:35:36.230 Ashwini Sharma: Yeah.

362 00:35:36.230 00:35:40.339 Uttam Kumaran: Maybe the… maybe all four of us can just do that right now, because I’ll log in on my site, too.

363 00:35:40.790 00:35:41.230 Demilade Agboola: I have…

364 00:35:41.230 00:35:41.580 Michael Thorson: Yeah.

365 00:35:41.580 00:35:43.960 Demilade Agboola: triggered to factor out.

366 00:35:43.960 00:35:44.620 Awaish Kumar: What?

367 00:35:44.850 00:35:50.059 Awaish Kumar: Like, region is using, like, we are using cluster in… for Redshift.

368 00:35:52.200 00:35:58.679 Awaish Kumar: I’m… I’m… I just logged in and opened the Redshift cluster, but I can’t see anything.

369 00:35:59.160 00:36:03.629 Awaish Kumar: So… Which region, like, can I select?

370 00:36:06.790 00:36:22.110 Michael Thorson: That… we honestly, like, aren’t super up to speed on that access. I think we just have, like, that account we can try to upgrade to, like, admin permissions, though. I think it’s pretty limited in the, like, what you can actually view in the console. Is that the issue?

371 00:36:22.600 00:36:23.220 Awaish Kumar: Yeah.

372 00:36:24.030 00:36:24.660 Michael Thorson: Yes.

373 00:36:25.230 00:36:32.230 Michael Thorson: We can take a follow-up action to just increase visibility in that account. That’s, like, kind of been on our backlog list for a while.

374 00:36:33.030 00:36:33.620 Uttam Kumaran: Okay.

375 00:36:37.150 00:36:39.369 Ashwini Sharma: Can you forward the text, if you have received?

376 00:36:40.230 00:36:40.780 Michael Thorson: Yep.

377 00:36:41.080 00:36:44.619 Michael Thorson: Got it. There you go. I’ll send it in Zoom chat right now.

378 00:36:45.180 00:36:46.249 Michael Thorson: This is good.

379 00:36:56.590 00:36:58.269 Ashwini Sharma: And what’s the repo name?

380 00:36:59.710 00:37:04.619 Ashwini Sharma: Okay, there is a Spence, GraphQL, I think there’s the POC thing, right?

381 00:37:05.180 00:37:07.109 Demilade Agboola: Just can you follow the new text you received?

382 00:37:07.110 00:37:16.699 Michael Thorson: No, it should be, Magic Spoon prefixed and Magic Spoon, dbt are the repos that you should be inside.

383 00:37:22.340 00:37:24.679 Demilade Agboola: Michael, can you follow the new text that you received?

384 00:37:25.040 00:37:25.950 Ashwini Sharma: Yep.

385 00:37:26.850 00:37:30.529 Michael Thorson: Are you seeing it in there, or do we need to go, just make sure that…

386 00:37:30.530 00:37:33.250 Ashwini Sharma: Now, let me share my screen.

387 00:37:33.810 00:37:34.630 Michael Thorson: Sweet.

388 00:37:38.790 00:37:39.550 Michael Thorson: Got it.

389 00:37:43.570 00:37:45.600 Ashwini Sharma: Oh, they.

390 00:37:46.340 00:37:47.620 Michael Thorson: Let’s GitHub…

391 00:37:47.990 00:37:50.950 Ashwini Sharma: One second, let me… how do I share the other screen?

392 00:37:53.030 00:37:54.240 Michael Thorson: One second…

393 00:38:09.550 00:38:11.800 Ashwini Sharma: Yeah, are you listening? Is this the one?

394 00:38:12.890 00:38:20.049 Michael Thorson: No, that’s… this is my private, I don’t own the repos, so go back to IT provisions, just like the homepage.

395 00:38:21.170 00:38:21.970 Michael Thorson: Yeah.

396 00:38:22.250 00:38:23.350 Michael Thorson: Top right.

397 00:38:24.540 00:38:27.050 Michael Thorson: Is it shared with news?

398 00:38:27.470 00:38:32.110 Michael Thorson: Oh, interesting. Yeah, one second, let me make sure it’s… the access is shared with IT provision.

399 00:38:32.110 00:38:36.279 Uttam Kumaran: Yeah, Ashwini, go here in the top right and click on Organizations.

400 00:38:37.120 00:38:38.069 Uttam Kumaran: Yeah, I’m here.

401 00:38:41.370 00:38:48.020 Michael Thorson: should be… Yeah, Magic Spoon. IT?

402 00:38:48.650 00:38:49.680 Ashwini Sharma: This one?

403 00:38:50.600 00:38:52.180 Michael Thorson: Let me make sure…

404 00:38:52.180 00:38:52.700 Ashwini Sharma: dear?

405 00:38:55.520 00:38:56.920 Uttam Kumaran: And then go to repos.

406 00:38:59.830 00:39:05.690 Michael Thorson: I’m in there right now to just make sure… I’ll send you a link to the repo. This could be…

407 00:39:14.920 00:39:20.150 Michael Thorson: Here’s a link to it as well, just… maybe that… maybe this is… an oversight, apologies.

408 00:39:21.700 00:39:27.029 Michael Thorson: Tributors… I’m in the private reaper right now, it’s called magicspoon-dbt, so…

409 00:39:28.640 00:39:30.389 Michael Thorson: I think that was the oversight.

410 00:39:33.800 00:39:35.710 Ashwini Sharma: Magic Spoons PI.

411 00:39:36.540 00:39:39.470 Ashwini Sharma: No, I don’t think I saw that anywhere.

412 00:39:40.330 00:39:41.910 Demilade Agboola: Okay, cool. You don’t register yet.

413 00:39:42.140 00:39:42.780 Demilade Agboola: I said, huh.

414 00:39:42.780 00:39:47.890 Michael Thorson: Yeah, cool. Yeah, we’ll add you as contributors this, like, right now, then. Sorry about that.

415 00:39:48.160 00:39:54.560 Michael Thorson: Yeah, this is, the IT provision is, like, a new GitHub account, so I think it was just an oversight on our side.

416 00:39:55.390 00:39:56.950 Michael Thorson: That’s why we set it up.

417 00:39:57.200 00:40:04.770 Uttam Kumaran: And Ashwini, you can actually add, like, you can add multiple GitHub users, you don’t have to do an incognito.

418 00:40:04.980 00:40:10.969 Uttam Kumaran: So if you go… if you go to your GitHub, and you just go to the top right, you can just add account, actually.

419 00:40:13.380 00:40:17.480 Uttam Kumaran: I was about to do the same thing, but then I realized, like, you could just have multiple accounts.

420 00:40:18.930 00:40:22.780 Uttam Kumaran: So if you go to the top, yeah… And…

421 00:40:22.950 00:40:25.589 Uttam Kumaran: You click on this, or, like, right here.

422 00:40:26.280 00:40:27.110 Uttam Kumaran: Yeah.

423 00:40:27.840 00:40:29.050 Uttam Kumaran: Antarctica.

424 00:40:29.050 00:40:30.290 Ashwini Sharma: Oh, okay.

425 00:40:30.750 00:40:31.680 Uttam Kumaran: Yeah.

426 00:40:34.790 00:40:38.030 Uttam Kumaran: But I’m also… I’m next in line for SMS, so you gotta get.

427 00:40:38.030 00:40:40.180 Demilade Agboola: Oh, okay.

428 00:40:42.230 00:40:44.929 Ashwini Sharma: I have it in incognito right now.

429 00:40:44.930 00:40:46.210 Uttam Kumaran: Alright, alright.

430 00:40:46.830 00:40:48.250 Ashwini Sharma: Let me go this way.

431 00:40:49.190 00:40:51.089 Uttam Kumaran: Gonna blow up your phone, Michael.

432 00:40:51.350 00:40:54.400 Michael Thorson: Yeah, no, it’s… it comes through a shared Slack channel, so I’m just passing.

433 00:40:54.400 00:40:55.400 Uttam Kumaran: Oh, okay.

434 00:40:55.400 00:40:56.040 Ashwini Sharma: Mac.

435 00:40:56.040 00:40:57.039 Michael Thorson: Yeah, so I’m trying to.

436 00:40:57.040 00:41:00.400 Uttam Kumaran: How did you guys get 2FA… how did you guys get the 2FA?

437 00:41:00.600 00:41:02.669 Uttam Kumaran: through a phone number to come through Slack.

438 00:41:02.830 00:41:06.810 Uttam Kumaran: Like, I’ve always tried to… I tried to set something up, Via Twilio.

439 00:41:06.810 00:41:11.979 Michael Thorson: Google… a Google phone. Voice? A Google Voice that has a Slack integration, yeah.

440 00:41:12.310 00:41:14.960 Uttam Kumaran: Okay, I need to… I need to go think about that.

441 00:41:15.270 00:41:21.440 Uttam Kumaran: Because what we… what we started doing, we just have, like, we just started using passkeys and 1Password, which, like…

442 00:41:21.820 00:41:23.590 Uttam Kumaran: It’s a little bit more shareable.

443 00:41:29.420 00:41:31.040 Uttam Kumaran: Okay, I’m in on my side.

444 00:41:32.810 00:41:35.420 Demilade Agboola: Also, can I get the code for dbt?

445 00:41:36.460 00:41:38.339 Michael Thorson: Yep. One sec.

446 00:41:59.510 00:42:00.450 Michael Thorson: Second.

447 00:42:09.460 00:42:16.159 Michael Thorson: Alright, here comes DBT… Yeah, thanks for the patience. Should be adding your…

448 00:42:16.500 00:42:27.050 Michael Thorson: like, both repos shortly to the IT provisions. This spins GraphQL… this was, like, a proof of concept, whatever you’re poking around right now, so, like…

449 00:42:27.350 00:42:41.950 Michael Thorson: I just included it and shared it with you, just in case the… like, I have a lot of the information in the shared Excel document, but this is, like, a proof of concept, just, like, a one-off Python script to, like, ping you the GraphQL.

450 00:42:42.420 00:42:50.930 Michael Thorson: Yeah, this code is pretty messy and, like, not super well set up, and we really need to, like, refactor some of this information into…

451 00:42:51.110 00:42:53.110 Michael Thorson: the prefix.

452 00:42:53.510 00:42:54.160 Uttam Kumaran: Cool.

453 00:42:54.630 00:42:55.200 Michael Thorson: Yeah, great.

454 00:42:55.200 00:42:56.450 Uttam Kumaran: Okay, at least we have to start.

455 00:42:57.100 00:43:13.530 Michael Thorson: Yeah. This was, yeah, just to, like, get the data out that’s in Omni. We, like, uploaded it to the warehouse just, like, a one-off, just to make sure everything looks good in the API, and is one-to-one with our manual extract. So yeah, this is all just kind of, like, scratch work, basically.

456 00:43:13.740 00:43:14.400 Uttam Kumaran: Okay.

457 00:43:14.970 00:43:27.650 Michael Thorson: Yeah, but just wanted to include it in case it is helpful, especially my variables will have a lot of context on the documentation and the filters we actually utilize a lot, so…

458 00:43:28.190 00:43:30.960 Michael Thorson: If you have any questions on, like, some of these…

459 00:43:31.250 00:43:45.760 Michael Thorson: parameters, like, let me know. Feel free to hit me up via Slack, and I can kind of walk you through some of the nuances. I just want to, like, help y’all through the documentation efficiently. There’s a lot of, like, filtering and measure opportunities, and a lot of, like.

460 00:43:46.260 00:43:49.819 Michael Thorson: Organizational knowledge kind of baked into these filters, so…

461 00:43:50.230 00:43:51.160 Uttam Kumaran: Okay.

462 00:43:53.500 00:43:58.579 Michael Thorson: But yeah, this is the primary, like, usefulness of this repo. It’s just this variables.

463 00:44:00.150 00:44:00.750 Uttam Kumaran: Okay.

464 00:44:01.240 00:44:08.359 Uttam Kumaran: Cool. And then I just, I log in through Prefect as well, so, I assume you’ll be able to forward me the…

465 00:44:10.740 00:44:28.589 Michael Thorson: Login. Yeah. Yeah, and then following up with that, too, I was gonna set up for the email-based two-factor pieces, I was gonna set up forwarding. Can I, yeah, great. Did you approve, did you verify the forwarding rules, email that went to the BrainForge inbox?

466 00:44:29.170 00:44:30.129 Uttam Kumaran: Yes, I just did.

467 00:44:30.440 00:44:40.230 Michael Thorson: Okay, great. So I’ll set up forwarding for, I think it’s prefix and whatever we can. I know it’s a little bit messy, and that’ll, again.

468 00:44:40.230 00:44:40.560 Uttam Kumaran: Sorry.

469 00:44:40.560 00:44:46.919 Michael Thorson: us to the… and get us to a good point until we can, like, set everyone up with the proper accounts. So, thanks for the flexibility here.

470 00:44:46.920 00:44:47.980 Uttam Kumaran: Yeah, I think it’s mainly…

471 00:44:47.980 00:44:48.880 Michael Thorson: onboarding, but…

472 00:44:49.130 00:44:54.279 Uttam Kumaran: No, no, I think it’s, so, mainly DBT and Omni,

473 00:44:54.780 00:45:01.729 Uttam Kumaran: PBT, Omni, and Prefect, if we can just make sure that those get forwarded, Yeah, that’s fun.

474 00:45:01.890 00:45:03.240 Awaish Kumar: We’ll need a LC footer.

475 00:45:04.240 00:45:05.919 Uttam Kumaran: and Redshift access.

476 00:45:06.650 00:45:09.359 Michael Thorson: Yeah, just forwarded the prefix.

477 00:45:10.650 00:45:18.180 Michael Thorson: Yeah, I’m looking everywhere for these login codes. It’s like, some are in email, some are in text.

478 00:45:18.480 00:45:21.179 Michael Thorson: As a prefix, I just forwarded it to your inbox as well.

479 00:45:21.520 00:45:22.090 Uttam Kumaran: Okay, great.

480 00:45:25.440 00:45:31.209 Demilade Agboola: Also, I just… made a request for the profiles.yaml, dbt profiles.yaml.

481 00:45:31.870 00:45:32.660 Demilade Agboola: Great.

482 00:45:32.660 00:45:34.500 Michael Thorson: Yeah, I just shared that with you as well.

483 00:45:34.810 00:45:35.999 Demilade Agboola: Okay, appreciate it.

484 00:45:36.000 00:45:43.570 Michael Thorson: Yeah, call out there is, like, if you are running any dbt, just, like, make sure you’re using a dev schema.

485 00:45:43.960 00:45:46.330 Michael Thorson: To dump any, like, table builds into.

486 00:45:47.650 00:45:51.839 Uttam Kumaran: And is Prefect backed by any GitHub repository, or no?

487 00:45:53.430 00:46:07.359 Michael Thorson: Yes, it is as well. I just reached out to our other analytics partner to get access, for both Prefect and dbt. I think that’s the biggest gap right now, so I’ll track that down and make sure it’s shared ASAP.

488 00:46:07.610 00:46:08.550 Uttam Kumaran: Okay. Okay.

489 00:46:09.400 00:46:10.060 Michael Thorson: Yeah.

490 00:46:10.940 00:46:13.829 Uttam Kumaran: I mean, I think… yeah, Srini, I think you should…

491 00:46:13.980 00:46:18.819 Uttam Kumaran: You may be able to do everything via the UI, but I’m not sure. Maybe you’ll have to check.

492 00:46:24.830 00:46:32.400 Michael Thorson: Yeah, and that shouldn’t be a problem. I think it’s just, like, just waiting for our analytics partner to respond and actually click the button, so…

493 00:46:32.670 00:46:33.200 Uttam Kumaran: Okay.

494 00:46:33.200 00:46:40.210 Michael Thorson: Yeah, apologies, that wasn’t done ahead of time, that was a gap. I haven’t been in the IT Provisions GitHub account very much.

495 00:46:47.780 00:46:53.820 Michael Thorson: Great. Any… so does anyone else need to log in to any of these, like, prefixed dbt.

496 00:46:53.820 00:47:02.099 Uttam Kumaran: Yeah, Away shall be shall really be the primary one for AWS, so I don’t think all of us need to really be in there, but as long as he has access to…

497 00:47:02.340 00:47:07.520 Uttam Kumaran: Redshift and the console, that’s primarily him. All of us…

498 00:47:07.630 00:47:10.670 Uttam Kumaran: me and Ashwini and Awish will be in Prefect. Demi…

499 00:47:10.780 00:47:15.040 Uttam Kumaran: I’m not gonna be in Omni or anything, so… right now, so that’ll just be you.

500 00:47:15.990 00:47:18.500 Uttam Kumaran: But those will go to email, so I feel okay.

501 00:47:19.620 00:47:20.150 Michael Thorson: Okay.

502 00:47:22.480 00:47:23.160 Michael Thorson: Cool.

503 00:47:24.560 00:47:24.890 Uttam Kumaran: Okay.

504 00:47:24.890 00:47:36.319 Michael Thorson: Yeah, just, keep us updated on Slack if you need anything. Again, like, thanks for the patience with the logins. I know this isn’t, like, best, most efficient way to get everyone into the systems, but I guess it’s just…

505 00:47:36.710 00:47:38.160 Michael Thorson: the set of the relationship.

506 00:47:38.160 00:47:52.760 Uttam Kumaran: And we’ve dealt with, we’ve dealt with everything, so I just want to make sure we’re… we’re good. I mean, we haven’t lost a lot… we’ve been poking around all last week and stuff, so I think after today, we’re in a good spot. But yeah, let’s, I think we’ll chat internally.

507 00:47:52.760 00:48:01.410 Uttam Kumaran: today, and then I’ll probably send a note tomorrow just about plan for the rest of the week, and then I’ll go ahead and book time, and put time on the calendar for

508 00:48:01.670 00:48:04.970 Uttam Kumaran: next week, around Monday this time, or Tuesday, so…

509 00:48:05.870 00:48:06.650 Mary Burke: Sounds great.

510 00:48:07.330 00:48:07.980 Uttam Kumaran: Cool.

511 00:48:08.040 00:48:25.260 Michael Thorson: Yeah. Okay. And of course, yeah, like, just feel free to tag us also in the Google Docs. I know I had kind of some notes split across this onboarding doc, as well as, like, the Spins GraphQL kind of guide, so feel free to just, like, toss, like, tag us directly in the documents as well, if you do have any questions, like.

512 00:48:25.260 00:48:34.530 Uttam Kumaran: Happy to, again, like, hop on a huddle if it requires it, or just, like, respond directly in the doc. Like, we’re super… we made ourselves available this week. We know it’s gonna be a little…

513 00:48:34.530 00:48:42.620 Michael Thorson: a bit of a learning curve to figure out our whole tech stack and where everything lives, so… yeah, just let us know. Happy to be supportive from whatever we can.

514 00:48:43.140 00:48:43.840 Uttam Kumaran: Perfect.

515 00:48:44.900 00:48:45.720 Uttam Kumaran: Okay.

516 00:48:46.030 00:48:49.020 Uttam Kumaran: Alright, well, thank you, and enjoy the holidays if we don’t chat before then.

517 00:48:50.250 00:48:51.420 Mary Burke: Thank you, same to you guys.

518 00:48:51.420 00:48:52.890 Demilade Agboola: Okay.

519 00:48:52.890 00:48:53.650 Uttam Kumaran: Thank you.