Meeting Title: -️-data-eng-roadmap Date: 2024-06-11 Meeting participants: Nicolas Sucari, Patrick Trainer, Uttam Kumaran
WEBVTT
1 00:00:00.020 ⇒ 00:00:02.137 Patrick Trainer: Or the okay. I always thought the
2 00:00:03.570 ⇒ 00:00:06.009 Patrick Trainer: the AI companion was the.
3 00:00:06.390 ⇒ 00:00:07.160 Patrick Trainer: But that’s.
4 00:00:07.160 ⇒ 00:00:11.835 Nicolas Sucari: Taking notes. I think I’m making a summary, but not recording.
5 00:00:12.260 ⇒ 00:00:13.100 Patrick Trainer: Good point
6 00:00:14.057 ⇒ 00:00:16.210 Patrick Trainer: cool. And so
7 00:00:16.260 ⇒ 00:00:28.279 Patrick Trainer: what I’ve double like what I was saying. It’s I had to establish, I mean one a team, but then also kind of like, how that team ideally should do work.
8 00:00:28.460 ⇒ 00:00:31.560 Patrick Trainer: And then how we think about work.
9 00:00:32.000 ⇒ 00:00:38.779 Patrick Trainer: And so one of my goals and one of my goals here is basically to
10 00:00:39.170 ⇒ 00:01:03.469 Patrick Trainer: kind of enable everyone to be autonomous or or as autonomous as they can. So when like, if you say you don’t have like tickets flowing in. How do you get people on onto the same page to kind of like, create their own tickets to have that agency of like. Okay, the this is the scope and the boundaries of where I should be thinking. And these are the problems and
11 00:01:04.390 ⇒ 00:01:04.849 Patrick Trainer: like
12 00:01:05.000 ⇒ 00:01:11.229 Patrick Trainer: problem areas that we should initially, or that we should be focusing on.
13 00:01:11.270 ⇒ 00:01:15.229 Patrick Trainer: And so like, I think of, like when I think of
14 00:01:15.813 ⇒ 00:01:30.200 Patrick Trainer: like work in general, I don’t necessarily like. Like strict hierarchies where things trickle down from like a a CTO down to a manager down to somebody else, and then that gets disseminated
15 00:01:30.527 ⇒ 00:01:45.930 Patrick Trainer: the other way. I think it. It really should take kind of like either a flat or bottom up approach where the people in the trenches are like. I’m noticing this. This fits in within our framework, and this is kind of what we need to do.
16 00:01:45.990 ⇒ 00:01:53.629 Patrick Trainer: And it’s the top that sets that guideline and and that framework for for execution.
17 00:01:54.053 ⇒ 00:01:58.659 Patrick Trainer: And so that’s kind of what I’ve done here. And then
18 00:01:59.240 ⇒ 00:02:12.370 Patrick Trainer: what this allows us to do is like, we’ll go through the framework. And then we can actually, piece by piece, according to each focus area put in like concrete
19 00:02:13.115 ⇒ 00:02:18.169 Patrick Trainer: like actions of things that need or should be implemented.
20 00:02:18.800 ⇒ 00:02:20.140 Uttam Kumaran: Sharing screen. By the way, pat.
21 00:02:20.140 ⇒ 00:02:21.570 Patrick Trainer: No, I’m not. I’m I’m.
22 00:02:21.570 ⇒ 00:02:22.149 Uttam Kumaran: I was actually.
23 00:02:22.150 ⇒ 00:02:30.749 Patrick Trainer: I’m I’m just given like the the preface of Like How To how to think about all of this. So I will go and
24 00:02:30.790 ⇒ 00:02:33.420 Patrick Trainer: share my screen now.
25 00:02:33.710 ⇒ 00:02:36.170 Patrick Trainer: And so I’ve actually
26 00:02:37.010 ⇒ 00:02:44.759 Patrick Trainer: so for this. But this is a good way to kind of see everything. Be able to see all of this.
27 00:02:45.490 ⇒ 00:02:46.130 Uttam Kumaran: Yes.
28 00:02:46.410 ⇒ 00:02:53.380 Patrick Trainer: Okay, cool. So pretty much like what I was saying of, like, we have these kind of like groups
29 00:02:53.480 ⇒ 00:03:03.750 Patrick Trainer: of like what we should be focusing on and in on the highest level. That’s observability. What’s running current state.
30 00:03:04.090 ⇒ 00:03:20.559 Patrick Trainer: how we do work and then how we also say, like like, when we find bugs like how we set up our issues and then also going into documentation and then writing clearly, especially because we want to be able to
31 00:03:20.970 ⇒ 00:03:27.739 Patrick Trainer: understand our our work in kind of like a linear fashion. Just see what we’re doing across time.
32 00:03:28.434 ⇒ 00:03:40.721 Patrick Trainer: And so going into like observability. I know you can all kind of read this, and it’s kind of like a Powerpoint slide, so I don’t want to go through necessarily every
33 00:03:41.770 ⇒ 00:03:45.230 Patrick Trainer: bullet point there, but as as a high level
34 00:03:45.490 ⇒ 00:03:54.000 Patrick Trainer: like for observability, we should be able to zoom in and out and understand our data landscape and by landscape I’m
35 00:03:54.240 ⇒ 00:04:04.589 Patrick Trainer: kind of talking about the entire or the entirety of like all facets of our infrastructure. So like our Github actions. Our pipelines
36 00:04:05.140 ⇒ 00:04:31.870 Patrick Trainer: deployed dashboards, etc, etc, a, and also including, like we have light dash, real and then all these other tools kind of like what is currently deployed and then being able to zoom in to like this is the code that powers XY, and Z. These are the workflows that that update XY, and z, and so on, and so on
37 00:04:32.652 ⇒ 00:04:39.259 Patrick Trainer: that kind of like segues into like the the current state of production.
38 00:04:39.893 ⇒ 00:04:45.100 Patrick Trainer: And this goes into like, how do we find what we need?
39 00:04:45.569 ⇒ 00:04:57.249 Patrick Trainer: And so that’s either like establishing like data dictionaries or naming conventions understanding dependencies upstream and downstream and then as well as like, where
40 00:04:57.400 ⇒ 00:05:12.839 Patrick Trainer: do others go in discovering those those insights? It’s like, so we want to basically like tee up that infrastructure to where somebody can like have little context and be able to find it quickly.
41 00:05:13.744 ⇒ 00:05:25.030 Patrick Trainer: Next part is going into like current state, and that that’s like, what’s running. So we want to be able to understand. Like
42 00:05:25.543 ⇒ 00:05:35.220 Patrick Trainer: like load times, we’re like, like freshness volume, that sort of things which we kinda have but then we also want to understand.
43 00:05:35.601 ⇒ 00:05:55.828 Patrick Trainer: What’s happening in our warehouse. Specifically so like, what tables see the most action which tables from the dashboards are getting the most heavy use. How are our users using those? What are they doing? In those areas? What problems are they having? And so on and so forth.
44 00:05:56.270 ⇒ 00:05:57.630 Patrick Trainer: and then
45 00:05:57.640 ⇒ 00:06:07.975 Patrick Trainer: going forth of like, how are we understanding and tracking our build times like, how fast can we execute our
46 00:06:09.410 ⇒ 00:06:11.430 Patrick Trainer: there are like pull requests
47 00:06:11.480 ⇒ 00:06:28.610 Patrick Trainer: when we merge and domain? How long does it take? Essentially, it’s like it’s it’s looking at our cycle times of like, how what’s our our turnaround time on from start to development to deployed in production, and then that iteration cycle between that
48 00:06:28.950 ⇒ 00:06:46.479 Patrick Trainer: and so, and then it also hits into alerting in monitoring and being able to be like, how do we become aware when something either breaks or is down? And so that we’re able to catch it before our users, either like
49 00:06:46.630 ⇒ 00:06:52.310 Patrick Trainer: brain force customers, or like us, are trying to do it. And so we’re not like
50 00:06:52.610 ⇒ 00:07:01.070 Patrick Trainer: being like like kind of like today, like downstream tests were failing. And we weren’t necessarily aware of that. Like, we need to
51 00:07:01.500 ⇒ 00:07:03.170 Patrick Trainer: understand that like.
52 00:07:03.860 ⇒ 00:07:07.249 Patrick Trainer: okay, like, only a section of our tests
53 00:07:07.530 ⇒ 00:07:24.719 Patrick Trainer: are cover covering like half of our code. And then a 3rd of our tests were weren’t running. And so like, there’s a coverage mismatch that we were somewhat blind to. And so we want to be able to kind of like, see that holistically
54 00:07:25.590 ⇒ 00:07:30.349 Patrick Trainer: and then there goes into like the next bullet point of of workflow.
55 00:07:30.590 ⇒ 00:07:32.119 Patrick Trainer: And so this
56 00:07:32.470 ⇒ 00:07:34.879 Patrick Trainer: mainly hints on kind of like.
57 00:07:35.544 ⇒ 00:07:39.905 Patrick Trainer: just like best practices in how we want to.
58 00:07:41.406 ⇒ 00:07:45.440 Patrick Trainer: like, build out like brainforged data engineering.
59 00:07:45.470 ⇒ 00:08:12.519 Patrick Trainer: and so kind of like establishing either like best practices or conventions. How do we even down to like? How do we structure our project? So take, for example. Like we bring on an intern right? And they come in. They’re writing like documentation. What should they name the documentation? How should their documents like
60 00:08:12.620 ⇒ 00:08:20.940 Patrick Trainer: should it be in like passive or active voice? What like? How should it actually be written, or
61 00:08:21.789 ⇒ 00:08:35.060 Patrick Trainer: like, even down to the tools that we use, like, what like introducing like Vs code and the extensions that help us do stuff fast and then also it. It gives people
62 00:08:35.490 ⇒ 00:08:52.710 Patrick Trainer: kind of like the confidence to either create new files, new folders and then establishing that hierarchy so that there it’s not like this like, Oh, does this go there? But it doesn’t go there. They have kind of like, a guiding map of like.
63 00:08:52.760 ⇒ 00:08:59.530 Patrick Trainer: okay, this relates to this area. I’m gonna put this in here, and then it also give empowers them to like.
64 00:08:59.880 ⇒ 00:09:04.999 Patrick Trainer: organize it in a way that fits in the larger system that may not already be there.
65 00:09:05.020 ⇒ 00:09:13.919 Patrick Trainer: One of the problems that a lot of teams run into it’s like a like a folder structure will be created by
66 00:09:14.560 ⇒ 00:09:16.369 Patrick Trainer: Udem 6 months ago.
67 00:09:16.470 ⇒ 00:09:39.550 Patrick Trainer: and then we’ll never create another folder anywhere, and we’ll just jam everything into the In into the existing ones where that folder structure may be like may have been good or good enough 6 months ago, but now that things have changed and our reality is different, it’s like you can’t predict the future. And so you gotta be able to kind of like, react to that.
68 00:09:40.500 ⇒ 00:09:41.920 Patrick Trainer: This.
69 00:09:42.360 ⇒ 00:09:46.100 Patrick Trainer: then sits kind of like below. How we set up
70 00:09:46.622 ⇒ 00:09:47.920 Patrick Trainer: like our issues
71 00:09:48.341 ⇒ 00:09:54.819 Patrick Trainer: in everything. And like, how do we? And this also kind of goes into our dev workflow
72 00:09:54.880 ⇒ 00:10:10.780 Patrick Trainer: of like, how are we showing that like either, we found a bug. How do we document that? How do we talk about re reproducibility? Basically give context to your future self? If you’re like.
73 00:10:11.050 ⇒ 00:10:38.070 Patrick Trainer: have it, and then you leave to go back to it, or to also, and also to give context to like those around us or or us and so that like, we can see a ticket, understand what’s happening, and then use that context in in respect to like the context that we already have like, if if Jacob finds a bug
74 00:10:38.210 ⇒ 00:10:49.369 Patrick Trainer: and it’s described, well, I may be able to like, see that, and not necessarily think it’s like something related strictly to a dashboard. But understand like, how my work can affect
75 00:10:49.921 ⇒ 00:10:57.920 Patrick Trainer: that and like it it, because Jacob or somebody else isn’t gonna have the context of, like the system
76 00:10:58.150 ⇒ 00:11:06.360 Patrick Trainer: that we’re running on a on in another team or in another section. It allows, like each team to kind of relate of like.
77 00:11:06.360 ⇒ 00:11:29.418 Patrick Trainer: okay, how how am I involved in this? And like that, can be set up through like either templates or just examples, etc, etc, and like what is like a good baseline, or at least necessary things to include in in that. And that goes into setting up like work as well. So like when we
78 00:11:29.860 ⇒ 00:11:33.559 Patrick Trainer: create issues and like, we put them into
79 00:11:34.270 ⇒ 00:11:50.143 Patrick Trainer: like a ready for this week state, like, we want to make sure that like, things are labeled correctly. The issue is good. Somebody’s actually assigned to that issue and then, like we’ve established or
80 00:11:50.670 ⇒ 00:11:57.170 Patrick Trainer: tagged on labels to that. So we can understand kind of like, how it fits into the bigger picture
81 00:11:57.590 ⇒ 00:12:00.889 Patrick Trainer: and then going into writing clearly.
82 00:12:00.940 ⇒ 00:12:02.320 Patrick Trainer: So it’s like
83 00:12:03.480 ⇒ 00:12:16.290 Patrick Trainer: development work is more than just writing code. It’s like, at the end of the day we are writers or authors of many different things, not just code. And like
84 00:12:16.810 ⇒ 00:12:18.619 Patrick Trainer: as we need to
85 00:12:18.830 ⇒ 00:12:36.710 Patrick Trainer: write code, that it, like one works but 2 is readable and clear we need to be able to do that in our documentation as well, and that trickles all the way down into commit messages. So instead of just going in saying like.
86 00:12:38.240 ⇒ 00:12:49.469 Patrick Trainer: shit, this was hard or just like bullshit, commit messages where the or even the same, commit messages like, we want to be able to understand like, okay, this commit
87 00:12:49.510 ⇒ 00:13:00.010 Patrick Trainer: introduces this or like. If this, if this commit, is merged, it will do this, or this is added to it, so that we can see in our get history.
88 00:13:00.180 ⇒ 00:13:06.420 Patrick Trainer: understand at a glance all of the changes that were made, instead of just having some like opaque
89 00:13:07.110 ⇒ 00:13:08.962 Patrick Trainer: kind of like
90 00:13:10.120 ⇒ 00:13:16.370 Patrick Trainer: analogous type message where it really doesn’t mean anything like you might as well leave it blank
91 00:13:17.071 ⇒ 00:13:27.498 Patrick Trainer: and then that goes into like reviewing pull requests of like, How like do we actually look at the code? Do we offer constructive feedback
92 00:13:27.930 ⇒ 00:13:31.879 Patrick Trainer: or like, actually like.
93 00:13:32.190 ⇒ 00:13:50.140 Patrick Trainer: take the time to make to double check, to proofread somebody’s work rather than just like taking a glance over it and given it. The looks good to me thumbs up and and merging it, and then everything break like kind of like the goal there is to never break the build on main
94 00:13:50.545 ⇒ 00:14:02.259 Patrick Trainer: and like, break it all you need and pull requests. But it it. It should be like a a, a pairing, or or even with multiple reviewers. It should be like a peer reviewed thing
95 00:14:03.471 ⇒ 00:14:21.120 Patrick Trainer: this. Then writing clearly, filters down into documentation, which it kind of hits in like the commit messages and how we do issues. But this is more kind of like creating run books. What is this about?
96 00:14:21.791 ⇒ 00:14:25.010 Patrick Trainer: How does this work, or what are the overall concepts?
97 00:14:27.030 ⇒ 00:14:28.470 Patrick Trainer: Excuse me, there’s a
98 00:14:28.810 ⇒ 00:14:40.290 Patrick Trainer: a framework that I really like. It’s called diataxis. And it’s basically there’s like A, it’s like a 2 way, like a Cartesian axis of like
99 00:14:40.490 ⇒ 00:14:48.069 Patrick Trainer: of different types of documents. There’s like how documents there’s like demonstration documents.
100 00:14:48.100 ⇒ 00:14:59.900 Patrick Trainer: There’s like more technical kind of documents, and then I’ll I’ll provide the link here. But then, also like tutorials as well. And so we.
101 00:14:59.900 ⇒ 00:15:12.600 Uttam Kumaran: Yeah, that’s great. Actually, cause I, we have like all these different types. And I even don’t have a framework for doing that. So I would love to have like. There’s a difference between, how do we do this versus like, yeah, technical docs, like.
102 00:15:12.600 ⇒ 00:15:14.660 Patrick Trainer: Right? Right? Yeah.
103 00:15:14.830 ⇒ 00:15:18.450 Patrick Trainer: yeah, because it’s it. It all goes into like.
104 00:15:18.680 ⇒ 00:15:25.609 Patrick Trainer: kind of like, what is the intent of the document? And who’s reading it like if I’m
105 00:15:26.030 ⇒ 00:15:31.990 Patrick Trainer: like, say, debugging something or need like, there’s an issue on on Snowflake, like
106 00:15:32.330 ⇒ 00:15:37.200 Patrick Trainer: I don’t really need to look at like the guide of
107 00:15:37.460 ⇒ 00:15:42.550 Patrick Trainer: what snowflake like of the philosophical parts of like how snowflake
108 00:15:42.770 ⇒ 00:16:05.650 Patrick Trainer: partitions, tables, or something, I need to know, like the commands, or something, or like this is like Step one. You go into the table. Step 2. Query, the information, schemas, etc, etc, going. And then that goes into also like run books. So like when we see an alert pop up in slack and it looks critical like
109 00:16:06.280 ⇒ 00:16:28.290 Patrick Trainer: being able to quickly like, find and disseminate like, okay, this is, this is what’s going on. This is what’s breaking and like, basically, it’s like the, it’s like a fire drill like you. There’s stop, drop and roll you find the nearest exit. You go out. You go a quarter mile out. You stand in the
110 00:16:28.580 ⇒ 00:16:36.110 Patrick Trainer: parking garage or something. It’s like being being prepared and proactive. For in like anticipating
111 00:16:36.180 ⇒ 00:16:38.140 Patrick Trainer: things when they break.
112 00:16:38.270 ⇒ 00:16:39.300 Patrick Trainer: And so.
113 00:16:39.300 ⇒ 00:16:53.047 Nicolas Sucari: For, for example, now like that, we are having, like all the all those tests like we don’t have like a guide, not a guide, or, for example, a documentation on like, what is each of those tests that we are doing right like?
114 00:16:53.710 ⇒ 00:16:56.520 Nicolas Sucari: Is there a place where someone like
115 00:16:56.620 ⇒ 00:16:58.850 Nicolas Sucari: joining, joining us
116 00:16:59.141 ⇒ 00:17:10.399 Nicolas Sucari: and having no clue of like. What those tests are like is a place where to go and see. Like, okay, this test is going through this this our test is going. There is no place to see that right.
117 00:17:10.400 ⇒ 00:17:17.480 Patrick Trainer: Right? Right? Right? Or think of like to like the anomaly detection. It’s like, okay, we’ve detected
118 00:17:17.660 ⇒ 00:17:20.300 Patrick Trainer: a big spike in order items.
119 00:17:20.869 ⇒ 00:17:21.869 Nicolas Sucari: Yeah, what should we do?
120 00:17:21.869 ⇒ 00:17:22.909 Patrick Trainer: Oak. Okay.
121 00:17:22.910 ⇒ 00:17:23.500 Nicolas Sucari: A.
122 00:17:23.500 ⇒ 00:17:35.599 Patrick Trainer: What is what like? What’s next? What should we go into the table and see what was added in the last day? Or should we look at a different table.
123 00:17:35.600 ⇒ 00:17:36.990 Nicolas Sucari: Yeah, yeah, so that’s perfect.
124 00:17:37.391 ⇒ 00:17:39.798 Patrick Trainer: It’s like it. It’s giving like,
125 00:17:40.886 ⇒ 00:17:51.633 Patrick Trainer: I’ve or even like I’ve done this before. This is how we should do it the next time and then under kind of like having that retrospective of like this.
126 00:17:52.060 ⇒ 00:17:53.139 Patrick Trainer: do this
127 00:17:54.080 ⇒ 00:17:57.839 Patrick Trainer: and and kind of like being able to iterate on that
128 00:17:57.960 ⇒ 00:18:12.319 Patrick Trainer: written down so somebody can go go and find it, and if they find something new they go back and write and show like what they did the steps, the processes and then, like what?
129 00:18:12.570 ⇒ 00:18:13.940 Patrick Trainer: Why did they
130 00:18:15.330 ⇒ 00:18:18.879 Patrick Trainer: think or like? What was what was their thought process? There.
131 00:18:19.370 ⇒ 00:18:41.870 Nicolas Sucari: And and that’s why, yeah, I like, I agree with you, like, we need to start like documenting and having like that framework in order to understand what types document we need to to do. And probably it would be easier also, once we identified those like different types create templates on that people can do them like quickly. That is so obviously possible. And that’s probably why I
132 00:18:41.870 ⇒ 00:18:54.060 Nicolas Sucari: also like introduce notion, and we are starting to use notion as our like Wiki for everything. So we can have like a space there with all of that information I know sometimes it’s easier for you guys to have it on Github. But
133 00:18:54.700 ⇒ 00:18:56.089 Nicolas Sucari: it is not like
134 00:18:56.230 ⇒ 00:19:01.719 Nicolas Sucari: Github is not so like, you cannot go and easily find those documents as in notion. Probably.
135 00:19:02.230 ⇒ 00:19:04.750 Nicolas Sucari: But yeah, I mean, we can.
136 00:19:04.750 ⇒ 00:19:11.530 Patrick Trainer: And I believe there’s I mean this is neither here nor there. But I believe there’s github integrations with notion that can keep things in sync.
137 00:19:11.530 ⇒ 00:19:12.679 Nicolas Sucari: Yeah, so we can get.
138 00:19:12.680 ⇒ 00:19:19.440 Patrick Trainer: Github also has wikis, which. So we don’t necessarily need to write everything like in.
139 00:19:19.440 ⇒ 00:19:20.000 Nicolas Sucari: Yeah.
140 00:19:20.356 ⇒ 00:19:37.443 Patrick Trainer: In a folder, or in Markdown, or anything like it, can be directly in that wiki. And we can have, like everything deep linked, and but that’s neither here nor there yet. And so, like, we’ll we’ll handle that later. But I I totally agree with you.
141 00:19:37.960 ⇒ 00:19:38.999 Patrick Trainer: basically kind of.
142 00:19:39.000 ⇒ 00:19:40.400 Uttam Kumaran: So I can ask a question.
143 00:19:40.400 ⇒ 00:19:41.350 Patrick Trainer: Yeah, go for it.
144 00:19:42.330 ⇒ 00:19:46.586 Uttam Kumaran: On the on the notion of like Async and
145 00:19:47.330 ⇒ 00:19:49.809 Uttam Kumaran: like tools. The one thing that I
146 00:19:50.130 ⇒ 00:19:58.040 Uttam Kumaran: we’ve I think we we’re gonna the one thing I learned about this business is like everything in stages. But there’s also a way for us to just have a North star.
147 00:19:58.080 ⇒ 00:20:15.809 Uttam Kumaran: One thing that I realized, you know, just being an engineering is that when I was talking to Nico about this this morning is like meetings about meetings or meetings, about updates are not a good use of anybody’s time. Instead, what actually gives everybody a lot of energy is meetings about like solving problems, right? And so.
148 00:20:15.810 ⇒ 00:20:16.140 Patrick Trainer: Right.
149 00:20:16.140 ⇒ 00:20:23.740 Uttam Kumaran: The one thing that I would love to see how we can lean on tools of automation is using things like slack, but especially things like loom
150 00:20:23.770 ⇒ 00:20:25.760 Uttam Kumaran: to actually record
151 00:20:26.310 ⇒ 00:20:37.049 Uttam Kumaran: like voice or walkthroughs. And I want, like engineers on our team to have those tools at their capability, basically like, what is a run book of like what you should do before scheduling like a meeting.
152 00:20:37.050 ⇒ 00:20:38.160 Patrick Trainer: Right, right.
153 00:20:38.517 ⇒ 00:20:46.380 Uttam Kumaran: And you know, like all of us, work differently. And you know we can all be productive leveraging these Async methods.
154 00:20:46.400 ⇒ 00:20:52.119 Uttam Kumaran: and I want to try to lean on some of that. So one thing is like, I think loom is probably like a next
155 00:20:52.150 ⇒ 00:21:05.339 Uttam Kumaran: thing that we can start to look into but that’s something that you know. It’d be great to even have like trainings and and stuff on that like how to record your 1st name and publish that, or like how to add that to an issue, and things like that.
156 00:21:05.340 ⇒ 00:21:13.809 Patrick Trainer: Right? Right? Yeah. And it it’s just to double tap on that. It’s like it goes into like, how do we set
157 00:21:13.910 ⇒ 00:21:34.190 Patrick Trainer: ourselves and our team up to be asynchronous as possible, because, like even right now, like, we have people across many time zones, especially like Brian, like, he lives on our time zone, basically. But then, on the weekend like has to shift and like, take care of his kids. Which is probably really difficult, like, how.
158 00:21:34.190 ⇒ 00:21:39.759 Uttam Kumaran: So gonna we’re also gonna continue to hire people on any time zone just looking for the smartest people, whoever they are.
159 00:21:39.760 ⇒ 00:21:42.319 Patrick Trainer: Exactly. And so like
160 00:21:42.600 ⇒ 00:21:45.030 Patrick Trainer: we don’t want necessarily to
161 00:21:45.160 ⇒ 00:21:46.979 Patrick Trainer: PIN people to
162 00:21:47.150 ⇒ 00:21:50.310 Patrick Trainer: us time if they’re not in the Us like. That’s
163 00:21:50.410 ⇒ 00:22:02.590 Patrick Trainer: not really fair at least, in my opinion. Like, how can like if I’m working on an issue, or something like, How can I get halfway done? And then
164 00:22:03.090 ⇒ 00:22:27.750 Patrick Trainer: confidently pass it off to somebody else, and they’d be able to pick off or pick up where I left off and we should be able to do that really at any time, or even just for, like myself, say, I, am working on an issue again today. And then S, some big fire happens that takes the whole entire next day.
165 00:22:28.810 ⇒ 00:22:30.879 Patrick Trainer: day, 2 after that.
166 00:22:31.330 ⇒ 00:22:53.840 Patrick Trainer: like, you’re not back in that we in those weeds. And you need to come back up to speed and like be in that mindset that you were 2 days ago. How do you pick that up like almost instantly? And that’s that’s like the true nature of like Async programming. And it’s like, I think of it very much like open source projects. It’s like any like
167 00:22:54.000 ⇒ 00:22:59.710 Patrick Trainer: anybody can commit or make a pull request to an open source project.
168 00:22:59.940 ⇒ 00:23:06.349 Patrick Trainer: and the hundreds, or however many people are working on that project like
169 00:23:06.420 ⇒ 00:23:17.569 Patrick Trainer: they can work at any given time like it doesn’t matter when they’re doing it. They’re just like they do the work when it’s most convenient to them, or when they’re most like
170 00:23:17.740 ⇒ 00:23:24.519 Patrick Trainer: of of mind and and contribute that way it kind of like takes the
171 00:23:24.760 ⇒ 00:23:29.859 Patrick Trainer: like the pressure off and allows people to kind of like, do their best work
172 00:23:30.870 ⇒ 00:23:33.409 Patrick Trainer: that I mean, that’s my, that’s my
173 00:23:33.580 ⇒ 00:23:35.480 Patrick Trainer: pretty strong opinion on that.
174 00:23:35.810 ⇒ 00:23:54.180 Patrick Trainer: And so so that goes into so we have like these areas of focus that we’re looking in and like these can be exactly like disseminated out into tickets like, when were things last loaded like, I’m sure, Zoom, you could think of 10
175 00:23:54.260 ⇒ 00:23:55.940 Patrick Trainer: different things of like.
176 00:23:56.560 ⇒ 00:24:07.339 Patrick Trainer: okay, like, what can we look at and understand what we’re loading, or even like, how long are build times like, that’s a an immediate like. Okay, let’s track our our
177 00:24:07.380 ⇒ 00:24:15.379 Patrick Trainer: Github actions usage over time and understand like, if our things are getting slow or fast and like
178 00:24:15.600 ⇒ 00:24:18.340 Patrick Trainer: like, it is just kind of like.
179 00:24:18.730 ⇒ 00:24:31.450 Patrick Trainer: this isn’t meant to be like just this. But it it like sets up us up for like, these just yeah areas of focus, like, what should we be concentrating on?
180 00:24:32.413 ⇒ 00:24:38.349 Patrick Trainer: And then this pairs with this, these guiding principles?
181 00:24:38.935 ⇒ 00:24:44.750 Patrick Trainer: And so these are just like practices that I’ve observed over like my past.
182 00:24:44.760 ⇒ 00:24:55.940 Patrick Trainer: what 8 years of data engineering and just like all of the shit that I’ve read on like from top tier, like Fang company development
183 00:24:56.817 ⇒ 00:25:03.752 Patrick Trainer: guidelines, and like what it means to like, be a good developer or a senior developer, or
184 00:25:04.250 ⇒ 00:25:07.020 Patrick Trainer: just like what is good
185 00:25:07.080 ⇒ 00:25:08.250 Patrick Trainer: development
186 00:25:08.750 ⇒ 00:25:24.189 Patrick Trainer: in terms of data, engineering and data teams. And so like, I said, here, it’s like, it’s not meant to be dogmatic. But these are things that like just work really well, and that we should strive to to
187 00:25:24.220 ⇒ 00:25:26.230 Patrick Trainer: the like aim for.
188 00:25:26.400 ⇒ 00:25:32.060 Patrick Trainer: And so that goes into discoverability, minimizing cognitive load.
189 00:25:33.030 ⇒ 00:25:34.729 Patrick Trainer: giving everybody like
190 00:25:34.770 ⇒ 00:25:41.139 Patrick Trainer: data, intuition, making use robust and being friendly to the dev
191 00:25:41.701 ⇒ 00:25:45.790 Patrick Trainer: and so kind of like with discoverability, like
192 00:25:45.950 ⇒ 00:25:50.760 Patrick Trainer: when you’re 1st starting out, or even like a user like you don’t know what you don’t know.
193 00:25:50.790 ⇒ 00:25:53.910 Patrick Trainer: And if you’re sitting there faced with
194 00:25:54.020 ⇒ 00:25:56.589 Patrick Trainer: a thousand different tables in Snowflake.
195 00:25:56.690 ⇒ 00:26:02.770 Patrick Trainer: or you’re start, or somebody starts working on a Dbt project. For example, like.
196 00:26:02.960 ⇒ 00:26:30.559 Patrick Trainer: Where do you go to look look for something? Or where do you go to to find your change and like that takes like very valuable time to like, see where things are, or input that into context, or put it into your mental model. And so we want to make that as frictionless as possible. And the big kind of like point here is like the need for an intuitive taxonomy.
197 00:26:30.710 ⇒ 00:26:44.339 Patrick Trainer: And that’s like, think of how a library is organized. We essentially that, like they have like history, science, it’s the Dewey decimal system. And like, we essentially wanna like remodel that
198 00:26:45.330 ⇒ 00:26:52.499 Patrick Trainer: in terms of our data, our data warehouse, our metrics or kpis or dashboards. So you can like.
199 00:26:52.750 ⇒ 00:27:02.369 Patrick Trainer: just drive down and understand. Like, okay, I’m working on orders and need to go into here and do this. And like, I’ll find the order items. These are the things that
200 00:27:02.820 ⇒ 00:27:04.120 Patrick Trainer: so on and so forth.
201 00:27:04.820 ⇒ 00:27:13.950 Patrick Trainer: that segues into minimizing the like cognitive load. And that’s making things as simple as possible, not necessarily just like
202 00:27:14.812 ⇒ 00:27:21.740 Patrick Trainer: like, not easy and not full full fledged. I don’t mean that, I mean as like.
203 00:27:22.200 ⇒ 00:27:25.029 Patrick Trainer: how do you pair down the cruft.
204 00:27:25.050 ⇒ 00:27:29.729 Patrick Trainer: or like extra extraneous things that are just distracting
205 00:27:29.860 ⇒ 00:27:50.860 Patrick Trainer: and like really distill, like the key points in in what you’re doing. But then also making it easy to again like, kind of like, discover these things. It’s just to basically like stop confusion. Like one example is like Dan using the old vital signs while we’re
206 00:27:50.980 ⇒ 00:27:54.439 Patrick Trainer: pushing the new vital signs like.
207 00:27:54.460 ⇒ 00:28:09.190 Patrick Trainer: it’s he was having to like, think about things. Okay, use this. Don’t use this one, this tile, this broken. So go over here. It just keeps things. It. It makes things difficult. And there’s confusion. And we need to to minimize that
208 00:28:09.950 ⇒ 00:28:14.879 Patrick Trainer: going for further. We want to have intuitive data
209 00:28:14.890 ⇒ 00:28:16.480 Patrick Trainer: and basically
210 00:28:16.930 ⇒ 00:28:42.499 Patrick Trainer: make it easy for our users like pool parts, etc, or even like our analysts. Or this is more kind of like in context of like a larger company where we may have like support or sales, or like C-suite of basically like, how do you get non technical users that aren’t seeing this day to day
211 00:28:42.500 ⇒ 00:28:50.929 Patrick Trainer: up to speed and kind of like, understand the causes and effects relationships and what they’re actually looking at.
212 00:28:50.940 ⇒ 00:28:52.689 Patrick Trainer: So they’re not just like
213 00:28:53.250 ⇒ 00:28:56.000 Patrick Trainer: making their making assumptions.
214 00:28:56.290 ⇒ 00:29:21.930 Patrick Trainer: often making wrong assumptions. And like, it’s it’s so they don’t. So they don’t do that. And that’s like a huge step to making things self serve. It’s like we can. We can set up like dashboards galore but if they don’t know how to use it, and don’t understand, like the underlying of like, what this actually is like there
215 00:29:21.970 ⇒ 00:29:23.510 Patrick Trainer: kind of like useless
216 00:29:25.800 ⇒ 00:29:30.879 Patrick Trainer: Then we go into usability that basically just makes things like, does it work?
217 00:29:31.446 ⇒ 00:29:39.130 Patrick Trainer: And it is it working fast enough for what we need? And and can we maintain it?
218 00:29:39.440 ⇒ 00:29:41.219 Patrick Trainer: That’s just like
219 00:29:41.870 ⇒ 00:29:47.340 Patrick Trainer: boiled down? Yeah, does it work and then, being dev friendly
220 00:29:47.470 ⇒ 00:29:49.210 Patrick Trainer: is like.
221 00:29:49.550 ⇒ 00:30:15.769 Patrick Trainer: we’ve got to consider ourselves as users of this, these data or data pipelines or workflows, orchestration, etc. Those are all kind of the same. But like you get what I’m saying it. It’s we. We need to be able to like work and make contributions in that, or even say, like, Hey, this doesn’t make sense in an easy and intuitive way
222 00:30:16.179 ⇒ 00:30:29.019 Patrick Trainer: to to make like our jobs easier on ourselves. And so like we already do that through like automation. And so it’s like, it’s things like those to where we can
223 00:30:29.240 ⇒ 00:30:35.190 Patrick Trainer: take out the extraneous bullshit and like focus on what actually matters.
224 00:30:37.950 ⇒ 00:30:39.320 Nicolas Sucari: Perfect. Yep.
225 00:30:40.850 ⇒ 00:30:47.616 Patrick Trainer: And so that’s that. That’s kinda my my spiel here. And
226 00:30:48.600 ⇒ 00:30:56.199 Patrick Trainer: I see these as kind of like a a living document, and we can form them in terms of like
227 00:30:56.760 ⇒ 00:31:01.159 Patrick Trainer: what makes sense most to to our company.
228 00:31:02.210 ⇒ 00:31:10.400 Patrick Trainer: like, I said, like I wrote this a couple of years ago for a different company. And so it like it may not line up exactly, but
229 00:31:10.440 ⇒ 00:31:22.069 Patrick Trainer: I think I’ve made it like general enough to where like it’s, it’s a really good foundation for us to build upon. And then we just like fill in the specifics.
230 00:31:24.790 ⇒ 00:31:25.450 Patrick Trainer: Jeff.
231 00:31:25.450 ⇒ 00:31:33.159 Nicolas Sucari: I think for me, it’s it’s perfect. Yeah. I think it’s generally enough in order to start. And as you said, this is, gonna be a living document, and we can yeah.
232 00:31:33.350 ⇒ 00:31:34.770 Nicolas Sucari: make it grow
233 00:31:35.161 ⇒ 00:31:39.690 Nicolas Sucari: but this. But use this as a starting point is kind of good. I think.
234 00:31:39.890 ⇒ 00:31:40.490 Patrick Trainer: Cool.
235 00:31:40.650 ⇒ 00:31:42.320 Patrick Trainer: Yeah. And so
236 00:31:42.540 ⇒ 00:31:43.870 Patrick Trainer: I have.
237 00:31:45.770 ⇒ 00:31:47.639 Patrick Trainer: like a Pr open.
238 00:31:48.415 ⇒ 00:31:54.980 Patrick Trainer: I just put it into like our actions repo. I’m kinda using
239 00:31:55.460 ⇒ 00:31:59.019 Patrick Trainer: brain forge actions repo as like
240 00:32:00.230 ⇒ 00:32:01.860 Patrick Trainer: kind of like the internal
241 00:32:02.190 ⇒ 00:32:12.759 Patrick Trainer: stuff, or like stuff that doesn’t relate to clients. There’s probably a better spot for it, but that even goes into like the cognitive load and taxonomy of like.
242 00:32:12.860 ⇒ 00:32:23.810 Patrick Trainer: Where the hell do we put this? Does it go here? Here do we drop into pool parts just as a link. It’s like, we don’t have that plan at the moment. And so it’s just, it’s still kind of ubiquitous.
243 00:32:25.010 ⇒ 00:32:26.239 Patrick Trainer: But yeah, I just
244 00:32:26.270 ⇒ 00:32:30.130 Patrick Trainer: drop that in the chat. So if
245 00:32:30.160 ⇒ 00:32:32.460 Patrick Trainer: we want to make edits
246 00:32:32.500 ⇒ 00:32:34.000 Patrick Trainer: to this
247 00:32:36.470 ⇒ 00:32:37.790 Patrick Trainer: like, let’s
248 00:32:38.040 ⇒ 00:32:48.020 Patrick Trainer: like check check out the branch and make commits against it, or, like, create your own branch from it, and merge it back into this, and then we can
249 00:32:48.180 ⇒ 00:32:54.950 Patrick Trainer: kind of like, codify it and merge it back into like what our like production document docs look like.
250 00:32:57.560 ⇒ 00:32:58.310 Patrick Trainer: Yes.
251 00:33:03.113 ⇒ 00:33:16.856 Uttam Kumaran: I think my question is, gonna be like, where? What is some? What is something actual? We can try this week. And then also thinking about like, potentially, like working on a little bit of a larger like.
252 00:33:17.650 ⇒ 00:33:36.750 Uttam Kumaran: you know, roadmap that could potentially break into tickets that affect clients would also affect our platform. I think there’s, I think this is like this, these doctor, like the North Star dogs of like each area that we want to know we’ve made impacts on but thinking, thinking at a
253 00:33:36.760 ⇒ 00:33:41.528 Uttam Kumaran: like a weekly level for this week, or even this month, are there like a couple of things that
254 00:33:41.840 ⇒ 00:33:47.450 Uttam Kumaran: you know seem like the highest value, things that we can begin to circulate around and make sure that we have
255 00:33:48.120 ⇒ 00:33:52.130 Uttam Kumaran: little tasks going. You know, primarily to
256 00:33:52.557 ⇒ 00:34:00.780 Uttam Kumaran: like, I think you, Augustine. And then any of the interns that are interested in in you know the data engineering network there.
257 00:34:00.780 ⇒ 00:34:06.439 Patrick Trainer: Right right? And so I do. Have that as well.
258 00:34:07.577 ⇒ 00:34:10.030 Patrick Trainer: And let me
259 00:34:10.230 ⇒ 00:34:13.340 Patrick Trainer: kind of like find this and bring it up.
260 00:34:14.489 ⇒ 00:34:15.760 Patrick Trainer: And y’all see this.
261 00:34:18.520 ⇒ 00:34:19.999 Patrick Trainer: that it’s like a.
262 00:34:23.190 ⇒ 00:34:23.800 Uttam Kumaran: I mean.
263 00:34:23.800 ⇒ 00:34:26.259 Nicolas Sucari: Can you? Can you? Can you make it bigger a little bit.
264 00:34:26.260 ⇒ 00:34:29.619 Patrick Trainer: Oh, oh, okay, I I was just seeing if you could.
265 00:34:30.239 ⇒ 00:34:32.189 Nicolas Sucari: We. We can see it in the screen.
266 00:34:33.929 ⇒ 00:34:35.289 Uttam Kumaran: I can just see the S.
267 00:34:36.120 ⇒ 00:34:37.050 Patrick Trainer: Oh, really.
268 00:34:37.710 ⇒ 00:34:38.630 Patrick Trainer: you can’t see.
269 00:34:38.639 ⇒ 00:34:39.299 Nicolas Sucari: Yeah, but.
270 00:34:40.030 ⇒ 00:34:41.920 Patrick Trainer: Like for anything.
271 00:34:41.920 ⇒ 00:34:43.370 Uttam Kumaran: And didn’t like switching.
272 00:34:43.370 ⇒ 00:34:44.250 Patrick Trainer: In the middle.
273 00:34:46.230 ⇒ 00:34:47.219 Nicolas Sucari: I can see it.
274 00:34:47.500 ⇒ 00:34:49.140 Patrick Trainer: Yeah, let me take Vs code out.
275 00:34:49.320 ⇒ 00:34:50.811 Uttam Kumaran: Okay. Maybe it’s just me.
276 00:34:52.199 ⇒ 00:34:55.289 Patrick Trainer: I mean, I can stop and reshare.
277 00:34:56.340 ⇒ 00:34:57.310 Nicolas Sucari: But I think that’s okay.
278 00:34:57.310 ⇒ 00:34:59.150 Uttam Kumaran: Okay. Now, I can see 24. I can see.
279 00:34:59.150 ⇒ 00:35:01.100 Nicolas Sucari: Yeah, if you go to 2424.
280 00:35:01.100 ⇒ 00:35:01.979 Patrick Trainer: Where are you plan? I’ll look.
281 00:35:01.980 ⇒ 00:35:02.320 Nicolas Sucari: Yeah.
282 00:35:02.320 ⇒ 00:35:03.100 Patrick Trainer: Window.
283 00:35:03.290 ⇒ 00:35:04.240 Nicolas Sucari: Yeah, perfect.
284 00:35:04.240 ⇒ 00:35:06.620 Patrick Trainer: Cool and so
285 00:35:07.030 ⇒ 00:35:09.299 Patrick Trainer: talking into into kind of like
286 00:35:09.420 ⇒ 00:35:11.720 Patrick Trainer: more discrete terms and tickets.
287 00:35:13.660 ⇒ 00:35:15.040 Patrick Trainer: hold on!
288 00:35:16.570 ⇒ 00:35:19.150 Patrick Trainer: I see my number is messed up there.
289 00:35:26.390 ⇒ 00:35:28.599 Patrick Trainer: And so we kind of have like these.
290 00:35:30.210 ⇒ 00:35:39.150 Patrick Trainer: kind of like higher level, and they go into kind of like our our principles of like or our focus areas of like, we have
291 00:35:39.450 ⇒ 00:35:45.090 Patrick Trainer: infrastructure that we need to do. We have some needs, maybe not yet
292 00:35:45.100 ⇒ 00:35:51.920 Patrick Trainer: in integration and orchestration. I’m thinking of, like our slack documents that we were talking about yesterday
293 00:35:53.183 ⇒ 00:35:59.370 Patrick Trainer: and then we’ve got our quality and monitoring and then alerting and incident management.
294 00:35:59.570 ⇒ 00:36:04.290 Patrick Trainer: and so like infrastructure, like, I mean, we’ve got most of this already set up
295 00:36:05.103 ⇒ 00:36:15.026 Patrick Trainer: like using snowflake and like managing our like actions. If we do need to set up. Set up any sort of like streaming data
296 00:36:15.710 ⇒ 00:36:27.364 Patrick Trainer: like, get that set up? How are we managing that infrastructure like, are we using terraform is everything as code like, how are we creating our our like real dashboards?
297 00:36:28.040 ⇒ 00:36:32.829 Patrick Trainer: going on, etc, etc. And then, like.
298 00:36:32.910 ⇒ 00:36:48.929 Patrick Trainer: we’re looking at data, integration, orchestration so like ingestion, how are we pulling things into our warehouse like we use 5 tran. But we seem to also be kind of like manually creating tables from Csvs. So we need to think about that.
299 00:36:49.509 ⇒ 00:36:59.630 Patrick Trainer: We need to think about how we manage our workflows. How do we know that they want work. How do we test them? How do like?
300 00:37:00.280 ⇒ 00:37:07.599 Patrick Trainer: Basically like, we set them up and then how do they all kinda like relate to each other. And do they make sense
301 00:37:08.021 ⇒ 00:37:25.829 Patrick Trainer: and then we’re looking into quality and monitoring. We’re doing a lot of this through like. Dbt tests as well as animal elementary. How do we know like this is like more of so like the current state of our systems? Like
302 00:37:26.160 ⇒ 00:37:41.549 Patrick Trainer: going into monitoring snowflake and our resources like, how much are we spending? Are we like brutalizing the bill? Are are we using or like over over provisioning warehouses, warehouse size.
303 00:37:41.550 ⇒ 00:37:56.249 Patrick Trainer: and where we can use something smaller? Or can we like repartition tables to make things work better and then we also like, we’re not necessarily there yet, but and I don’t know if elementary
304 00:37:57.032 ⇒ 00:37:59.480 Patrick Trainer: can work with logs.
305 00:37:59.870 ⇒ 00:38:05.800 Patrick Trainer: But I’ve put that in there as well. In in the past. I’ve used like
306 00:38:05.890 ⇒ 00:38:10.839 Patrick Trainer: aws like cloud trail and then like elastic search
307 00:38:11.261 ⇒ 00:38:23.909 Patrick Trainer: and like that cabana, and like Prometheus and those sorts of things setting that up data dogs super expensive. So I don’t. I don’t think we should use that
308 00:38:24.760 ⇒ 00:38:25.960 Patrick Trainer: and then we’ve.
309 00:38:25.960 ⇒ 00:38:35.429 Uttam Kumaran: Yeah, I think I think. And Pat, you know, like part of it is overall like we’re doing this for our sanity. But also I want to come to the table to clients, basically with.
310 00:38:35.730 ⇒ 00:38:42.080 Uttam Kumaran: you know, ideally, one option. But if they have budget constraints, then maybe 2 options. Obviously, hey, here. Here’s
311 00:38:42.090 ⇒ 00:39:02.209 Uttam Kumaran: cause we’re not only like here we’re not only like, Hey, we’ll do the labor. We’re also gonna go buy a bunch of shovels for you. And here are the shovels that we like to use. And here’s what we can use. And so across all this, I think it’s nice, because we have several iterations like getting at the perfect staff. But also this is part of gonna be part of my like sales disqualification processes.
312 00:39:02.270 ⇒ 00:39:07.950 Uttam Kumaran: I don’t. I know if we we need if we need these to guarantee success, then
313 00:39:08.000 ⇒ 00:39:19.690 Uttam Kumaran: we need to make sure that there’s budget from everybody to to do this and then that like. That’s basically how this cascades to the sales side. So I again, ideally, we have. We have everything from
314 00:39:20.120 ⇒ 00:39:29.710 Uttam Kumaran: warehouse to some sort of testing framework to Dbt to using Github actions. And then basically, I can roll that up and say, like, Hey, you’re a standard
315 00:39:29.780 ⇒ 00:39:47.999 Uttam Kumaran: your standard company like, probably you need like 2,000 a month for infrastructure budget. And like, that’s something that like, I have confidence in saying, cause everybody’s gonna want to scope that out and then again, as part of this, we also provide them with like. Here we have all of our cost management stuff. We have our alerting framework and
316 00:39:48.280 ⇒ 00:39:50.789 Nicolas Sucari: I mean, I think it will be worth
317 00:39:51.440 ⇒ 00:39:58.731 Nicolas Sucari: worth outing there also something about data security we are not seeing there. But yeah.
318 00:39:59.160 ⇒ 00:40:02.280 Patrick Trainer: It’s it’s basically like we’re selling that.
319 00:40:02.840 ⇒ 00:40:08.220 Patrick Trainer: We’re one professionals in the field and 2 like we know what the fuck we’re talking about
320 00:40:09.415 ⇒ 00:40:10.599 Patrick Trainer: and like
321 00:40:11.050 ⇒ 00:40:20.370 Patrick Trainer: 3, you’re getting an entire package like we’re not just like modeling your ship state or integrating your ship station in Amazon.
322 00:40:21.002 ⇒ 00:40:23.329 Patrick Trainer: Data. We’re providing, like
323 00:40:23.700 ⇒ 00:40:35.679 Patrick Trainer: the entire spectrum of like what an analytics workflow is. And we’re providing like to use an analogy of a car like we’re also putting airbags and seat belts in your car.
324 00:40:36.366 ⇒ 00:40:41.243 Patrick Trainer: Like we’re we’re we’re may making things safe and and giving
325 00:40:42.150 ⇒ 00:40:49.569 Patrick Trainer: like our users the peace of mind that they can trust us. And when we establish that trust like
326 00:40:50.420 ⇒ 00:40:52.020 Patrick Trainer: like, no. And
327 00:40:52.380 ⇒ 00:41:03.390 Patrick Trainer: that’s like that’s huge. That’s what’s gonna keep them like resubscribing coming back. And then also, just like giving us kind of like the like.
328 00:41:04.770 ⇒ 00:41:16.069 Patrick Trainer: almost like leeway in less of a like needing to to prove ourselves on, on every single update. It’s like, if we come out of the gate with like.
329 00:41:16.380 ⇒ 00:41:28.170 Patrick Trainer: we know, we’d need this. We know we need this and then XY, and Z. It’s they can have that confidence to where it’s like we’re not having to start from 0 every single time.
330 00:41:30.340 ⇒ 00:41:35.149 Patrick Trainer: then also this I mean, this is more related to
331 00:41:36.030 ⇒ 00:41:51.379 Patrick Trainer: like reporting. Well, yeah, relates to reporting and data vis but basically like understanding kind of like, what is already out there. What’s powering on are those things documented like? How should they be used? And and then also, like
332 00:41:52.450 ⇒ 00:41:55.420 Patrick Trainer: like knowledge, sharing and kind of like
333 00:41:55.860 ⇒ 00:41:58.889 Patrick Trainer: outlining what these things mean.
334 00:41:59.945 ⇒ 00:42:03.879 Patrick Trainer: And then we go into our
335 00:42:04.750 ⇒ 00:42:16.200 Patrick Trainer: continuous improvement and like a lot of devops like a lot of, I I think core data engineering work. It’s like a a split brain mix between
336 00:42:17.840 ⇒ 00:42:28.899 Patrick Trainer: like true engineering. And then, or like a like infrastructure work as well as a lot of like Sri type work.
337 00:42:29.010 ⇒ 00:42:33.360 Patrick Trainer: And so how do we set ourselves up for success in that.
338 00:42:33.650 ⇒ 00:42:35.540 Patrick Trainer: and like to
339 00:42:35.620 ⇒ 00:42:37.999 Patrick Trainer: go back to the previous document
340 00:42:38.040 ⇒ 00:42:39.050 Patrick Trainer: just like
341 00:42:39.190 ⇒ 00:43:01.840 Patrick Trainer: minimizing the cognitive load letting tests run automatically having things in like terraform, making things just like just work like when we push into main but then it also goes into like, not just continuous improvement of our code and brain force itself, but also of upscaling our people
342 00:43:02.230 ⇒ 00:43:24.659 Patrick Trainer: so like that goes into when writing good docs like, that’s how like we share knowledge. That’s how we like, get others thinking about other things and giving that context kind of that, standing on the shoulders of giant things but then also sending people into conferences or introducing them to people that we’ve talked to
343 00:43:25.200 ⇒ 00:43:34.309 Patrick Trainer: in the past. But then also we want to like Foster, that culture of of continuous learning and being like
344 00:43:34.600 ⇒ 00:43:40.799 Patrick Trainer: kind of like, just hungry to to understand things and also like.
345 00:43:41.040 ⇒ 00:43:51.270 Patrick Trainer: feel psychologically safe in the sense that like, it’s okay that you, if you don’t know something especially like if it’s new.
346 00:43:53.250 ⇒ 00:44:08.559 Patrick Trainer: but feeling safe in the sense to to say, Hey, I’m not really familiar with this, like, let’s get me up to speed without fearing any sort of like backlash or or judgment, and whatnot, because then that like
347 00:44:08.980 ⇒ 00:44:16.070 Patrick Trainer: that rises the tide and rise and tide floats all boats and and that’s really what we’re what we’re looking to do here.
348 00:44:20.110 ⇒ 00:44:20.860 Patrick Trainer: Yep.
349 00:44:21.670 ⇒ 00:44:22.600 Patrick Trainer: cool.
350 00:44:25.930 ⇒ 00:44:26.640 Uttam Kumaran: Yep.
351 00:44:26.640 ⇒ 00:44:31.923 Patrick Trainer: And so that’s that’s what I’ve prepared for, y’all, and I’ve spoken for
352 00:44:32.400 ⇒ 00:44:33.560 Patrick Trainer: a bit now. So.
353 00:44:33.560 ⇒ 00:44:35.299 Uttam Kumaran: No, this is great dude. This is.
354 00:44:35.680 ⇒ 00:44:46.120 Uttam Kumaran: this is like, I mean again, this is like everything. And like, this is how I think, too, because I’m like, I basically see all these 7 things all at once, all the time.
355 00:44:46.569 ⇒ 00:44:51.320 Uttam Kumaran: So for me, and and I assume, but particularly for Nico now is like
356 00:44:51.990 ⇒ 00:44:57.529 Uttam Kumaran: like, if we were to, if we were to go towards, if we were to rate each of these on like a
357 00:44:57.700 ⇒ 00:44:59.080 Uttam Kumaran: 5 out of 5.
358 00:44:59.220 ⇒ 00:45:07.549 Uttam Kumaran: And that may be the way we think of this is like, if we have, like a scale rating of of each of these, where are we today. And then what are the things we need to do
359 00:45:07.590 ⇒ 00:45:11.929 Uttam Kumaran: across both clients to make sure that we’re out of 5
360 00:45:11.930 ⇒ 00:45:12.580 Nicolas Sucari: Yeah, yeah.
361 00:45:12.580 ⇒ 00:45:18.839 Uttam Kumaran: You know, and that helps us one that helps us one. Again, it’s kind of your air bag analogy. But also
362 00:45:18.980 ⇒ 00:45:24.340 Uttam Kumaran: again, like, I want us to really dog through these types of things. Because this is.
363 00:45:24.370 ⇒ 00:45:29.089 Uttam Kumaran: I’m I’m really, really confident that on the sales side, this is what’s gonna allow us to really
364 00:45:29.270 ⇒ 00:45:36.259 Uttam Kumaran: breach like a different level of like enterprise contracts where we really have this stuff buttoned up.
365 00:45:36.450 ⇒ 00:45:43.209 Uttam Kumaran: Not only do we have a buttoned up on like how we do it. But the fact that, like we have the tools ready and everybody’s trained to do it.
366 00:45:43.400 ⇒ 00:45:44.910 Patrick Trainer: Right, right.
367 00:45:48.270 ⇒ 00:45:49.030 Uttam Kumaran: Yeah.
368 00:45:52.190 ⇒ 00:45:55.629 Nicolas Sucari: Can you, Patrick? I know you already have
369 00:45:55.640 ⇒ 00:45:59.039 Nicolas Sucari: that pull request with some of this stuff, but if you can like
370 00:45:59.636 ⇒ 00:46:01.819 Nicolas Sucari: share this, and whatever.
371 00:46:01.820 ⇒ 00:46:10.210 Patrick Trainer: Yeah, I’ll I’ll add this in. I just like, so that like, this is what I type all of my documents in first, st and then I paste them in.
372 00:46:11.130 ⇒ 00:46:11.760 Nicolas Sucari: Okay.
373 00:46:11.760 ⇒ 00:46:12.453 Patrick Trainer: It’s, yeah.
374 00:46:12.800 ⇒ 00:46:13.540 Nicolas Sucari: You said you have.
375 00:46:13.540 ⇒ 00:46:15.940 Patrick Trainer: Event envy all it’s it’s amazing.
376 00:46:16.740 ⇒ 00:46:21.889 Patrick Trainer: It’s so bare bones. It’s it’s just a note taking application. But it’s really cool.
377 00:46:21.890 ⇒ 00:46:22.490 Nicolas Sucari: I’m sorry.
378 00:46:22.490 ⇒ 00:46:26.340 Patrick Trainer: That like like, search is creating
379 00:46:27.860 ⇒ 00:46:36.540 Patrick Trainer: like issues. And it’s like, Here’s here’s yours, hey? And but then also, you can be like.
380 00:46:36.730 ⇒ 00:46:53.199 Patrick Trainer: Hello, like it. It searches at the same time like data, and then it pulls down things like that. But then you can also like type, and then it just immediately like, if something’s not there. When you search, you can just press, enter.
381 00:46:53.240 ⇒ 00:46:57.220 Patrick Trainer: and then it goes in, and then and then you can like deep link
382 00:46:57.290 ⇒ 00:46:58.380 Patrick Trainer: things.
383 00:46:58.890 ⇒ 00:47:00.637 Patrick Trainer: And like, go
384 00:47:01.380 ⇒ 00:47:02.990 Patrick Trainer: So you can take link.
385 00:47:03.170 ⇒ 00:47:11.280 Patrick Trainer: Things go into it. And that’ll create the document there. Yeah, it’s it’s it’s super nice.
386 00:47:13.930 ⇒ 00:47:14.484 Patrick Trainer: Okay,
387 00:47:16.960 ⇒ 00:47:21.590 Nicolas Sucari: Great and then, if you can, if you can share to the
388 00:47:21.740 ⇒ 00:47:24.790 Nicolas Sucari: document framework that you had, or something like that.
389 00:47:24.790 ⇒ 00:47:25.230 Patrick Trainer: Yeah, yeah.
390 00:47:25.230 ⇒ 00:47:30.260 Nicolas Sucari: Yeah, that you mentioned. That would be great, too. And how we like
391 00:47:30.690 ⇒ 00:47:42.160 Nicolas Sucari: like, what? What’s our next step from this? Utam? What do you think like we should start like documenting like not not documenting this stuff, but having this type in mind for our next, like
392 00:47:42.625 ⇒ 00:47:51.749 Nicolas Sucari: tasks, and try to see how we can start fitting into what we need to do to take our clients into like these kind of steps to see. We we are like.
393 00:47:51.930 ⇒ 00:47:54.549 Nicolas Sucari: Accomplish all of these stuff for them.
394 00:47:54.550 ⇒ 00:47:54.880 Uttam Kumaran: Yeah.
395 00:47:54.880 ⇒ 00:47:55.609 Patrick Trainer: I, I.
396 00:47:55.610 ⇒ 00:47:59.930 Uttam Kumaran: Yeah, so as long as like I, I wanna solidify these basically and like, confirm
397 00:48:00.050 ⇒ 00:48:02.560 Uttam Kumaran: that, like, those 7 are
398 00:48:03.140 ⇒ 00:48:06.500 Uttam Kumaran: like what we need to see across every client, basically.
399 00:48:06.700 ⇒ 00:48:12.300 Patrick Trainer: Right? Yeah, I think there’s like a step in between like now and
400 00:48:12.500 ⇒ 00:48:15.800 Patrick Trainer: creating like implementation tickets, or like.
401 00:48:15.800 ⇒ 00:48:16.900 Uttam Kumaran: I liked.
402 00:48:16.900 ⇒ 00:48:19.880 Patrick Trainer: We need to do. XYZ. It’s like we need to.
403 00:48:20.259 ⇒ 00:48:28.889 Patrick Trainer: Like, if it’s either us 3 or even just kind of like across the company to like, set, and agree, and like to edit
404 00:48:29.200 ⇒ 00:48:34.810 Patrick Trainer: these things. Add, remove, etc.
405 00:48:34.930 ⇒ 00:48:41.430 Patrick Trainer: on these documents, like, we need to kind of like, think critically about this and like how we want to set up this framework
406 00:48:41.819 ⇒ 00:48:50.900 Patrick Trainer: and then and then, like, once, we’re like, Okay, this is good. This is the final draft. Then that’s when we start. Like, okay, we need
407 00:48:50.950 ⇒ 00:48:55.139 Patrick Trainer: this first, st and then we start prioritizing the the the work.
408 00:48:56.880 ⇒ 00:48:59.739 Uttam Kumaran: Tad like, how do you do you see this as like a
409 00:49:00.320 ⇒ 00:49:06.360 Uttam Kumaran: I mean, do you see this as like, okay, let’s, I mean. Let’s let’s take 2. Let’s say 2 different scenarios.
410 00:49:06.410 ⇒ 00:49:17.410 Uttam Kumaran: Let’s take one scenario where it’s like a new client, and then it’ll take a second scenario where we’re like in the middle of like spell. For example, how do you do you see us going through and basically being like
411 00:49:17.780 ⇒ 00:49:19.340 Uttam Kumaran: we need to
412 00:49:19.550 ⇒ 00:49:26.800 Uttam Kumaran: for each of those 7 categories, or whatever ends up being like we 1st are like, we, where do we have everything checked off?
413 00:49:27.080 ⇒ 00:49:31.580 Uttam Kumaran: Cause there’s some stuff that’s like maintenance, or some stuff that’s like processes are in place
414 00:49:31.660 ⇒ 00:49:34.769 Uttam Kumaran: like, how do you see this working for those 2 categories of clients.
415 00:49:35.050 ⇒ 00:49:35.580 Patrick Trainer: So.
416 00:49:35.580 ⇒ 00:49:40.230 Uttam Kumaran: I don’t wanna just talk about new clients because we do have folks that would need to adapt this forward.
417 00:49:40.230 ⇒ 00:49:40.590 Patrick Trainer: Right.
418 00:49:40.590 ⇒ 00:49:41.920 Uttam Kumaran: It, it.
419 00:49:41.920 ⇒ 00:49:50.520 Patrick Trainer: It’s so, there’s there’s a kind of like. And that’s what I like about this is that it’s it’s flexible in the sense that, like we can kind of like
420 00:49:51.170 ⇒ 00:50:00.230 Patrick Trainer: jump into it in the middle of things. So like, say, for for pool parts, like we’d be able to use this to to
421 00:50:01.500 ⇒ 00:50:21.310 Patrick Trainer: use. This is like the jumping off point of like, where, in data quality that we should be looking for like, if they’re paying for snowflake like should we be showing them like how much they’re spending on a month? Or if they’re paying for for 5 tran like, how can they track that as well as like
422 00:50:22.150 ⇒ 00:50:26.589 Patrick Trainer: like more than like is the data correct? But like
423 00:50:26.890 ⇒ 00:50:40.539 Patrick Trainer: are they is, are they getting their money’s worth, and then proving that and like integrating those tools. But then also, it gives us kind of like that checklist. Say, for, like new clients of like, Okay.
424 00:50:41.050 ⇒ 00:50:43.370 Patrick Trainer: we’ve already got these like systems up.
425 00:50:43.900 ⇒ 00:50:49.510 Patrick Trainer: We need to implement this in specific terms to these new clients. And we just like.
426 00:50:49.550 ⇒ 00:50:54.270 Patrick Trainer: kind of like, go down the checklist of like, okay, we need this check, this check.
427 00:50:54.856 ⇒ 00:51:03.609 Patrick Trainer: We need to put logging for them. And like, we need to send it. Or if, like, say, we ever run into like a situation where
428 00:51:04.194 ⇒ 00:51:07.990 Patrick Trainer: somebody’s like, okay, I need to send you
429 00:51:08.180 ⇒ 00:51:08.959 Patrick Trainer: like a
430 00:51:09.410 ⇒ 00:51:15.180 Patrick Trainer: Tsv file like on Friday, every weekend. Yeah, on Friday, like.
431 00:51:15.530 ⇒ 00:51:42.410 Patrick Trainer: we need to be able to think about like, okay, that like, how do we integrate that? What are the things that we need to do and then like, what are the things that need to come after? We implement just that integration. And it’s like, okay, we need to automate all of this. So we stop moving. So we don’t make like human mistakes and basically like, do it right the 1st time. And so we don’t need to think about it. And then we’re able to make that like reproducible
432 00:51:42.700 ⇒ 00:51:54.009 Patrick Trainer: across clients. So it’s like, if one puts in a Tsv, and then somebody’s like, okay, I need to send in some park files. It’s like, okay, we’ve already got kind of like the mechanism to move
433 00:51:54.700 ⇒ 00:51:57.109 Patrick Trainer: a file somewhere.
434 00:51:57.260 ⇒ 00:52:02.560 Patrick Trainer: Let’s just make it specific to Tsv, like, let’s override that
435 00:52:02.650 ⇒ 00:52:08.079 Patrick Trainer: from the Tsv. Into a parquet file, and then we just send it on down the pipe.
436 00:52:11.180 ⇒ 00:52:15.470 Patrick Trainer: does that kind of answer your your question. There, give you a little more perspective.
437 00:52:16.270 ⇒ 00:52:17.863 Uttam Kumaran: Yeah, that makes sense.
438 00:52:19.040 ⇒ 00:52:22.470 Uttam Kumaran: cool. So like, let’s set up a follow update to confirm these.
439 00:52:22.860 ⇒ 00:52:25.929 Uttam Kumaran: I don’t know if we want to do Friday, or maybe Monday.
440 00:52:26.150 ⇒ 00:52:29.950 Uttam Kumaran: I need some time to sit on it and think about it, cause I wanna make sure it has everything.
441 00:52:29.950 ⇒ 00:52:30.730 Patrick Trainer: Yeah, for sure.
442 00:52:30.730 ⇒ 00:52:37.939 Uttam Kumaran: Ideally. What we’ll do is we’ll basically look on the next meeting. We’ll basically try to apply this to stellar across the board.
443 00:52:37.940 ⇒ 00:52:38.350 Patrick Trainer: Yeah.
444 00:52:38.350 ⇒ 00:52:42.009 Uttam Kumaran: And then also begin a to do list of documentation across our.
445 00:52:42.010 ⇒ 00:52:42.710 Patrick Trainer: Yeah.
446 00:52:42.830 ⇒ 00:52:48.260 Patrick Trainer: it’s it’s so like, I was saying, like, I, I encourage like, let’s
447 00:52:48.450 ⇒ 00:52:51.130 Patrick Trainer: take that Pr and basically
448 00:52:51.750 ⇒ 00:52:55.580 Patrick Trainer: have a a pull review of it. And like.
449 00:52:55.700 ⇒ 00:53:00.540 Patrick Trainer: enter your comments of like this like we should think about this here.
450 00:53:01.240 ⇒ 00:53:05.830 Patrick Trainer: we could probably take this out. And it’s basically just like a review
451 00:53:06.222 ⇒ 00:53:24.180 Patrick Trainer: and then we can, I think, like coming back to it like Friday, we’ll be able to go through that review and either accept or reject changes I, and that will tee us up to be like, okay, this is going to be the final draft like, think of it as like the the
452 00:53:24.790 ⇒ 00:53:28.410 Patrick Trainer: document that’s in the in the Pr, that’s like a Google Doc.
453 00:53:28.530 ⇒ 00:53:30.090 Patrick Trainer: And then the
454 00:53:31.062 ⇒ 00:53:38.449 Patrick Trainer: the review process is like adding comments in the sidebar and highlighting shit, yeah.
455 00:53:38.840 ⇒ 00:53:39.480 Patrick Trainer: okay.
456 00:53:39.570 ⇒ 00:53:40.250 Uttam Kumaran: Yeah.
457 00:53:41.240 ⇒ 00:53:59.770 Nicolas Sucari: Great, and then we can like start to try to fit into full parts. And, Stella, what we are, what we, what comes out from that document and see what we need to do to ramp up if we are missing something right, and that will give us, like our like starting point documentation for future clients. Probably.
458 00:54:00.310 ⇒ 00:54:00.930 Patrick Trainer: Right.
459 00:54:00.930 ⇒ 00:54:01.500 Nicolas Sucari: Yeah.
460 00:54:02.470 ⇒ 00:54:04.800 Nicolas Sucari: yeah, this is nice. It’s nice. Yeah.
461 00:54:04.800 ⇒ 00:54:08.450 Patrick Trainer: Hope I’ve given you all a couple of things to think about.
462 00:54:09.310 ⇒ 00:54:10.210 Nicolas Sucari: Everyone. Thanks.
463 00:54:10.210 ⇒ 00:54:11.840 Uttam Kumaran: Back on the line.
464 00:54:13.300 ⇒ 00:54:14.190 Uttam Kumaran: Sorry.
465 00:54:14.840 ⇒ 00:54:18.730 Uttam Kumaran: When do we want to get back on the line on these like next week?
466 00:54:19.060 ⇒ 00:54:34.239 Patrick Trainer: Yeah, I mean, yeah, we can. We can take a week like, take a couple of days. It’s Tuesday. So Wednesday, Thursday, Friday, 3 days and like, look it over or like, take yeah. Just take some time to chew on it and then.
467 00:54:35.080 ⇒ 00:54:35.639 Uttam Kumaran: You know.
468 00:54:35.640 ⇒ 00:54:37.320 Nicolas Sucari: It’s a mid Monday, Tuesday. Yeah.
469 00:54:37.320 ⇒ 00:54:41.119 Patrick Trainer: Yeah, yeah, make either Monday or Tuesday, the the
470 00:54:41.970 ⇒ 00:54:55.739 Patrick Trainer: the either like final review, or where we review in a group talk it over, and then we can approve, disprove changes and do that. I also strongly suggest, like
471 00:54:56.390 ⇒ 00:55:01.049 Patrick Trainer: allowing a lot of the discussion to happen on the Pr itself.
472 00:55:01.130 ⇒ 00:55:03.300 Patrick Trainer: because, like, we don’t want
473 00:55:03.580 ⇒ 00:55:08.789 Patrick Trainer: these discussions of like, like, we also want to track our like, our decisions of.
474 00:55:08.790 ⇒ 00:55:09.180 Nicolas Sucari: Yeah.
475 00:55:09.180 ⇒ 00:55:11.760 Patrick Trainer: Why we did things, and like
476 00:55:11.790 ⇒ 00:55:27.550 Patrick Trainer: talking in slack is great, but like we’ll probably lose it, and then would have to like search and go back and do all that. So like keeping that discussion. Or I like discussion for docs to be on that. Pr, so we can understand the context of
477 00:55:27.650 ⇒ 00:55:29.769 Patrick Trainer: kind of what we were thinking at that time.
478 00:55:30.530 ⇒ 00:55:31.370 Patrick Trainer: Yeah, excellent.
479 00:55:31.590 ⇒ 00:55:32.220 Nicolas Sucari: Great.
480 00:55:33.230 ⇒ 00:55:35.763 Uttam Kumaran: Okay, great. Alright. Let’s follow up next week.
481 00:55:36.080 ⇒ 00:55:39.600 Patrick Trainer: Cool dang, and we ended right at 2 30. Look at that.
482 00:55:40.650 ⇒ 00:55:41.220 Uttam Kumaran: Nice.
483 00:55:41.220 ⇒ 00:55:42.850 Patrick Trainer: That is nice, nice.
484 00:55:44.095 ⇒ 00:55:44.430 Uttam Kumaran: Steve.
485 00:55:44.560 ⇒ 00:55:46.960 Nicolas Sucari: Thank you. Thank you, Patrick. Alright, perfect!
486 00:55:46.960 ⇒ 00:55:49.200 Uttam Kumaran: Thanks, guys, thanks, Patrick. Appreciate it.
487 00:55:49.520 ⇒ 00:55:50.280 Nicolas Sucari: Hey, mate.