Meeting Title: Brainforge Data Engineering Process Review Date: 2026-02-20 Meeting participants: Awaish Kumar, Ashwini Sharma
WEBVTT
1 00:00:00.000 ⇒ 00:00:02.190 Awaish Kumar: Let’s talk to you.
2 00:01:33.040 ⇒ 00:01:34.650 Ashwini Sharma: Abish, are you able to hear me?
3 00:02:31.880 ⇒ 00:02:33.570 Awaish Kumar: Hi, Kashrini.
4 00:02:33.840 ⇒ 00:02:34.780 Ashwini Sharma: Hey, Amish.
5 00:02:35.310 ⇒ 00:02:36.179 Awaish Kumar: How you doing?
6 00:02:36.990 ⇒ 00:02:39.649 Ashwini Sharma: I’m good, man. Yeah.
7 00:02:40.010 ⇒ 00:02:41.020 Awaish Kumar: Good work. Okay.
8 00:02:41.020 ⇒ 00:02:41.740 Ashwini Sharma: How are you?
9 00:02:42.250 ⇒ 00:02:43.470 Awaish Kumar: I’m good as well.
10 00:02:43.710 ⇒ 00:02:48.480 Awaish Kumar: So, actually, we, in the recent weeks, we didn’t have time to,
11 00:02:48.650 ⇒ 00:02:53.710 Awaish Kumar: talk, regarding, like, what we discussed last time with Utam.
12 00:02:54.530 ⇒ 00:02:56.430 Ashwini Sharma: Oh, okay, yeah, yeah, yeah, yeah, I don’t…
13 00:02:56.430 ⇒ 00:03:12.390 Awaish Kumar: We had to conduct some sessions, and I just wasn’t able to get some time. I asked him to push for a week last time, but now that we are end of that week as well, maybe today we can
14 00:03:12.610 ⇒ 00:03:17.159 Awaish Kumar: talked through? How… Things are going.
15 00:03:17.520 ⇒ 00:03:21.449 Awaish Kumar: Right? We can come up with a plan, like,
16 00:03:21.890 ⇒ 00:03:29.159 Awaish Kumar: Now, again, one of my past experiences when you are doing something like that, We just need a…
17 00:03:29.520 ⇒ 00:03:35.059 Awaish Kumar: Kind of a discussion, or we can, like, talk about approaches.
18 00:03:35.310 ⇒ 00:03:39.330 Ashwini Sharma: And in the next session, we can maybe go through some…
19 00:03:39.350 ⇒ 00:03:44.520 Awaish Kumar: actual tasks, right? And as a kind of a demo or… Things like that, like…
20 00:03:44.520 ⇒ 00:03:45.489 Ashwini Sharma: We will be…
21 00:03:45.720 ⇒ 00:03:48.069 Awaish Kumar: Together, we’ll be solving one thing.
22 00:03:48.840 ⇒ 00:03:49.470 Ashwini Sharma: Okay.
23 00:03:50.130 ⇒ 00:04:00.630 Awaish Kumar: And, like, we’ll be working on a single ticket for an hour, and we will see how we, can approach this. But for today, I think we can just,
24 00:04:01.490 ⇒ 00:04:14.549 Awaish Kumar: kind of a… make it, like, more of, understand, like, more of a kind of a, what’d you say, kind of interview session, you can say. Like, I just… I just would like to… I would try to understand
25 00:04:14.770 ⇒ 00:04:17.889 Awaish Kumar: How you are working, your working style.
26 00:04:18.000 ⇒ 00:04:36.409 Awaish Kumar: on how you approach things, and is there anything where we can actually… can help, like, to speed things up, or to optimize? And in that session, we are… we will figure out what we should do next.
27 00:04:37.110 ⇒ 00:04:39.020 Awaish Kumar: You know, in our next session.
28 00:04:40.900 ⇒ 00:04:43.030 Ashwini Sharma: Yeah, okay, sure, yeah.
29 00:04:44.060 ⇒ 00:04:53.059 Awaish Kumar: Yeah, so I… I have some questions for you, like, so, like, right now, like, I know you’re, like, do you…
30 00:04:54.060 ⇒ 00:04:54.990 Awaish Kumar: Wow.
31 00:04:55.130 ⇒ 00:05:04.720 Awaish Kumar: like, as an… like, you have been working as a data engineer, so I just would like to understand how you, as an engineer, or…
32 00:05:05.690 ⇒ 00:05:12.790 Awaish Kumar: architect. How do you approach, like, if you are given a task, which is,
33 00:05:13.840 ⇒ 00:05:26.509 Awaish Kumar: which lacks the requirements, right? Which lacks the, concrete, requirements. It’s uncertain for you, like, you just got some ticket, there’s nothing. How would you then approach it?
34 00:05:27.440 ⇒ 00:05:31.420 Ashwini Sharma: So basically, like, you know, when it’s not clear what needs to be done.
35 00:05:31.560 ⇒ 00:05:40.820 Ashwini Sharma: I first ask, right, whether it is Utham, or if it is, Utham also does not know about it, then I’ll reach out to the client, right?
36 00:05:41.010 ⇒ 00:05:42.159 Ashwini Sharma: In the channel.
37 00:05:43.000 ⇒ 00:05:46.900 Ashwini Sharma: To understand what exactly they mean by that requirement.
38 00:05:47.230 ⇒ 00:05:50.889 Ashwini Sharma: And once they explain, I proceed with my work.
39 00:05:51.100 ⇒ 00:06:00.330 Ashwini Sharma: But before I reach out to the client, I would have explored, like, does it mean this thing, or does it mean that thing? At least try to, you know.
40 00:06:00.580 ⇒ 00:06:03.009 Ashwini Sharma: Understand what client could have meant.
41 00:06:03.340 ⇒ 00:06:16.409 Ashwini Sharma: You know, based on, like, the current understanding that we have about the client and its functionality. But other than that, like, if nothing is coming up, then I just wait until client can respond to that.
42 00:06:18.270 ⇒ 00:06:26.090 Awaish Kumar: Yeah, so… Like, if we take an example of Spin’s API pipeline, So, like…
43 00:06:26.250 ⇒ 00:06:33.370 Awaish Kumar: the concern was, like, it’s not like you can’t write Python code, or it’s not like,
44 00:06:33.670 ⇒ 00:06:41.100 Awaish Kumar: Like, everybody at Brainforge can write code, can write pipelines. It was more like an approach of
45 00:06:41.330 ⇒ 00:06:46.120 Awaish Kumar: Figuring the requirements out early in the process.
46 00:06:47.010 ⇒ 00:06:53.689 Awaish Kumar: Such that… We can actually…
47 00:06:55.970 ⇒ 00:06:59.699 Awaish Kumar: Come up with a plan before we start execution.
48 00:07:01.390 ⇒ 00:07:02.709 Awaish Kumar: I… so…
49 00:07:03.680 ⇒ 00:07:09.439 Awaish Kumar: if you are, like… so, how would you… how would I normally will approach if I have some…
50 00:07:09.610 ⇒ 00:07:13.259 Awaish Kumar: Next to, like, Expin’ API pipeline, where I’m working, if I…
51 00:07:13.660 ⇒ 00:07:22.740 Awaish Kumar: So, I would just go first with, like, gathering the requirements?
52 00:07:23.100 ⇒ 00:07:29.100 Awaish Kumar: As much as possible. So, like, number one is how is Prince API working?
53 00:07:29.300 ⇒ 00:07:38.740 Awaish Kumar: approach all the possible solutions. Like, in the Springs API, we figure out there is an API endpoint which can give you some data, and then there is an export endpoint as well.
54 00:07:39.160 ⇒ 00:07:43.139 Awaish Kumar: Right? But we then… we’ll figure that out later in the process.
55 00:07:44.670 ⇒ 00:07:45.070 Ashwini Sharma: Yeah, yeah.
56 00:07:45.070 ⇒ 00:07:49.059 Awaish Kumar: I’ve already spent a lot of time on it. Yeah. And that’s…
57 00:07:49.580 ⇒ 00:07:55.630 Awaish Kumar: Where, like, there are two things. Number one is requirements gathering, number one is
58 00:07:55.850 ⇒ 00:08:01.990 Awaish Kumar: Thinking about all the alternative approaches, and coming up with a plan before execution.
59 00:08:04.340 ⇒ 00:08:05.380 Awaish Kumar: So…
60 00:08:06.070 ⇒ 00:08:20.559 Awaish Kumar: how would I know, like, that’s… I or anybody else, like, what should be the approaches? Like, you… we have to maybe, like, we decided to come up with a process where we build technical specification.
61 00:08:20.750 ⇒ 00:08:27.900 Awaish Kumar: from… Pick, like, documents before we start any execution, so…
62 00:08:28.500 ⇒ 00:08:34.070 Awaish Kumar: It’s like… now that, when we have that process, what we can actually do is.
63 00:08:34.419 ⇒ 00:08:48.809 Awaish Kumar: We explore all the alternatives, that’s number one. We come up with best possible solution of… in terms of implementation. Then, on the other hand, we have requirements from the
64 00:08:49.100 ⇒ 00:08:58.280 Awaish Kumar: the client. So, number one requirement is, okay, I need to get this data on a weekly level, and then we need to aggregate it.
65 00:08:58.410 ⇒ 00:09:01.500 Awaish Kumar: So, I don’t… I’m not sure when they…
66 00:09:02.150 ⇒ 00:09:06.760 Awaish Kumar: When they made that request, like, for example, if I know
67 00:09:06.940 ⇒ 00:09:14.649 Awaish Kumar: If I know that requirement early in the process, like, I get ACV values on a weekly level.
68 00:09:15.130 ⇒ 00:09:27.670 Awaish Kumar: then I’m thinking about using that data to come up with weekly, aggregates. So, like, my first question would be then, like, okay, is it additive aggregate, right? Is SCV value
69 00:09:27.770 ⇒ 00:09:32.940 Awaish Kumar: Can I… if I get a SCV value or TCV value at an,
70 00:09:33.520 ⇒ 00:09:38.910 Awaish Kumar: weekly level, is it possible for me to actually create 4 weeks and get out of it?
71 00:09:39.230 ⇒ 00:09:44.139 Awaish Kumar: If not, then, like, that’s… That’s where we have to stop, and…
72 00:09:44.650 ⇒ 00:09:49.740 Awaish Kumar: And ask again, like, okay, we are not able to get this metric.
73 00:09:51.100 ⇒ 00:09:53.589 Awaish Kumar: If we get… just get weekly aggregates. So…
74 00:09:54.410 ⇒ 00:10:02.400 Awaish Kumar: what should be the next plan? Like… like, these should be… these questions should be… have been asked, like, very early in the…
75 00:10:02.880 ⇒ 00:10:05.910 Awaish Kumar: In the process.
76 00:10:06.810 ⇒ 00:10:13.409 Ashwini Sharma: Yeah, I think this question was asked, right? It is not that it was… it was discussed, and they had said that, you know.
77 00:10:13.520 ⇒ 00:10:20.500 Ashwini Sharma: Are you able to match this GDP and ACV values, right? And when we tried to match it, it was not matching.
78 00:10:20.610 ⇒ 00:10:24.380 Ashwini Sharma: Right? The dollars were obviously aggregating, right? And that’s when it
79 00:10:24.870 ⇒ 00:10:27.689 Ashwini Sharma: out that it was not, additive.
80 00:10:28.160 ⇒ 00:10:31.589 Ashwini Sharma: Right? It was not average, it was not narrative, yeah.
81 00:10:32.210 ⇒ 00:10:50.690 Awaish Kumar: when… yeah, I’m not saying it was not asked, but when I’m saying it is… it was asked, then at that time, I… what I remember is we already went through a lot of, different iterations, of you working on a Swins API, validation, building and setup.
82 00:10:50.830 ⇒ 00:10:57.090 Awaish Kumar: And, testing out, and then we sent out some data, and then again,
83 00:10:57.380 ⇒ 00:11:01.529 Awaish Kumar: So, like, we started the execution already, right?
84 00:11:02.080 ⇒ 00:11:08.280 Awaish Kumar: So, like, it’s not before execution. We already started with execution, we didn’t confirm
85 00:11:09.040 ⇒ 00:11:12.099 Awaish Kumar: These metrics are additive or not.
86 00:11:12.390 ⇒ 00:11:14.600 Awaish Kumar: And things like that. My…
87 00:11:16.190 ⇒ 00:11:22.950 Awaish Kumar: So, from, like, the learning from that is just that, like, we… Explore?
88 00:11:23.570 ⇒ 00:11:29.240 Awaish Kumar: What… the requirements, and then just ask the relevant questions.
89 00:11:29.370 ⇒ 00:11:32.190 Awaish Kumar: Come up with all set of questions and just…
90 00:11:32.370 ⇒ 00:11:38.999 Awaish Kumar: get their answers before execution. On the other end side, on technical side, we just need to explore different
91 00:11:39.180 ⇒ 00:11:47.630 Awaish Kumar: Alternatives, and the export solution maybe works much better than pipeline API solution, so it’s…
92 00:11:50.050 ⇒ 00:11:57.300 Awaish Kumar: Yeah, that’s why, whole summary of that, like, it’s not about, like,
93 00:11:57.540 ⇒ 00:12:02.469 Awaish Kumar: what you did, it’s more… I’m more… I’m just taking an example to sh… like, to just…
94 00:12:02.630 ⇒ 00:12:08.050 Awaish Kumar: explain, if I… if we come up with similar… Task again.
95 00:12:08.650 ⇒ 00:12:11.510 Awaish Kumar: How would we normally… should approach it?
96 00:12:15.560 ⇒ 00:12:16.300 Ashwini Sharma: Okay.
97 00:12:19.540 ⇒ 00:12:22.020 Awaish Kumar: So, yeah, that’s one of the things.
98 00:12:22.260 ⇒ 00:12:28.590 Awaish Kumar: Then, yeah, that was more, like, me talking. Now I just want to hear from you, like.
99 00:12:28.720 ⇒ 00:12:33.850 Awaish Kumar: What are your… What do you, like, enjoy doing the most?
100 00:12:35.710 ⇒ 00:12:41.990 Ashwini Sharma: what did I enjoy doing the most? I think, see, this Spins API was a different kind of beast, right? So…
101 00:12:42.020 ⇒ 00:12:56.660 Ashwini Sharma: Initially, so they have this Prefect code, they have a framework in that Prefect, pipeline framework where all the components fit together, right? So there is a data extraction component that extracts the data by hitting the APIs, and then there is a…
102 00:12:56.660 ⇒ 00:13:11.539 Ashwini Sharma: data transformation component that can do some initial transformation within the pipeline, and then there is a data load component, which, what it does is, like, it will read the ingested data as a data frame, and then it will write it into Redshift. That is how it works.
103 00:13:12.010 ⇒ 00:13:20.339 Ashwini Sharma: And initially, like, I put my entire code within that framework only, right? Let’s make it work through this framework.
104 00:13:20.630 ⇒ 00:13:28.469 Ashwini Sharma: And that’s where it consumed a lot of time, because the pipeline used to fail a lot of times, right, because of the data volume.
105 00:13:28.830 ⇒ 00:13:31.910 Ashwini Sharma: And, finally, like, I…
106 00:13:31.910 ⇒ 00:13:39.770 Awaish Kumar: That’s okay, that’s okay. I think we are… we don’t have to go through all that. I know all that, like, I’ve been with you all the…
107 00:13:40.410 ⇒ 00:13:45.800 Awaish Kumar: With all that development, right? I know how we,
108 00:13:46.070 ⇒ 00:13:52.280 Awaish Kumar: approach that, like, I understand that, like, we… that’s what I’m saying.
109 00:13:52.700 ⇒ 00:13:59.019 Awaish Kumar: We… the data value was big, it was failing, but we had the export API, right?
110 00:13:59.440 ⇒ 00:14:03.240 Awaish Kumar: But we didn’t explore that as an alternative solution.
111 00:14:04.430 ⇒ 00:14:13.229 Ashwini Sharma: Yeah, one of the reasons why we didn’t explore, that thing was, we didn’t get this, AWS, admin
112 00:14:13.370 ⇒ 00:14:15.510 Ashwini Sharma: For quite some time, right?
113 00:14:15.640 ⇒ 00:14:23.079 Ashwini Sharma: And, initially, like, I was just… you remember, right? The dbt administration, the…
114 00:14:23.290 ⇒ 00:14:32.140 Ashwini Sharma: what do you call it? Prefect and AWS. All of these things were under the control of the other vendor who was doing the development.
115 00:14:32.900 ⇒ 00:14:40.740 Awaish Kumar: I, I… I completely get it. I’m just saying that, like, It is not that…
116 00:14:41.130 ⇒ 00:14:58.460 Awaish Kumar: you were blocked. It’s just that we didn’t surface it to the, for example, to the UTAM or to the team, that, okay, we have these two solutions, we can’t do S3XTX right now, that is the better solution, but I can’t do it right now because I’m blocked. I’m taking.
117 00:14:58.460 ⇒ 00:14:58.900 Ashwini Sharma: Yeah, instantly.
118 00:14:58.900 ⇒ 00:14:59.729 Awaish Kumar: I’ve been bought.
119 00:15:00.170 ⇒ 00:15:09.129 Ashwini Sharma: Right, instead of… no, so basically, like, that portion of the solution was not explored, because I was just trying to load it via S3, like… like the other pipelines, right?
120 00:15:09.360 ⇒ 00:15:18.279 Ashwini Sharma: It’s only… and it loaded also, right? It’s not… initially, the pipeline failed for quite some time, but later it started loading, and we spent a lot of time in QA.
121 00:15:18.340 ⇒ 00:15:33.180 Ashwini Sharma: Right? Now, when we were spending in QA, we realized that a lot of mismatches were there, and that’s when we reached out to Spins, right? And then they suggested that don’t do it this way, do it through export. Yeah, yeah. So, that’s the whole point, right?
122 00:15:33.180 ⇒ 00:15:44.869 Awaish Kumar: That’s what exactly I’m saying, like, if we have explored and made it clear that I know the solution, but I can’t do it because I’m blocked by some access from client.
123 00:15:44.970 ⇒ 00:15:49.490 Awaish Kumar: Then it wouldn’t be you as someone who didn’t…
124 00:15:49.860 ⇒ 00:16:01.270 Awaish Kumar: made it clear, but now that everybody in the document can read it, okay, Ashwini has said that there’s a better solution, but we were blocked, that… so it will be understandable for everybody.
125 00:16:01.450 ⇒ 00:16:08.379 Ashwini Sharma: No, okay, so for that thing… to suggest that thing, I should have known that my solution is bad, right?
126 00:16:08.640 ⇒ 00:16:12.270 Ashwini Sharma: the solution that I am following, you know, to do the extraction.
127 00:16:12.400 ⇒ 00:16:23.460 Awaish Kumar: Normally, I think, now that we know, as data engineer, that, like, API… if we have… if you have access to an API, right? Like.
128 00:16:23.600 ⇒ 00:16:31.009 Awaish Kumar: If I know I have a data source which gives me data through an API, where I have to hit… I have to manage a…
129 00:16:31.080 ⇒ 00:16:43.840 Awaish Kumar: pipeline, which hits the API, there’s a rate limit, there are failures, I need to retry, maybe, the network issues, but then there is… the same source has data… provides me data in S3 directly.
130 00:16:43.840 ⇒ 00:16:44.210 Ashwini Sharma: No, no.
131 00:16:44.210 ⇒ 00:16:49.189 Awaish Kumar: Data engineer, where would you… what would you, like, what would you pick as a solution?
132 00:16:49.190 ⇒ 00:16:53.540 Ashwini Sharma: That is not how it works, right? Probably, you got it wrong. See?
133 00:16:53.740 ⇒ 00:16:55.619 Ashwini Sharma: I’m still hitting the API.
134 00:16:55.720 ⇒ 00:17:04.579 Awaish Kumar: I understand, like, you’re hitting the API, but now it’s a little bit different, like, you are not actually getting the records in
135 00:17:05.839 ⇒ 00:17:10.910 Awaish Kumar: Like, single… record as an API response.
136 00:17:11.579 ⇒ 00:17:28.139 Ashwini Sharma: No, earlier also, I was not getting that. See, the difference between the current approach and the previous approach is when I hit the API, right, it used to generate a CSV file in Google Cloud Service. Okay. Okay? And we have Redshift in AWS.
137 00:17:28.359 ⇒ 00:17:41.799 Ashwini Sharma: Okay. Okay. Now, what I used to do is, I used to read that CSV file that is generated, and then I used to write whatever I read from that CSV file into S3 through the framework.
138 00:17:42.329 ⇒ 00:17:54.809 Ashwini Sharma: Okay, it’s important to this particular last statement. Through the framework, there is a framework that this other vendor has created, where all the pipelines work, they follow a certain structure, right? There is an extract component, load.
139 00:17:54.810 ⇒ 00:17:59.150 Awaish Kumar: Yeah, I’m not talking about Prefect right now. Let’s talk about just the API endpoint.
140 00:17:59.150 ⇒ 00:18:11.959 Ashwini Sharma: Yeah, right, right, okay. Yeah, yeah, yeah. So, so listen, right? That is what I’m saying. Okay, I used to read that CSV file, which is generated in GCS, and then used to write it as a CSV file in.
141 00:18:11.960 ⇒ 00:18:13.180 Awaish Kumar: Okay. S3.
142 00:18:13.180 ⇒ 00:18:15.619 Ashwini Sharma: Okay? Through the pipeline.
143 00:18:15.770 ⇒ 00:18:29.760 Ashwini Sharma: Right? Now the approach is, again, the same GCS file is generated in… sorry, CSV file is generated in GCS. Now what I do is, I run another API in addition to that, which will move that CSV file into
144 00:18:30.240 ⇒ 00:18:31.180 Ashwini Sharma: S3.
145 00:18:31.340 ⇒ 00:18:33.059 Ashwini Sharma: That is the only thing that is different.
146 00:18:33.280 ⇒ 00:18:41.029 Awaish Kumar: So, you are saying that the file that was being generated in your GCP, right?
147 00:18:41.090 ⇒ 00:18:42.470 Ashwini Sharma: Yeah, GCS, yeah.
148 00:18:42.470 ⇒ 00:18:45.709 Awaish Kumar: file, GCS, that was correct, both.
149 00:18:45.980 ⇒ 00:18:47.789 Ashwini Sharma: That was correct, yes.
150 00:18:47.790 ⇒ 00:18:54.249 Awaish Kumar: So, just moving from GCS to S3 missed the data.
151 00:18:55.040 ⇒ 00:18:59.050 Ashwini Sharma: No, yeah, yeah, so when I was doing the move from GCS to S3.
152 00:18:59.050 ⇒ 00:19:02.099 Awaish Kumar: Yeah, yeah, that’s… Yeah, data was being missed.
153 00:19:02.100 ⇒ 00:19:02.670 Ashwini Sharma: Yeah.
154 00:19:05.230 ⇒ 00:19:06.820 Awaish Kumar: Actually, I’m…
155 00:19:07.190 ⇒ 00:19:17.739 Awaish Kumar: I’m not sure how were you moving that, but why would… if you are just moving from Google Cloud Storage to S3, in whatever way, I don’t know, but…
156 00:19:18.000 ⇒ 00:19:20.749 Awaish Kumar: Why do rows drop doing that?
157 00:19:21.400 ⇒ 00:19:23.889 Ashwini Sharma: I don’t have an answer to that, right?
158 00:19:24.360 ⇒ 00:19:26.860 Awaish Kumar: But that is what was happening.
159 00:19:27.420 ⇒ 00:19:36.169 Awaish Kumar: But, like, with my experience, I suspect that the file that was being generated, that was not quite right. That was miserable.
160 00:19:36.310 ⇒ 00:19:38.189 Ashwini Sharma: No, no, that is not true.
161 00:19:39.540 ⇒ 00:19:42.819 Awaish Kumar: Okay, then I wouldn’t… actually, I would need an answer, like…
162 00:19:42.820 ⇒ 00:19:50.369 Ashwini Sharma: Yeah, I don’t know why it was missing, but it was missing, and that… when it was missing, that’s when they said, don’t do it this way, move it to…
163 00:19:50.370 ⇒ 00:19:50.890 Awaish Kumar: Yeah, buddy.
164 00:19:50.890 ⇒ 00:19:53.330 Ashwini Sharma: directly, and then load it from S3, right?
165 00:19:53.580 ⇒ 00:20:00.940 Awaish Kumar: I understand. I completely, like, I completely understand your statement. I’m just trying to now understand
166 00:20:01.690 ⇒ 00:20:06.019 Awaish Kumar: You can, like, you have quite a lot of experience as well, like, why would.
167 00:20:06.020 ⇒ 00:20:15.620 Ashwini Sharma: Yeah, yeah, even I would like to know that, why it is missing, but I don’t have an answer, and honestly, I don’t have the time to investigate why it is missing also, right? Because the way we are working, we need to move fast.
168 00:20:15.620 ⇒ 00:20:18.890 Awaish Kumar: I think it’s really hard for anybody to…
169 00:20:19.100 ⇒ 00:20:34.039 Awaish Kumar: to actually get that thing, like, moving from one cloud storage to another cloud storage will drop… like, we can drop a file, that is understandable. But okay, while moving that, I was… I got network issue and file didn’t move.
170 00:20:34.220 ⇒ 00:20:34.800 Awaish Kumar: That is.
171 00:20:34.800 ⇒ 00:20:35.600 Ashwini Sharma: Yeah, no.
172 00:20:35.740 ⇒ 00:20:36.870 Awaish Kumar: But, the roads from…
173 00:20:36.870 ⇒ 00:20:37.280 Ashwini Sharma: I’m a fight.
174 00:20:37.280 ⇒ 00:20:39.390 Awaish Kumar: How can they miss?
175 00:20:39.390 ⇒ 00:20:46.750 Ashwini Sharma: That is what happened. See, it’s not one file, right? It generates, like, around 80-90 files.
176 00:20:47.110 ⇒ 00:20:51.270 Ashwini Sharma: Okay, sometimes when the extract is big, it will generate a lot of files, right?
177 00:20:51.270 ⇒ 00:21:01.149 Awaish Kumar: Then we just have to make… we… our pipeline should be, mature to just… to figure… to find out that if the file is not missing.
178 00:21:02.110 ⇒ 00:21:06.889 Ashwini Sharma: everything was there in place. There was not a single… see, I have written pipelines externally.
179 00:21:06.890 ⇒ 00:21:25.010 Awaish Kumar: Like, you’re telling me now that there were multiple files, that’s what I’m saying. If you are saying all the files were being moved, then rows shouldn’t drop. If a file is missing, if file did not move, then we can miss the data. I understand that. Out of 60, maybe 59 moved.
180 00:21:25.010 ⇒ 00:21:34.379 Ashwini Sharma: I’ll explain you how the files were being moved, right? Maybe then you can understand, like, what might be happening. So it generates the file in GCS, and
181 00:21:34.380 ⇒ 00:21:49.059 Ashwini Sharma: what the pipeline does is, like, it will try to read that file, right? So it’s reading into a data frame, everything, right? And I’m running the pipeline locally, okay? And now, once it reads into a data frame, it will try to write that data frame into S3.
182 00:21:49.690 ⇒ 00:21:51.130 Ashwini Sharma: Using the data framing.
183 00:21:51.380 ⇒ 00:21:55.889 Ashwini Sharma: That’s where something was happening, because of which the rows were missing when it went over.
184 00:21:55.890 ⇒ 00:22:15.660 Awaish Kumar: Okay, that is… now that is really clear, that when you are reading to a data frame, then it’s not a file, like, file-to-file task for, as you mentioned. Now it is the same. What I was saying, it’s now… the approach you are taking right now is you’re kind of reading a file and getting all the rows out of it in a data frame.
185 00:22:16.100 ⇒ 00:22:22.390 Awaish Kumar: And then we are trying to store it again to some S3. And while doing that, it’s possible that
186 00:22:22.650 ⇒ 00:22:24.909 Awaish Kumar: One of, like, your data frame.
187 00:22:25.050 ⇒ 00:22:32.860 Awaish Kumar: got crashed somewhere. Like, you got… we got some silent exception that just failed the rules, or whatever.
188 00:22:32.990 ⇒ 00:22:33.840 Awaish Kumar: Right?
189 00:22:34.010 ⇒ 00:22:40.770 Awaish Kumar: So that’s… So that could have happened, yeah. The point is, like, yeah.
190 00:22:40.980 ⇒ 00:22:46.460 Ashwini Sharma: If I don’t see an error happening, right, during the entire moment, I will not know that an error has happened, right?
191 00:22:46.460 ⇒ 00:22:47.199 Awaish Kumar: Yeah, but…
192 00:22:47.200 ⇒ 00:22:49.849 Ashwini Sharma: I see that everything is working fine.
193 00:22:50.200 ⇒ 00:22:52.230 Awaish Kumar: Yeah, like, now let me just…
194 00:22:52.470 ⇒ 00:22:59.090 Awaish Kumar: Let’s pause here and do… get… go by… go through the solution step by step.
195 00:22:59.500 ⇒ 00:23:04.839 Awaish Kumar: So, first step, you hit the API, you generate a GCS file, okay?
196 00:23:05.030 ⇒ 00:23:05.590 Ashwini Sharma: Yep.
197 00:23:05.750 ⇒ 00:23:14.580 Awaish Kumar: And you are saying that GCS file, according to your investigation, or whatever validation you did, that has all the rows, and it is not missing any rows.
198 00:23:14.830 ⇒ 00:23:16.640 Awaish Kumar: Yes. Correct? That is correct, yeah.
199 00:23:17.080 ⇒ 00:23:22.210 Awaish Kumar: So now that I have a file from SPIN’s API in some GCS.
200 00:23:22.320 ⇒ 00:23:25.179 Ashwini Sharma: Gcs is client storage, right?
201 00:23:25.990 ⇒ 00:23:28.190 Ashwini Sharma: No, it’s, it’s, spin stories.
202 00:23:28.650 ⇒ 00:23:30.349 Awaish Kumar: Because it spins out, so it’s okay.
203 00:23:30.540 ⇒ 00:23:39.260 Awaish Kumar: So, we are saying that files got generated on Spin’s API’s cloud storage, and you validated that file, it has all the rows.
204 00:23:39.570 ⇒ 00:23:40.400 Awaish Kumar: And then…
205 00:23:40.400 ⇒ 00:23:40.900 Ashwini Sharma: Yeah.
206 00:23:41.030 ⇒ 00:23:45.879 Awaish Kumar: like, Spin’s API actually created a file which has all the rows for us.
207 00:23:46.590 ⇒ 00:23:48.790 Ashwini Sharma: So it will not create one file, it will create multiple.
208 00:23:48.790 ⇒ 00:23:51.719 Awaish Kumar: Yeah. Like, okay, file or files.
209 00:23:51.720 ⇒ 00:23:52.160 Ashwini Sharma: Yeah.
210 00:23:52.160 ⇒ 00:24:08.909 Awaish Kumar: it created, which has all the rows. Now, I have a pipeline. Now, my pipeline is not failing on a, like, on… at the point where Sprint’s API is sending incorrect data. My API… now my pipeline is failing while moving data
211 00:24:09.020 ⇒ 00:24:11.520 Awaish Kumar: from Google Cloud Storage to S3.
212 00:24:11.850 ⇒ 00:24:13.150 Ashwini Sharma: It’s not failing.
213 00:24:13.500 ⇒ 00:24:16.460 Awaish Kumar: Attaining means it’s missing rows.
214 00:24:16.600 ⇒ 00:24:22.340 Ashwini Sharma: Yeah, and yeah, that… but when it is running perfectly, you don’t know it is missing rows, right?
215 00:24:24.330 ⇒ 00:24:27.540 Awaish Kumar: When it is running perfectly.
216 00:24:27.540 ⇒ 00:24:30.449 Ashwini Sharma: When it is not throwing any exception, then you don’t know that it’s.
217 00:24:30.450 ⇒ 00:24:35.559 Awaish Kumar: Is that a… Yeah, let’s… let’s go through it step by step.
218 00:24:35.670 ⇒ 00:24:37.859 Ashwini Sharma: Okay. Now that we have…
219 00:24:38.360 ⇒ 00:24:42.600 Awaish Kumar: So, we are… I’m not here to…
220 00:24:43.440 ⇒ 00:24:50.400 Awaish Kumar: judge you how you did. I’m here to help you understand What could have been done?
221 00:24:50.590 ⇒ 00:24:53.739 Awaish Kumar: So that you know if a rose may miss.
222 00:24:54.320 ⇒ 00:24:59.180 Awaish Kumar: Or it triggers some alert, if there is any failure.
223 00:25:02.850 ⇒ 00:25:08.580 Awaish Kumar: like, we saw the similar issue with Eden Catalyst API as well, right?
224 00:25:09.650 ⇒ 00:25:13.889 Awaish Kumar: the files, the data that we pushed to the catalyst.
225 00:25:14.170 ⇒ 00:25:25.160 Awaish Kumar: the pings were right. We were pinging the same order again and again, because our data frame was not writing through BigQuery, and we didn’t got alert.
226 00:25:25.330 ⇒ 00:25:26.260 Awaish Kumar: Right?
227 00:25:27.210 ⇒ 00:25:27.880 Ashwini Sharma: Yep.
228 00:25:28.280 ⇒ 00:25:29.630 Awaish Kumar: So…
229 00:25:30.920 ⇒ 00:25:38.499 Awaish Kumar: It’s the same, like, it could be the same thing right here. Like, we missed some except… some kind of…
230 00:25:40.150 ⇒ 00:25:42.409 Awaish Kumar: like, we… maybe I wrote some…
231 00:25:42.520 ⇒ 00:25:54.550 Awaish Kumar: Maybe there’s some try-accept, I don’t know, I haven’t read your code, but maybe there was some failure, but it was silently happening, so that you didn’t actually know, yeah, it didn’t trigger it.
232 00:25:54.960 ⇒ 00:25:59.959 Awaish Kumar: similar to Eden, where we are pinging the catalyst, but we are not able to figure out
233 00:26:00.130 ⇒ 00:26:09.499 Awaish Kumar: that our pipeline is failing, or it’s, like, just pinging Catalyst, spamming Catalyst, because we were not able to write to BigQuery.
234 00:26:09.910 ⇒ 00:26:13.030 Awaish Kumar: Because we didn’t have the trigger for that, we didn’t have any…
235 00:26:13.220 ⇒ 00:26:15.730 Awaish Kumar: No notification set up for that pipeline.
236 00:26:15.730 ⇒ 00:26:20.540 Ashwini Sharma: Notification was not there, but yeah, exception was being generated, it was printed in the console.
237 00:26:20.920 ⇒ 00:26:23.420 Awaish Kumar: Like, it was silent for…
238 00:26:23.790 ⇒ 00:26:30.149 Awaish Kumar: For someone… for me, like, I’m not going regularly into Dexter. For me, it was a silent error, and I don’t know…
239 00:26:30.190 ⇒ 00:26:45.219 Awaish Kumar: I thought pipeline is just working fine. So, similar to this, I’m saying… I’m not saying that it wasn’t generating… it wasn’t… might not be generating an error, maybe the data frame, while reading… while you are reading the data from…
240 00:26:45.250 ⇒ 00:27:00.500 Awaish Kumar: C file, you said if data frame is empty, and then because of network issues, our data didn’t download it perfectly in a data frame, and it was empty, and we say, if it is empty, just continue, or something like that. If I… and it didn’t… it will not generate an error then, and we will miss rows.
241 00:27:00.590 ⇒ 00:27:07.189 Awaish Kumar: It could be anything. It could be, we are here to find out, you know, these edge cases and…
242 00:27:07.340 ⇒ 00:27:25.450 Awaish Kumar: and come up with a better solution. It’s not just… it’s not, like… like, it could be anything. In the code, you… like, that’s a perfect scenario, where I would say, if my… I’m… I read it from a file, and my data frame don’t read any rows, if the file is empty, I don’t want to proceed with the loading.
243 00:27:25.830 ⇒ 00:27:30.040 Awaish Kumar: That’s a common thing, right? But… Maybe because of…
244 00:27:30.390 ⇒ 00:27:37.790 Awaish Kumar: Some network issue, whatever issue, my data frame didn’t load this file, and it was empty, and it didn’t move forward.
245 00:27:38.950 ⇒ 00:27:42.589 Ashwini Sharma: Yeah, if it is empty, it will not load. That is a known thing, right? Yeah.
246 00:27:42.590 ⇒ 00:27:58.899 Awaish Kumar: Yeah, but that’s the point. While reading from the CSV file, maybe some… there was some timeout issue happened or something, that’s why it didn’t load, and we didn’t check for retry or anything. Maybe, like, we didn’t…
247 00:27:59.260 ⇒ 00:28:07.470 Awaish Kumar: we didn’t explore all the edge cases. Could be… could be anything. Like, just think about… it could be anything, right? And…
248 00:28:07.800 ⇒ 00:28:16.389 Awaish Kumar: what could else be? Like, you know that there are files which has data, which has complete data. That’s what you are telling me.
249 00:28:16.760 ⇒ 00:28:21.080 Awaish Kumar: Right? If you’re telling me, like, I have 60 files which has all the data, then what.
250 00:28:21.080 ⇒ 00:28:23.199 Ashwini Sharma: No, some of them, some of them will be empty.
251 00:28:24.110 ⇒ 00:28:26.960 Awaish Kumar: Yeah, yeah, that’s, like…
252 00:28:27.300 ⇒ 00:28:40.829 Awaish Kumar: I mean, my point is, like, you’re the one who was telling me that when you generate files using Spins API, those files, doesn’t matter how many generated, 10, 20, 50, they will have
253 00:28:40.960 ⇒ 00:28:44.269 Awaish Kumar: All the rows. They don’t miss the rows, right?
254 00:28:44.270 ⇒ 00:29:00.409 Ashwini Sharma: No, like, if… if there is… see, there are multiple conditions in the… when you hear the API, right? Now, you hear the… you don’t hear the API only once, okay? You hear the API multiple times, because it’s… that’s how the pipeline is designed, right? Okay, listen to me.
255 00:29:00.410 ⇒ 00:29:07.679 Awaish Kumar: My point is, like you mentioned, that the CSV file that is being generated Using Sprint’s API,
256 00:29:07.900 ⇒ 00:29:12.420 Awaish Kumar: Must have all the rows, like, or you can say.
257 00:29:12.700 ⇒ 00:29:18.140 Awaish Kumar: does have the similar data what you generate through Export API.
258 00:29:18.340 ⇒ 00:29:19.190 Awaish Kumar: Right?
259 00:29:21.200 ⇒ 00:29:24.870 Ashwini Sharma: Okay, let me speak more clearly, right?
260 00:29:24.970 ⇒ 00:29:40.469 Ashwini Sharma: you hit the API once, it will generate a set of CSV files, right? You hit the API again, it will generate another set of CSC files. You hit the API again, it’s going to generate, right? Every time you hit the API, it’s going to generate some files. Now, some of these files may be empty.
261 00:29:40.670 ⇒ 00:29:42.069 Ashwini Sharma: And that is okay.
262 00:29:43.300 ⇒ 00:29:49.109 Awaish Kumar: Yeah, yeah, my whole point is that, Ashwini, my whole point is that
263 00:29:49.460 ⇒ 00:29:54.350 Awaish Kumar: So, I’m just saying, like, we discussed already, like, I said.
264 00:29:54.600 ⇒ 00:30:08.859 Awaish Kumar: if you are hitting an API, on getting rows by rows data, you might be missing some… some API request fail, and you missed the data. But then you mentioned that your approach is same, kind of, right now, also.
265 00:30:08.960 ⇒ 00:30:21.640 Awaish Kumar: you generate a CSV file, which has all the data. Now you’re just sending in another API request to move data from GCS to S3 through some better way, that’s why you are not missing rows.
266 00:30:21.920 ⇒ 00:30:26.380 Awaish Kumar: Previously, you were writing some Python script, which reads…
267 00:30:27.320 ⇒ 00:30:31.129 Awaish Kumar: data from GCS in a data frame and then loads this to S3, right?
268 00:30:32.000 ⇒ 00:30:32.610 Ashwini Sharma: Yep.
269 00:30:32.940 ⇒ 00:30:38.629 Awaish Kumar: So, in both conversations, we both agree that our first SPINS API
270 00:30:38.830 ⇒ 00:30:45.279 Awaish Kumar: Which… first files, CSV file generated by Spring API is correct?
271 00:30:45.670 ⇒ 00:30:50.760 Awaish Kumar: And… It has the correct data.
272 00:30:51.660 ⇒ 00:30:52.500 Awaish Kumar: Right?
273 00:30:53.170 ⇒ 00:30:54.780 Ashwini Sharma: Okay, no, let me ask you a question.
274 00:30:54.780 ⇒ 00:30:55.366 Awaish Kumar: I’m… I’m.
275 00:30:56.060 ⇒ 00:30:56.470 Awaish Kumar: Yeah.
276 00:30:56.470 ⇒ 00:30:59.250 Ashwini Sharma: One quick question, like, before moving forward.
277 00:30:59.690 ⇒ 00:31:00.070 Awaish Kumar: Yeah, yeah.
278 00:31:00.070 ⇒ 00:31:11.989 Ashwini Sharma: the API that I’m running is running fine, right? The previous one. It is loading data into Redshift, right? I am not seeing any errors in the console where I’m running the pipeline, right?
279 00:31:12.190 ⇒ 00:31:16.439 Ashwini Sharma: How am I supposed to know that the data that I loaded was wrong?
280 00:31:16.590 ⇒ 00:31:21.760 Awaish Kumar: That’s… that’s what we are talking about right now. What you are saying, I…
281 00:31:22.150 ⇒ 00:31:28.230 Awaish Kumar: That’s why I… my first question was, maybe because Spin’s API was not giving us data.
282 00:31:28.370 ⇒ 00:31:33.019 Awaish Kumar: with our first endpoint, it was giving us incorrect data. That was my understanding.
283 00:31:33.340 ⇒ 00:31:35.800 Awaish Kumar: That the first endpoint, you were…
284 00:31:35.950 ⇒ 00:31:41.090 Awaish Kumar: The first solution you tried was missing some data, that’s why we moved to another solution, and it worked.
285 00:31:41.350 ⇒ 00:31:47.850 Awaish Kumar: But now, what you are saying with that, if I take that statement, Our,
286 00:31:48.790 ⇒ 00:31:57.060 Awaish Kumar: Sprint API actually generating same file which we are transferring in both the solutions. Yes. Right? In that case.
287 00:31:57.180 ⇒ 00:32:08.980 Awaish Kumar: Sprint API actually gives us the data in those CSV files in both solutions. That means that is correct. Only problem could… where could be the problem? Like, the problem could only.
288 00:32:08.980 ⇒ 00:32:20.519 Ashwini Sharma: No, no, you’re looking at the hindsight, no? You’re looking at the hindsight and then saying that where could be the problem. Now that the solution is working, you’re saying, okay, since this is working, the problem could have been earlier only.
289 00:32:20.520 ⇒ 00:32:22.790 Awaish Kumar: No, no, I get it, I get it.
290 00:32:22.790 ⇒ 00:32:27.959 Ashwini Sharma: In the hindsight, I also understand there is a problem in the way that I was loading the data into.
291 00:32:27.960 ⇒ 00:32:28.980 Awaish Kumar: That’s fair.
292 00:32:28.980 ⇒ 00:32:29.909 Ashwini Sharma: Whatever the problem was, right?
293 00:32:29.910 ⇒ 00:32:39.260 Awaish Kumar: That’s the… that’s the whole point, like, what… that’s… if… if you would have… if we would have, like, I won’t… I won’t want to blame anyone here, like, just…
294 00:32:39.520 ⇒ 00:32:46.910 Awaish Kumar: Let’s try to understand how we are going to get to the other, like, other ways, like, when we come up with the, okay, let’s discuss.
295 00:32:47.190 ⇒ 00:32:52.620 Awaish Kumar: Alternate, alternate, like, all the… let’s discuss the solution with the team before you start execution.
296 00:32:52.730 ⇒ 00:33:04.919 Awaish Kumar: If you would have told me that I’ve… I have files in GCS, I have files in S3, I want to move from files from here to there, and that’s why I’m using Python script, maybe I would tell you to use Polytomic.
297 00:33:05.260 ⇒ 00:33:07.580 Awaish Kumar: Maybe I would tell you to do something else.
298 00:33:08.210 ⇒ 00:33:22.119 Awaish Kumar: Right? We didn’t discuss that, right? We didn’t actually have that process. It’s not on blame on you or on me, like, it’s just, we didn’t have the process where you could just tell us, here are the solutions I’m thinking about, can you please review it?
299 00:33:22.260 ⇒ 00:33:29.689 Awaish Kumar: And then we can actually… you can get my feedback, Utam’s feedback, or Demi’s feedback, or anyone’s feedback, right, from the team.
300 00:33:29.760 ⇒ 00:33:45.760 Awaish Kumar: And maybe we could… when we are all together spending, like, 10-15 minutes reviewing the solution, we could come up with some better solution. Maybe I tried something else somewhere, or you have tried something somewhere that could help me, right? It could be anything.
301 00:33:46.750 ⇒ 00:33:53.119 Ashwini Sharma: Yeah, sure, next time, next time, if there is a situation like this, I’ll, I’ll review, review, get, get the architecture reviewed with you guys.
302 00:33:53.120 ⇒ 00:33:54.489 Awaish Kumar: Yeah, that’s the whole…
303 00:33:54.600 ⇒ 00:34:08.500 Awaish Kumar: So these are the things I’m talking about. Now, like, previously, I’m just post-morteming, post-mortem, like, post-morteming the script, that, okay, what would be the problem? As a, as a, like, I’m not saying, like.
304 00:34:08.760 ⇒ 00:34:14.820 Awaish Kumar: I’m not doing a post-mortem because you wrote it, I’m just doing… even if I wrote that.
305 00:34:14.960 ⇒ 00:34:27.040 Awaish Kumar: I… what would be my thoughts? Why is failing? Why am I missing data? I would go for, like, maybe… because some files are empty, and I wrote a condition where I’m saying, if data frame is empty, don’t load it.
306 00:34:27.290 ⇒ 00:34:39.569 Awaish Kumar: And maybe sometimes the file was not loading properly. It had the data. And because of this, my condition where I’m saying data frame… data frame is empty and don’t load it.
307 00:34:40.070 ⇒ 00:34:48.540 Awaish Kumar: I’m missing roads. Could be anything, right? I’m just… I was just coming up with all the possible scenarios which could fail my script.
308 00:34:48.810 ⇒ 00:34:50.150 Awaish Kumar: That was one thing.
309 00:34:50.340 ⇒ 00:34:52.479 Awaish Kumar: That’s how you think about
310 00:34:53.500 ⇒ 00:35:00.170 Awaish Kumar: if we… if we both sit together and do a task, that would I think about? Like, when I’m writing a task, what I would I…
311 00:35:00.390 ⇒ 00:35:05.819 Awaish Kumar: Like, where it could fail, where I should write try accept block, where I shouldn’t write that.
312 00:35:06.010 ⇒ 00:35:20.200 Awaish Kumar: where I want to make, like, want to silent… I want it to be silently fail, and where I want it to actually raise an exception if something doesn’t happen. So these are kind of things I would think about. This is the number one.
313 00:35:20.660 ⇒ 00:35:25.920 Awaish Kumar: Second one is just discussing the architecture with the team. That’s what I would go for.
314 00:35:26.390 ⇒ 00:35:32.260 Awaish Kumar: And… Like, then we… what else we could say?
315 00:35:34.090 ⇒ 00:35:38.650 Awaish Kumar: Apart from that.
316 00:35:43.140 ⇒ 00:35:46.259 Awaish Kumar: So, yeah, apart from that, I think,
317 00:35:46.430 ⇒ 00:35:55.610 Awaish Kumar: This was just one of the… Example I took, Oh… To discuss the…
318 00:35:59.690 ⇒ 00:36:02.690 Awaish Kumar: the… the approaches. The next thing would be…
319 00:36:03.090 ⇒ 00:36:05.710 Awaish Kumar: That’s why I want to understand from you, like, if…
320 00:36:05.950 ⇒ 00:36:16.660 Awaish Kumar: What… how you approach it, and how… what else you, like, enjoy doing, like, for example, analytics…
321 00:36:17.470 ⇒ 00:36:22.720 Awaish Kumar: part of it, or data engineering part, like, we… Oh…
322 00:36:23.070 ⇒ 00:36:30.350 Awaish Kumar: I just… I want to understand that as well, what… what… where you want to… where you enjoy being, like, being a building of DVD models.
323 00:36:30.500 ⇒ 00:36:36.350 Awaish Kumar: Building those kind of pipelines are… Or anything else?
324 00:36:37.210 ⇒ 00:36:40.390 Ashwini Sharma: No, I think dbt models and pipelines would be good. I think that’s where.
325 00:36:40.390 ⇒ 00:36:41.000 Awaish Kumar: I know.
326 00:36:42.920 ⇒ 00:36:43.860 Ashwini Sharma: No good.
327 00:36:45.230 ⇒ 00:36:46.200 Awaish Kumar: Okay.
328 00:36:47.180 ⇒ 00:36:49.960 Ashwini Sharma: I don’t know about analytics, like, I’ve not done analytics.
329 00:36:50.630 ⇒ 00:36:52.580 Awaish Kumar: Previously, right.
330 00:36:52.910 ⇒ 00:36:53.840 Ashwini Sharma: A-B testing.
331 00:36:53.840 ⇒ 00:36:54.180 Awaish Kumar: Okay.
332 00:36:54.180 ⇒ 00:36:54.950 Ashwini Sharma: like that.
333 00:36:59.340 ⇒ 00:37:01.689 Awaish Kumar: So I think, yeah, that’s…
334 00:37:01.940 ⇒ 00:37:07.219 Awaish Kumar: And, like, next time, like, do you have any tickets where you think we can collaborate?
335 00:37:08.850 ⇒ 00:37:17.829 Ashwini Sharma: Right now, I don’t think there is any such kind of ticket. I mean, the other tickets are straightforward and simple.
336 00:37:20.030 ⇒ 00:37:22.120 Ashwini Sharma: No, there is no such tickets.
337 00:37:22.330 ⇒ 00:37:23.110 Ashwini Sharma: Right now.
338 00:37:26.280 ⇒ 00:37:27.150 Awaish Kumar: Okay.
339 00:37:33.300 ⇒ 00:37:37.460 Ashwini Sharma: I don’t know, is there anything in the Eden OS that’s coming up that could be…
340 00:37:37.460 ⇒ 00:37:41.750 Awaish Kumar: Oh, dangish, it was just models. I think you, you created some…
341 00:37:42.210 ⇒ 00:37:42.960 Ashwini Sharma: Yeah, there’s a PR.
342 00:37:42.960 ⇒ 00:37:45.700 Awaish Kumar: So, but there’s no way for me to even…
343 00:37:46.090 ⇒ 00:37:50.080 Ashwini Sharma: There’s no way for me also to look at that. I mean, without data, what can I… For us?
344 00:37:50.080 ⇒ 00:37:53.969 Awaish Kumar: To validate our models, like, we need to have some data.
345 00:37:54.260 ⇒ 00:38:05.420 Awaish Kumar: Even only after that, I can actually… I can run carries, or I can ask you to run case when there is something. Right now, there are only column names, and they look good.
346 00:38:06.990 ⇒ 00:38:13.070 Ashwini Sharma: Yeah, yeah, I mean, like, without data, can’t verify any model, right? Not even the… those staging models,
347 00:38:14.350 ⇒ 00:38:15.459 Ashwini Sharma: Those raw models.
348 00:38:15.460 ⇒ 00:38:21.929 Awaish Kumar: Let’s see, there is nothing on Eden OS. Eden OS is just kind of modeling and reporting.
349 00:38:22.080 ⇒ 00:38:31.899 Awaish Kumar: We don’t have anything on Eden Noise from our side, like, we have to do modeling, and then maybe we can ask… get some help from
350 00:38:32.070 ⇒ 00:38:36.679 Awaish Kumar: team, on the Omni side, and get some reporting up through that.
351 00:38:36.830 ⇒ 00:38:38.590 Awaish Kumar: In the Omni.
352 00:38:39.320 ⇒ 00:38:41.130 Awaish Kumar: Or… yeah.
353 00:38:41.750 ⇒ 00:38:48.490 Awaish Kumar: And that’s all on that, but apart from that, like, Then, like… Pink, I…
354 00:38:49.840 ⇒ 00:38:52.680 Awaish Kumar: What else we can think of?
355 00:38:55.160 ⇒ 00:38:57.650 Awaish Kumar: We have this new, I think.
356 00:38:58.270 ⇒ 00:39:04.770 Awaish Kumar: scanner? I don’t know what that was. Like, we can take that, for example. That’s something new, right?
357 00:39:05.160 ⇒ 00:39:06.080 Awaish Kumar: Maybe we go.
358 00:39:06.080 ⇒ 00:39:06.580 Ashwini Sharma: Yeah, no.
359 00:39:06.580 ⇒ 00:39:07.130 Awaish Kumar: I’m wrong.
360 00:39:07.130 ⇒ 00:39:10.049 Ashwini Sharma: Yeah, so the requirement is not clear or anything, right?
361 00:39:10.050 ⇒ 00:39:13.760 Awaish Kumar: We don’t know the requirements, so what we can do, we can…
362 00:39:14.180 ⇒ 00:39:20.429 Awaish Kumar: Like, yeah, we can collaborate on that, like, we can sit together to get…
363 00:39:20.550 ⇒ 00:39:23.690 Awaish Kumar: transcript from, maybe, Utam’s and Catherine’s
364 00:39:23.920 ⇒ 00:39:28.360 Awaish Kumar: Meeting. Come up with some…
365 00:39:28.760 ⇒ 00:39:38.249 Awaish Kumar: Stuff that we understand from that, and then ask any… Like, the clarifying questions?
366 00:39:38.500 ⇒ 00:39:45.489 Awaish Kumar: Like, maybe one… create a one-pager for that, and we… again, maybe we can collaborate on these kind of things.
367 00:39:45.800 ⇒ 00:39:49.080 Awaish Kumar: Right now, because there’s no… pipeline…
368 00:39:50.610 ⇒ 00:39:54.639 Awaish Kumar: There’s no new pipeline building tasks right now, so we can just do…
369 00:39:55.090 ⇒ 00:39:56.790 Awaish Kumar: collaborate on this, like, I would…
370 00:39:56.970 ⇒ 00:40:02.139 Awaish Kumar: That’s how would I approach it, like, when I don’t have any clear requirements?
371 00:40:02.330 ⇒ 00:40:12.399 Awaish Kumar: from, from the client. Like, instead of sending them one-two messages, and then one, two messages.
372 00:40:12.600 ⇒ 00:40:16.630 Awaish Kumar: I will try to go to grab a… .
373 00:40:16.630 ⇒ 00:40:20.920 Ashwini Sharma: Yeah, I’ll grab some time with Otam, let’s pick his brain before, you know…
374 00:40:20.920 ⇒ 00:40:29.340 Awaish Kumar: Like, even before that, before grabbing time with Utam, I’m saying it does… did he have the meeting?
375 00:40:29.630 ⇒ 00:40:31.260 Awaish Kumar: like, in Zora.
376 00:40:31.260 ⇒ 00:40:31.760 Ashwini Sharma: Thank you so much.
377 00:40:31.760 ⇒ 00:40:33.970 Awaish Kumar: If yes, we could get the transcript.
378 00:40:34.460 ⇒ 00:40:35.350 Ashwini Sharma: Mmm…
379 00:40:35.860 ⇒ 00:40:43.149 Ashwini Sharma: Well, he didn’t mention anything, I think it was just Slack conversation that is happening, he’s just posting Slack messages from a different…
380 00:40:43.420 ⇒ 00:40:44.780 Ashwini Sharma: conversation.
381 00:40:45.260 ⇒ 00:40:45.799 Ashwini Sharma: It’s a private.
382 00:40:45.800 ⇒ 00:40:47.340 Awaish Kumar: Yeah, he’s hoping…
383 00:40:48.340 ⇒ 00:40:56.650 Awaish Kumar: Yeah, we can… we can, like, go through that. We can ask, like, we can ask him if… if it is coming from some Zoom. If not, then just…
384 00:40:57.560 ⇒ 00:41:06.970 Awaish Kumar: whatever is there, just put it in a document, and all your questions that you asked in Slack, just put it also, all of them in the document, and we can then…
385 00:41:07.150 ⇒ 00:41:16.470 Awaish Kumar: ask Catherine to, okay, we have all the questions listed here, please reply. She can also write down answers at the bottom of that question, so it would be easier
386 00:41:17.280 ⇒ 00:41:18.790 Awaish Kumar: In terms of communication.
387 00:41:20.610 ⇒ 00:41:27.999 Ashwini Sharma: Yeah, sure, we can do that. I’ll add my questions to a doc, and then, be shared it with Catherine.
388 00:41:30.670 ⇒ 00:41:40.870 Awaish Kumar: And, like, I don’t have anything right now. I will also try to find out anything, and maybe on Monday,
389 00:41:41.470 ⇒ 00:41:48.539 Awaish Kumar: We, like, on Monday, we can work together on one of the tasks.
390 00:41:48.770 ⇒ 00:41:54.929 Awaish Kumar: And maybe, like, one of us will just share a screen, maybe I will just share my screen.
391 00:41:55.030 ⇒ 00:42:00.580 Awaish Kumar: I will work through it. We just brain… we just talk about it, like, brainstorm together.
392 00:42:00.860 ⇒ 00:42:03.150 Awaish Kumar: But only one has to do, right?
393 00:42:03.150 ⇒ 00:42:03.510 Ashwini Sharma: Okay.
394 00:42:03.510 ⇒ 00:42:11.909 Awaish Kumar: We can just brainstorm together, like, what prompts to write, maybe, like, and then how to get the solution.
395 00:42:12.160 ⇒ 00:42:14.880 Awaish Kumar: And… and that’s all. And then,
396 00:42:15.290 ⇒ 00:42:21.119 Awaish Kumar: I will try to see if I have any such kind of… any ticket where we can do that, and then,
397 00:42:21.530 ⇒ 00:42:23.279 Awaish Kumar: I think that’s… that’s the plan.
398 00:42:24.790 ⇒ 00:42:26.609 Ashwini Sharma: Correct, yeah, we can do that.
399 00:42:27.050 ⇒ 00:42:37.550 Awaish Kumar: And also, in the meantime, if you come across anything, just send it over so I can review, and we can then… I can make it a part of agenda for Monday.
400 00:42:37.990 ⇒ 00:42:38.990 Ashwini Sharma: Okay, yeah.
401 00:42:38.990 ⇒ 00:42:39.660 Awaish Kumar: Okay.
402 00:42:42.500 ⇒ 00:42:44.340 Awaish Kumar: Okay, great, thank you.
403 00:42:44.500 ⇒ 00:42:46.160 Ashwini Sharma: Okay, alright, yeah, thanks.
404 00:42:46.670 ⇒ 00:42:47.390 Awaish Kumar: Bye.
405 00:42:47.650 ⇒ 00:42:48.570 Ashwini Sharma: Okay, bye.