Meeting Title: Data migration timelines Date: 2025-09-30 Meeting participants: Awaish Kumar, Ryon
WEBVTT
1 00:01:21.140 ⇒ 00:01:22.550 Ryon: Hey, Wish, how are you doing?
2 00:01:25.370 ⇒ 00:01:26.640 Awaish Kumar: I’m good, how about you?
3 00:01:27.390 ⇒ 00:01:28.690 Ryon: Doing well, doing well.
4 00:02:00.550 ⇒ 00:02:06.800 Ryon: I don’t want Mitesh to be on this, so I’ll wait to see what, and he comes on.
5 00:02:13.910 ⇒ 00:02:17.539 Awaish Kumar: So basically, I don’t.
6 00:02:17.540 ⇒ 00:02:30.580 Ryon: So we need to chat about the timelines a little bit here. I think we need to understand, not just the timelines a bit more, but the scope of the data, and exactly what it is that you guys might need from us to accelerate the process.
7 00:02:30.580 ⇒ 00:02:44.429 Ryon: In a nutshell, you know, four to six weeks is a bit longer than we were anticipating. So, the first thing is understanding, you know, why is it going to take, or why do you guys scope it, or estimate it taking so long, number one? Number two, what can we do to…
8 00:02:44.500 ⇒ 00:02:49.519 Ryon: Simplify the process for you, so that, you know, it doesn’t take that long, so…
9 00:02:51.970 ⇒ 00:02:54.999 Ryon: Interesting. This person also came through common.
10 00:02:55.000 ⇒ 00:03:00.990 Awaish Kumar: Wow, and… So, are we… will be… will we be waiting for Remo?
11 00:03:01.430 ⇒ 00:03:03.340 Awaish Kumar: Project to be done for this, or not?
12 00:03:03.500 ⇒ 00:03:17.620 Ryon: So, the reality is, like, yeah, we probably are gonna need the Remo project to be completed for this, in order to have you guys move ahead, but the ETA on that is sort of unknown at this point.
13 00:03:17.700 ⇒ 00:03:27.540 Ryon: Like, we know, but we also don’t. Like, I can sort of say it’s probably going to be roughly a month out, from where we currently are, so as far as the timing is concerned, I know that Amber was asking about the…
14 00:03:28.210 ⇒ 00:03:32.590 Ryon: planning page that I have, and right now I would say, okay, that’s a…
15 00:03:33.330 ⇒ 00:03:33.840 Awaish Kumar: Yeah, the…
16 00:03:33.840 ⇒ 00:03:34.970 Ryon: That’s able.
17 00:03:35.190 ⇒ 00:03:37.610 Awaish Kumar: In the remote proposal, we share
18 00:03:37.720 ⇒ 00:03:41.239 Awaish Kumar: Also, like, is about… about, like, 2 months.
19 00:03:41.570 ⇒ 00:03:42.810 Awaish Kumar: worked there.
20 00:03:43.980 ⇒ 00:03:54.319 Ryon: Yeah, yeah, it’s old. Like, your proposal is dated like you guys showed it a while ago, and my planning page is somewhat dated as well.
21 00:03:54.500 ⇒ 00:04:09.930 Ryon: Okay, so we know that, but what we need to understand from you guys is what… why do you guys believe, or what makes you think that it’s gonna take you up to, it’s gonna take you up to 2 months… well, not 2 months, a month and a half to do this.
22 00:04:10.320 ⇒ 00:04:17.450 Awaish Kumar: Like, okay, so let me… understand, like, understand it properly. So, number one is.
23 00:04:17.940 ⇒ 00:04:25.689 Awaish Kumar: the remote comes in terms of work, right? That is… is a separate project, and here we are discussing only the… only the
24 00:04:25.930 ⇒ 00:04:35.300 Awaish Kumar: migration of the… all the data, all the data pipelines, right, from bus to… to EMR platform, basically.
25 00:04:36.000 ⇒ 00:04:39.300 Awaish Kumar: Or removing it from. So, like, I want to…
26 00:04:40.130 ⇒ 00:04:52.730 Awaish Kumar: I want to make sure that we are not confusing between these two. These two are two different projects. Either we are going to wait for migrations or not. There’s a, like, either we are going to wait,
27 00:04:53.070 ⇒ 00:05:00.609 Awaish Kumar: for the remote project to be completed for this migration or not. That’s a separate thing, but, like, these are two different things.
28 00:05:01.110 ⇒ 00:05:09.700 Ryon: Okay, so let me parse it out in terms of time. So, the Remo project, which is where you guys are sort of advising on the setup.
29 00:05:09.810 ⇒ 00:05:20.310 Ryon: how much… that’s the first part of the project, basically, right? How much time are you estimating for that? Like, what’s the time frame for that? I know it’s kind of an open-ended question, but like, what’s the time?
30 00:05:20.310 ⇒ 00:05:28.770 Awaish Kumar: But, like, we have built all the deliverables, for that Remo project, where we have, like, the…
31 00:05:29.470 ⇒ 00:05:42.600 Awaish Kumar: the SME, like, the expert backend engineer, which is going to come in and see everything, the multi-tenancy things and all of that, looking for database architecture.
32 00:05:43.020 ⇒ 00:05:52.960 Awaish Kumar: He’s going to look at, like, software architecture and, everything between that, like, all the tools text being used, or how… how…
33 00:05:53.430 ⇒ 00:06:05.189 Awaish Kumar: easy it is to, like, scale it or not, like, all these things. So all these are, like, split out in a separate deliverable, and that’s what I was saying, that
34 00:06:05.310 ⇒ 00:06:09.900 Awaish Kumar: That spans across, like, almost, like, 6 to 8 weeks.
35 00:06:11.640 ⇒ 00:06:15.459 Ryon: Okay, so the actual act of migrating the data…
36 00:06:15.650 ⇒ 00:06:27.940 Ryon: the part where you guys are going to pull over all the existing data from what exists in Basket or BigQuery over to the Remo world, we’ll call it the Remo EMR. How long do you guys think that’ll take?
37 00:06:28.420 ⇒ 00:06:31.590 Awaish Kumar: Okay, now, this is one more ask.
38 00:06:31.760 ⇒ 00:06:32.940 Awaish Kumar: So…
39 00:06:33.450 ⇒ 00:06:41.239 Awaish Kumar: I want to, like… like, number one was consult… consulting remote. We just, like, put it aside. Second thing is.
40 00:06:42.410 ⇒ 00:06:59.279 Awaish Kumar: migration of data pipelines from Basque to Remo. That means we… Remo will be a platform where it will run the operations, and we are going to move all of our data pipelines to read the data from Remo’s webhooks
41 00:06:59.370 ⇒ 00:07:07.540 Awaish Kumar: Bring the data in, and perform all the… adjust the… Data transformation based on
42 00:07:07.650 ⇒ 00:07:18.890 Awaish Kumar: new data structure, and then, like, continue to, solve the reports. That’s one thing. Second thing, what I’m getting from you is migrating the data from
43 00:07:19.690 ⇒ 00:07:32.279 Awaish Kumar: Basque platform to remote. Is that any scope for us? I’m not sure about that. Like, we don’t have, like, access to… we do have access to warehouse. Do you want us to…
44 00:07:32.470 ⇒ 00:07:41.999 Awaish Kumar: also, like, work on reverse detailing, like, working on getting the data from warehouse and putting it back to the remote platform itself.
45 00:07:42.330 ⇒ 00:07:48.480 Ryon: We need… we need all the data from whatever existed in BASC, Piped through.
46 00:07:48.910 ⇒ 00:07:50.000 Ryon: to the Remo.
47 00:07:50.150 ⇒ 00:07:50.690 Ryon: AMR.
48 00:07:50.690 ⇒ 00:07:51.490 Awaish Kumar: Okay.
49 00:07:52.200 ⇒ 00:07:56.980 Awaish Kumar: And that, we are going to… Do, like, through…
50 00:07:57.060 ⇒ 00:08:01.539 Ryon: Yes, that’s gonna be you guys, Brainforge, you guys are gonna have to do that, yes.
51 00:08:02.140 ⇒ 00:08:08.180 Awaish Kumar: Okay, so, like, okay, I didn’t know that, so, like, we… when we scoped out all the
52 00:08:08.390 ⇒ 00:08:23.419 Awaish Kumar: the document we have shared with you already is mostly, like, getting data from Waymo, and then adjusting the transformations, and then, basically continue to serve the reports we are serving in Tableau.
53 00:08:23.840 ⇒ 00:08:25.490 Ryon: Yes. So…
54 00:08:26.250 ⇒ 00:08:46.039 Ryon: That would be the process… so it sounds like there’s three things going on. The first of which is your guys’ data architecture consultancy with Remo. That’s part one, okay? Which, I’m not gonna speak to that, whatever the timeline is, the timeline is. The second part is you guys setting up the inbound data
55 00:08:46.040 ⇒ 00:08:49.160 Ryon: from Remo, in the new system that they create.
56 00:08:49.700 ⇒ 00:08:55.750 Ryon: But there’s a third component that I need you guys to work on, which is… we need to have…
57 00:08:55.960 ⇒ 00:09:07.759 Ryon: all the existing data from the data warehouse that BASC has provided us, piped over through to Remo, so that our MX team can, of course, see what’s there.
58 00:09:09.460 ⇒ 00:09:10.270 Awaish Kumar: Okay.
59 00:09:10.520 ⇒ 00:09:18.919 Awaish Kumar: How do you want us to support that? That being said, like, are you, like, will be providing the API endpoints?
60 00:09:19.080 ⇒ 00:09:21.630 Awaish Kumar: Using that, we will be pushing the data
61 00:09:21.760 ⇒ 00:09:24.460 Awaish Kumar: Like, how will we be pushing the data to remote place now?
62 00:09:25.690 ⇒ 00:09:29.860 Ryon: I need to talk to Cameron on that. The…
63 00:09:34.790 ⇒ 00:09:37.650 Ryon: I see one of two things he might do here.
64 00:09:38.090 ⇒ 00:09:47.930 Ryon: The first of which is he might just create new tables inside our existing data warehouse, which means your guys’ job is just to move data from one table to another, right?
65 00:09:48.080 ⇒ 00:09:49.299 Ryon: And then, you know.
66 00:09:49.530 ⇒ 00:09:53.810 Ryon: Make sure that it actually populates inside of the platform the way you expect it to.
67 00:09:55.130 ⇒ 00:10:02.729 Ryon: The second way is what you described, which is he provides you guys with an API endpoint, and then you just feed the data to the platform that way.
68 00:10:02.860 ⇒ 00:10:03.650 Ryon: Right.
69 00:10:05.900 ⇒ 00:10:18.040 Ryon: I am more inclined to believe he will do the former, not the latter, because I’ve been asking him to create APIs for a while, and he has not done any of that. These are for the intakes themselves.
70 00:10:18.040 ⇒ 00:10:18.440 Awaish Kumar: Right at the end.
71 00:10:18.440 ⇒ 00:10:34.020 Ryon: intakes themselves. Additionally, I think that if we were to create an API approach, he might treat those customers as new customers, not as existing customers. So that might change their state or some things about them inside of the platform itself.
72 00:10:37.700 ⇒ 00:10:44.019 Ryon: I’ll need to get clarity for you guys on that. Either way, what would be the timeframe for that to occur?
73 00:10:46.260 ⇒ 00:10:48.669 Awaish Kumar: Yeah, but for that, I need,
74 00:10:49.240 ⇒ 00:10:56.059 Awaish Kumar: I can prepare a notion doc on that, basically. For you, I can list out,
75 00:10:56.360 ⇒ 00:10:59.150 Awaish Kumar: The tables we are getting from past.
76 00:10:59.410 ⇒ 00:11:06.019 Awaish Kumar: and the timeline for writing all those pipelines. And for that, I will need, right.
77 00:11:06.170 ⇒ 00:11:11.400 Awaish Kumar: like, if it’s just a table, you want me to just, like, I have a table in warehouse.
78 00:11:11.620 ⇒ 00:11:19.129 Awaish Kumar: and you… in Google BigQuery, basically, and you want me to move that table exactly as it is to Postgres.
79 00:11:19.270 ⇒ 00:11:33.109 Awaish Kumar: That might be easier for me to do it. But if you want me to do it through API, that would maybe require me to do some scripting, like, I might have to write some pipeline, or a Python script, or something, that would increase the time, so I want to understand that.
80 00:11:34.060 ⇒ 00:11:37.760 Awaish Kumar: And then I can, list out the tables.
81 00:11:38.110 ⇒ 00:11:40.410 Awaish Kumar: And the timeline, and share it with you.
82 00:11:42.800 ⇒ 00:11:45.060 Ryon: That would be great. Thank you, appreciate that.
83 00:11:45.240 ⇒ 00:11:54.670 Ryon: But I just want to clarify, the proposal that you had sent me prior was covering the… this is what we’re going to set up to capture data from Remo.
84 00:11:54.790 ⇒ 00:11:55.840 Awaish Kumar: Yes. Right.
85 00:11:55.840 ⇒ 00:11:56.550 Ryon: Okay.
86 00:11:56.550 ⇒ 00:12:12.610 Awaish Kumar: That is a task in a way that it’s not just getting data from Remo, right? We have to comply with the existing data, which is coming from BASC. We might have data schema changes, right?
87 00:12:12.880 ⇒ 00:12:18.550 Awaish Kumar: Remo might be treating some… a few things, like, in a different way than Bas, and we have to…
88 00:12:18.970 ⇒ 00:12:22.529 Awaish Kumar: Like, work on, like, capturing all those business…
89 00:12:22.780 ⇒ 00:12:28.610 Awaish Kumar: edge cases, right? And for that, we need time, that’s what I meant.
90 00:12:28.830 ⇒ 00:12:29.690 Ryon: Okay.
91 00:12:30.070 ⇒ 00:12:40.619 Awaish Kumar: In a worst-case scenario, I know you can’t give me, like, an exact number, because you don’t have all the details, but let’s just say, in a worst-case scenario, what’s the maximum amount of time you think the beta migration would take?
92 00:12:41.610 ⇒ 00:12:44.940 Awaish Kumar: Just from, like, okay, we are talking about…
93 00:12:44.940 ⇒ 00:12:56.669 Ryon: I’m talking, like, not from… not from BigQuery to Postgres, I don’t actually know if he’s gonna allow that. I’m assuming he would, but, like, in that scenario, what do you figure? One sprint? Two weeks? Or one week?
94 00:12:56.670 ⇒ 00:12:59.270 Awaish Kumar: Yeah, that would be… that would be it.
95 00:12:59.730 ⇒ 00:13:04.749 Ryon: Okay, and then let’s say it’s the… let’s say it’s the API approach. You guys with that.
96 00:13:08.100 ⇒ 00:13:16.130 Awaish Kumar: That… that depends on, like, how much of the transformation we did, how far we are from, existing
97 00:13:16.290 ⇒ 00:13:18.580 Awaish Kumar: Boss… schema.
98 00:13:19.500 ⇒ 00:13:28.069 Ryon: Because you guys would have to connect everything over, basically. Yeah, that’s interesting. How about we say 3 to 4 weeks for that?
99 00:13:29.040 ⇒ 00:13:30.860 Ryon: Is that… Fair.
100 00:13:30.860 ⇒ 00:13:32.830 Awaish Kumar: Oh, that would be fair, yeah.
101 00:13:32.830 ⇒ 00:13:34.429 Ryon: Okay, is that fair? Okay.
102 00:13:34.670 ⇒ 00:13:36.330 Ryon: Alright,
103 00:13:43.890 ⇒ 00:13:44.730 Ryon: Okay.
104 00:13:45.190 ⇒ 00:13:47.049 Ryon: That’s better. I think…
105 00:13:47.330 ⇒ 00:14:00.540 Ryon: We’re still sort of figuring out some things on our end, so it’s good that we have all of this planned out, we’re aware. Thank you for providing us the information, the data. I’m gonna adjust a couple of things inside of Monday, but this helps a lot.
106 00:14:00.590 ⇒ 00:14:08.229 Ryon: So, yes, it was good. And then, yeah, you can inform, Robert and, Amber for our conversation. I’ll let,
107 00:14:08.950 ⇒ 00:14:10.070 Ryon: the afternoon summers.
108 00:14:10.320 ⇒ 00:14:17.120 Awaish Kumar: based on our conversation, I will, split out, like, write down 3 work streams we just talked about.
109 00:14:17.120 ⇒ 00:14:19.540 Ryon: And I will write down…
110 00:14:19.540 ⇒ 00:14:24.519 Awaish Kumar: The proposal at the bottom of that page regarding movement of data from,
111 00:14:24.890 ⇒ 00:14:39.029 Awaish Kumar: BigQuery to Postgres, and how much, like… I can estimate, like, how much it would take, and things like that, list down the tables we need, and, like, we are getting from PASC, and,
112 00:14:39.830 ⇒ 00:14:44.530 Awaish Kumar: Maybe then, like, you can say, okay, like…
113 00:14:44.660 ⇒ 00:14:50.989 Awaish Kumar: Which one are really needed, which are not, or whatever your feedback on that, we can continue over there.
114 00:14:51.430 ⇒ 00:14:55.990 Ryon: Okay, cool. One more question. Are you in the San Francisco Bay Area?
115 00:14:56.400 ⇒ 00:14:58.059 Awaish Kumar: No, I’m in Pakistan.
116 00:14:58.300 ⇒ 00:14:58.920 Ryon: Ow.
117 00:14:59.410 ⇒ 00:15:07.279 Ryon: Your background, I’m in the Bay Area, I was like, oh, late for you, I’ll let you go. Okay, cool.
118 00:15:07.920 ⇒ 00:15:14.729 Ryon: Let me know, yep, all this sounds good. Just, let me know when things are updated, and I’ll add it to my planning. Cool?
119 00:15:15.180 ⇒ 00:15:16.100 Awaish Kumar: Okay, thank you.
120 00:15:16.270 ⇒ 00:15:17.250 Ryon: Thank you, Rash.