Meeting Title: [Eden] Improving Data QA Processes Date: 2025-06-19 Meeting participants: Robert Tseng, Demilade Agboola, Awaish Kumar, Fireflies.ai Notetaker Tigran, Tigran Sahakyan, Josh
WEBVTT
1 00:02:19.550 ⇒ 00:02:20.429 Demilade Agboola: Hell of it!
2 00:02:22.924 ⇒ 00:02:26.490 Robert Tseng: Yeah, I mean, I know we don’t have too much time, so I guess
3 00:02:26.910 ⇒ 00:02:40.359 Robert Tseng: I know that you you mentioned that the table structures are different. I mean, Josh is not gonna care about that. He’s gonna know why. Why, why have we said we fixed it? And there’s still things going wrong. So I think that’s pretty much what he’s been asked
4 00:02:40.950 ⇒ 00:02:56.820 Robert Tseng: and I don’t know what our Qa process is. I mean, people are checking our work every time now, every day, by just referencing vast and looking at stuff. But I mean, I don’t. I don’t know how to structure like what we’re what we’re doing. Like, I I don’t really know what we’re doing. So
5 00:02:57.445 ⇒ 00:03:04.284 Robert Tseng: yeah, I mean, yesterday I was just dealing with the accounting issue, so I didn’t. I didn’t look at this. So
6 00:03:05.200 ⇒ 00:03:09.829 Robert Tseng: yeah, if you could help me, I think that would be helpful.
7 00:03:11.120 ⇒ 00:03:19.630 Demilade Agboola: Yeah, I’m literally, as we’re speaking, trying to put it to together, a document for failure points. So like every failure point so we can mitigate for that.
8 00:03:20.253 ⇒ 00:03:26.299 Demilade Agboola: So that’s from bigquery to Dbt to tableau where anything can fail. So things like
9 00:03:26.510 ⇒ 00:03:33.282 Demilade Agboola: so that we know when things are going wrong in every single space. And that I have a call today with
10 00:03:34.250 ⇒ 00:04:00.329 Demilade Agboola: Well, Tom, what’s setting up meta plane? And so I want to have this doc ready before that call, so that one I could show him. If he has any other failure points we can look at it. And 2, we can ensure that like that’s also integrated into what we’re looking for when we’re setting up meta plane so that we don’t have any like random failures from certain places. So this will include things like freshness. Dbt runs must run
11 00:04:02.740 ⇒ 00:04:12.489 Demilade Agboola: Numbers should like see the setting threshold, like, you know. So we don’t have weird joint. So row counts would not jump massively. Things like that. So that’s that’s what the document is.
12 00:04:14.430 ⇒ 00:04:21.670 Demilade Agboola: but yeah, I I definitely understand that like, yeah, Josh would definitely not be comfortable with the fact that we’ve had a couple of outages this week.
13 00:04:22.123 ⇒ 00:04:24.390 Demilade Agboola: We should definitely get things back on drug.
14 00:04:25.360 ⇒ 00:04:32.660 Robert Tseng: Yeah, so I mean, I don’t really know how Meta plane is set up like, I understand that there’s like some Ddt tests and stuff. But really it’s like.
15 00:04:33.210 ⇒ 00:04:46.499 Robert Tseng: if other people are Qa or work by just going to bask and checking there, and because they trust fast more than our data. Then I think that’s that’s a problem. Right? So like, I feel like, we need to be able to explain the difference
16 00:04:46.790 ⇒ 00:04:55.629 Robert Tseng: between what we are showing versus what Vast is showing like we kinda like. That’s always going to be the conversation, like, I think that’s what people are are telling us every day. So
17 00:04:56.138 ⇒ 00:04:59.319 Robert Tseng: I don’t know how that factors into what you’re describing.
18 00:05:01.768 ⇒ 00:05:19.280 Demilade Agboola: I think once we can handle the failure points that I just referenced. The the only differences we should see in between bask and ourselves are like different logic. So things like, Oh, we filtered out, canceled, abandoned, and error things like that.
19 00:05:19.630 ⇒ 00:05:20.455 Demilade Agboola: But
20 00:05:21.930 ⇒ 00:05:43.459 Demilade Agboola: Beyond that like, because we’re getting it from Basques web hook, web hook, and unless the web hook is out of date, so that we don’t have the right data or we’ve applied a different type of logic to what bask is applying. Everything should be the same. So I I want us to be able to get confidence to the point that we know that if there’s any disparity
21 00:05:44.080 ⇒ 00:05:51.220 Demilade Agboola: it’s either a web hook thing from bask, which obviously we can’t control that or 2.
22 00:05:51.550 ⇒ 00:05:59.760 Demilade Agboola: it’s just a difference in logic. And we can say, Hey, this is the reason behind our logic, and we stand by that unless they want to change the logic that we’re using.
23 00:06:01.870 ⇒ 00:06:18.350 Robert Tseng: Yeah. So I feel like we’re not fast at like getting that. I mean, I think their understanding was all of this was already set up before, which is why we fixed everything like there have. There were no outages for so for so long, and then everything blew up this past week. So it kind of just undoes like
24 00:06:18.500 ⇒ 00:06:19.400 Robert Tseng: at the
25 00:06:19.510 ⇒ 00:06:29.329 Robert Tseng: basically, I’ve just been accused of lying to to the team about everything that we’ve set up in terms of checks and things and guarantees that things aren’t breaking like, yeah. So
26 00:06:29.630 ⇒ 00:06:36.520 Robert Tseng: I I don’t know. I feel like I’m back at 0 in in terms of trust. And I need to like, tell them.
27 00:06:38.310 ⇒ 00:06:54.759 Robert Tseng: yeah, like, if if it’s really if they’re right and we didn’t have any of this actually set up before, and we pretty much just got by being lucky. Then I think that’s we need to acknowledge that, or like I I don’t. I don’t know like I like? That’s
28 00:06:55.330 ⇒ 00:07:01.670 Robert Tseng: I. I don’t. I don’t. I don’t know how to respond to that. So that’s that’s my biggest. That’s my biggest gap. I’m
29 00:07:01.830 ⇒ 00:07:20.940 Robert Tseng: this entire week I’ve been called a liar, that everything that we’ve been doing is just not really up up to the sla’s that I had promised, and I don’t know how to respond to that. I I just I don’t have any process. I can point them to like. All. All I’ve been able to show is that our team is scrambling to answer things, and like I,
30 00:07:21.080 ⇒ 00:07:34.399 Robert Tseng: it’s just been a very tough week. I I don’t. I don’t have real answers from them. So if he comes on, he’s gonna ask questions like, I want to be able to show him something. It seems like you’re not ready to really show him anything, either. So
31 00:07:34.570 ⇒ 00:07:42.380 Robert Tseng: I don’t know. I don’t know. It’s just gonna be a really awkward meeting. He’s just gonna be yelling at me again. So I I, yeah, okay, that’s how I think this is gonna go.
32 00:07:45.370 ⇒ 00:07:46.490 Demilade Agboola: They are.
33 00:08:30.330 ⇒ 00:08:37.480 Robert Tseng: Been just writing some notes down for myself. I I’ll be able to redirect traffic. I’m not gonna leave you guys just like floundering around. But it’s just
34 00:08:39.549 ⇒ 00:08:40.370 Robert Tseng: yeah.
35 00:08:50.300 ⇒ 00:08:52.379 Robert Tseng: I’m just gonna take a couple of minutes for myself.
36 00:11:13.600 ⇒ 00:11:18.145 Robert Tseng: Okay, you know what? Actually, for this setup? I think literally, nothing else matters.
37 00:11:18.940 ⇒ 00:11:20.640 Robert Tseng: Can we work on the stock now.
38 00:11:23.600 ⇒ 00:11:24.345 Demilade Agboola: Okay.
39 00:11:25.090 ⇒ 00:11:31.654 Robert Tseng: We we need to share. We need to be able to share our Qa, we need to reestablish trust in our Qa process.
40 00:11:32.720 ⇒ 00:11:33.999 Robert Tseng: yeah, like, I,
41 00:11:34.760 ⇒ 00:11:38.850 Robert Tseng: I’m I’m not gonna be able to work on anything, anyway, because I feel like, I’ve just
42 00:11:39.350 ⇒ 00:11:46.089 Robert Tseng: yeah, I’ve had. People are just asking me the same thing over and over again. So yeah, like.
43 00:11:47.010 ⇒ 00:11:56.900 Robert Tseng: I don’t know when you’re better playing conversation with Tom is. I still don’t understand. But I don’t know what. Yeah, if you could just pull up your doc and let’s just work through it together like I’m
44 00:11:57.340 ⇒ 00:12:06.919 Robert Tseng: I think I understand the narrative that needs to be told but I may not understand all the different steps, and so I think that’s why it’d be good for us to to do that together.
45 00:12:07.596 ⇒ 00:12:10.999 Robert Tseng: But yeah, why don’t? Why don’t we just do that for for stand up instead.
46 00:12:14.920 ⇒ 00:12:15.895 Demilade Agboola: Okay.
47 00:12:19.460 ⇒ 00:12:20.020 Robert Tseng: Hey!
48 00:12:29.760 ⇒ 00:12:30.550 Tigran Sahakyan: Hey, guys.
49 00:12:32.030 ⇒ 00:12:37.970 Robert Tseng: Hey, T. Grant, we’re not gonna be doing a traditional stand up today. I think, we’re just
50 00:12:38.220 ⇒ 00:12:42.792 Robert Tseng: we’re gonna be working on a a Qa, doc, that
51 00:12:43.720 ⇒ 00:12:51.010 Robert Tseng: yeah. I mean, just because there have been multiple outages this week. And I don’t think anything else really matters like, I feel like
52 00:12:51.110 ⇒ 00:12:54.670 Robert Tseng: that. It’s just this is just what
53 00:12:55.010 ⇒ 00:12:59.339 Robert Tseng: fundamental distrust in in our core data sets. And so
54 00:13:00.810 ⇒ 00:13:07.200 Robert Tseng: yeah, like that, we, we, it’s that we’re we’re just gonna be doing doing like a Co like a co working session right now.
55 00:13:08.230 ⇒ 00:13:09.659 Robert Tseng: can you share the stock with me?
56 00:13:10.710 ⇒ 00:13:13.390 Tigran Sahakyan: Okay, guys. Then I’ll jump off.
57 00:13:14.020 ⇒ 00:13:14.880 Robert Tseng: Okay. Thanks.
58 00:13:14.880 ⇒ 00:13:15.830 Tigran Sahakyan: Okay. Thanks.
59 00:13:38.170 ⇒ 00:13:38.850 Robert Tseng: Okay.
60 00:13:39.330 ⇒ 00:13:46.639 Demilade Agboola: I’m trying to like tweak it so I had the shorter form give it to chat, gpt to like, put it up, but I’m tweaking it to fit.
61 00:13:46.750 ⇒ 00:13:48.750 Demilade Agboola: Get on this use case specifically,
62 00:13:52.360 ⇒ 00:14:00.950 Demilade Agboola: oh, yeah. So the common issues, I can see we could be using still, data. Kids have actions can be failing, and we could have bad joins or filters.
63 00:14:04.820 ⇒ 00:14:05.620 Demilade Agboola: Okay.
64 00:14:10.560 ⇒ 00:14:12.410 Demilade Agboola: in case you wanna hop in.
65 00:15:20.190 ⇒ 00:15:28.520 Robert Tseng: Yeah, I’m just gonna be filling out the stuff in the in the 1st section that I kind of built out. There’s a code, red Doc, that I’ll probably pour into this. We can consolidate.
66 00:15:29.036 ⇒ 00:15:34.469 Robert Tseng: Yeah, I don’t know David wanted how you’re gonna involve away from this. But if you could. Just if you guys could
67 00:15:35.280 ⇒ 00:15:38.940 Robert Tseng: work together on on the other stuff that would be helpful.
68 00:15:39.640 ⇒ 00:15:49.019 Demilade Agboola: Yeah, I just shared our shared documents, so you can always hop in his own thoughts as well.
69 00:16:47.180 ⇒ 00:16:56.650 Robert Tseng: So for the bug that was called out this morning. I think I’m gonna have to put out another code red retro doc, on this. So I want to know if it’s like really the same issue.
70 00:16:57.010 ⇒ 00:17:00.690 Robert Tseng: If I could, I’ll replace your current share.
71 00:17:01.327 ⇒ 00:17:06.220 Robert Tseng: You guys have seen this notion. I’m assuming I’ve like written a lot of stuff in here.
72 00:17:08.310 ⇒ 00:17:17.879 Robert Tseng: right? This was addressing the product fan out issue that was leading to like the problems before. It seems like this morning’s issue is not the same thing. So
73 00:17:19.380 ⇒ 00:17:20.529 Robert Tseng: yeah, I guess.
74 00:17:22.950 ⇒ 00:17:25.049 Robert Tseng: Didn’t want to help me understand? Like.
75 00:17:25.359 ⇒ 00:17:31.279 Robert Tseng: is it different? Like is the root cause. Difference is the are the, you know, like.
76 00:17:31.580 ⇒ 00:17:42.760 Robert Tseng: if you were to adapt this notion, Doc, for the issue that was today like, well, how much overlap is there? Is this? Is this redundant to do the same? To to do another investigation?
77 00:17:43.430 ⇒ 00:17:47.339 Robert Tseng: yeah. Like, I don’t know. Like, what what do you think.
78 00:17:48.901 ⇒ 00:17:53.239 Demilade Agboola: I mean, you could use some of it, but like it’s a different, totally different issue.
79 00:17:54.780 ⇒ 00:18:06.359 Robert Tseng: Okay, as far as like, okay, we had this prevention plan. Things that we said we were going to do seems like we did none of it. And is that why we had the same? We had another issue pop up or
80 00:18:06.870 ⇒ 00:18:10.680 Robert Tseng: bye you get. I’m trying to say it’s like, okay
81 00:18:11.000 ⇒ 00:18:22.279 Robert Tseng: popped up. We put it out. We said we were going to do these things, prevent other issues. And then something else popped up again. And so we’re gonna have to establish whether or not they’re links. Or if there, it’s a separate thing that we have to go in.
82 00:18:22.430 ⇒ 00:18:30.029 Robert Tseng: Also put out, which is fine. I mean, things spring up that are not necessarily in our control. But I just need to be able to communicate that.
83 00:18:31.910 ⇒ 00:18:43.850 Demilade Agboola: Yeah, so this, this is an entirely different issue. So this was to prevent data quality, like, in terms of us getting higher numbers than we need to. What happened today was the numbers. There was a break
84 00:18:43.960 ⇒ 00:18:48.530 Demilade Agboola: in our like Cicd. And so we were not pushing out
85 00:18:48.950 ⇒ 00:18:56.069 Demilade Agboola: like the new numbers, basically. So you were stuck on like older numbers. So that’s a different. That’s an entirely different issue.
86 00:18:57.200 ⇒ 00:18:57.880 Robert Tseng: Okay.
87 00:19:00.660 ⇒ 00:19:03.110 Robert Tseng: Yeah. And so we just re-ran it. And it was fine.
88 00:19:04.315 ⇒ 00:19:14.719 Demilade Agboola: Yeah, I had to like refresh a setting like the setting, the model that was causing the issues. And then everything I started. Running has running has running fine since.
89 00:19:15.370 ⇒ 00:19:18.160 Robert Tseng: Yeah, what? What was the issue with the model? Do we know.
90 00:19:19.296 ⇒ 00:19:31.409 Demilade Agboola: So the the table that provides information for the personalized plan. There’s a new column that’s been added. That wasn’t reflecting.
91 00:19:31.620 ⇒ 00:19:43.899 Demilade Agboola: and so the updates like that, like that new column kind of just broke things. The the table needed to be refreshed in bigquery to be utilized properly.
92 00:19:44.750 ⇒ 00:19:49.349 Robert Tseng: Yeah, we added, that was that like a couple of weeks ago, when we added that.
93 00:19:49.520 ⇒ 00:19:50.380 Demilade Agboola: Yes.
94 00:19:51.790 ⇒ 00:19:52.820 Demilade Agboola: Yeah.
95 00:19:53.790 ⇒ 00:19:56.350 Robert Tseng: Then why did it break today? And not 2 weeks ago?
96 00:19:56.630 ⇒ 00:20:00.210 Demilade Agboola: Yeah, because it wasn’t reflecting properly within bigquery.
97 00:20:00.980 ⇒ 00:20:02.069 Robert Tseng: This entire time.
98 00:20:02.380 ⇒ 00:20:03.100 Demilade Agboola: Yeah.
99 00:20:04.800 ⇒ 00:20:05.540 Robert Tseng: Okay.
100 00:20:06.370 ⇒ 00:20:21.020 Demilade Agboola: Yeah. So I like, so basically, I repointed the the table back to the the what’s it called the the sheet. I was like, okay, like, get that? It was working within. It was working within bigquery.
101 00:20:21.200 ⇒ 00:20:23.339 Demilade Agboola: But Dbt needed to be refreshed.
102 00:20:23.480 ⇒ 00:20:27.089 Demilade Agboola: So that was where the like. The disparity came in.
103 00:20:29.010 ⇒ 00:20:33.359 Robert Tseng: So Dbt just hasn’t been running for for that model for the past 2 weeks.
104 00:20:33.560 ⇒ 00:20:34.589 Demilade Agboola: Oh, no, no, no!
105 00:20:34.710 ⇒ 00:20:36.949 Demilade Agboola: That that was yesterday.
106 00:20:37.240 ⇒ 00:20:41.009 Demilade Agboola: That’s when I was working on the like. The the.
107 00:20:41.780 ⇒ 00:20:47.109 Robert Tseng: So we’re day behind. So it’s not that the pipeline was broken. It’s just that the refresh didn’t happen in time.
108 00:20:48.190 ⇒ 00:20:57.650 Demilade Agboola: I mean the refreshes was happened every hour. So they have been trying to run, but everything that involves the models. Part of it was breaking.
109 00:20:58.800 ⇒ 00:21:02.089 Robert Tseng: Yeah. So I’m saying, that model hasn’t been working for 2 weeks.
110 00:21:02.730 ⇒ 00:21:06.630 Demilade Agboola: No, not 2 weeks. It’s not been working for like 12 h.
111 00:21:07.480 ⇒ 00:21:14.169 Robert Tseng: Okay. And it it broke because you were working on it and did something. And then it just didn’t. It like it didn’t refresh properly.
112 00:21:14.530 ⇒ 00:21:18.140 Demilade Agboola: Yeah. So the yeah, the Github actions would have not been running. Yes.
113 00:21:18.810 ⇒ 00:21:21.200 Demilade Agboola: So that’s what refreshes. Yeah.
114 00:21:22.660 ⇒ 00:21:31.589 Robert Tseng: Well, yeah. So I I guess, like, my question was just like, okay. And it seems like we set it up. You set up something. I just don’t know why it broke yesterday versus 2 like it didn’t break for the past 2 weeks.
115 00:21:32.350 ⇒ 00:21:33.350 Demilade Agboola: Because
116 00:21:33.870 ⇒ 00:21:43.340 Demilade Agboola: the the trying to get the table in to bigquery. I got it into bigquery, the new column in but Dbt. Which was what I did yesterday.
117 00:21:43.480 ⇒ 00:21:50.409 Demilade Agboola: like late yesterday. But Dbt. Needed me to also run something else like run a refresh as well.
118 00:21:51.320 ⇒ 00:21:55.840 Robert Tseng: So you added this column yesterday in bigquery, and then Dbt. The Dbt. Runs fail.
119 00:21:56.310 ⇒ 00:21:57.040 Demilade Agboola: Yeah.
120 00:21:58.790 ⇒ 00:22:04.969 Robert Tseng: Okay. So it is like related to like a recent change that we push. It’s not related to the 2 weeks thing. I think 2 weeks ago.
121 00:22:05.190 ⇒ 00:22:10.389 Demilade Agboola: No, okay, it’s a, it’s a table structure issue.
122 00:22:10.700 ⇒ 00:22:11.640 Demilade Agboola: But yeah.
123 00:22:11.640 ⇒ 00:22:21.539 Robert Tseng: Okay? So yeah, I mean, I, I get it. So Cscd, issue from a new life change that we made it just didn’t kind of flow through all the way from the models. Okay? Understood?
124 00:23:27.650 ⇒ 00:23:32.729 Robert Tseng: Okay? So I mean, I know we’re all we’re looking at the same thing. So let’s see.
125 00:23:34.140 ⇒ 00:23:35.850 Robert Tseng: it’s do anything.
126 00:23:39.455 ⇒ 00:23:40.560 Robert Tseng: Okay?
127 00:26:17.950 ⇒ 00:26:21.627 Robert Tseng: Okay? So I mean, you guys can keep working on the stock as
128 00:26:23.880 ⇒ 00:26:32.783 Robert Tseng: But yeah, I guess my outline here. So I need to be able to communicate what has been our process up to this point. How we’ve been queuing our models.
129 00:26:33.640 ⇒ 00:26:38.740 Robert Tseng: you know, and then, like what new changes were introduced? Yeah.
130 00:26:39.210 ⇒ 00:26:43.930 Demilade Agboola: So normally, we queue it through staging. So we have a staging portion.
131 00:26:44.780 ⇒ 00:26:47.679 Demilade Agboola: And then that’s how we
132 00:26:48.920 ⇒ 00:26:51.119 Demilade Agboola: test before we release the production.
133 00:26:52.010 ⇒ 00:27:03.648 Demilade Agboola: However, this particular model like this particular is not, it’s a sheet. So it’s it’s production. There’s no staging for it. So it’s the raw data. So
134 00:28:54.635 ⇒ 00:28:55.470 Josh : Yo.
135 00:28:56.830 ⇒ 00:28:57.480 Robert Tseng: Hey!
136 00:28:58.010 ⇒ 00:28:59.470 Josh : What’s up? Guys?
137 00:29:00.651 ⇒ 00:29:06.980 Robert Tseng: So we’re just we’re just working through just purely a Qa. Qa, Doc, on just.
138 00:29:09.190 ⇒ 00:29:12.429 Josh : To avoid the scenarios from today.
139 00:29:13.731 ⇒ 00:29:42.200 Robert Tseng: No, I mean this current invest. The current scenario has already been done. I think what I want to get out of this exercise is I want to be to communicate to the team. What is our Qa. Process been before? What risks do we already know we’re already there? But we just didn’t deal with. And then what new risks were introduced? The past week past 2 weeks that kind of contributed to the 2 errors that we found one already reported out on. That was just the fan out bug. And then this morning. That’s not a logical error. That’s just
140 00:29:43.095 ⇒ 00:29:48.709 Robert Tseng: I guess we in in patching something from yesterday we
141 00:29:49.010 ⇒ 00:29:55.340 Robert Tseng: made a change that didn’t push all the way through our Cdcd pipeline so
142 00:29:55.774 ⇒ 00:30:08.179 Robert Tseng: it didn’t make it through the Dbt checks, and then we didn’t catch it. This I mean, it’s not that we didn’t catch it this morning. It’s just that nobody is really on call to go and deal with it when it runs at like 3 or 4 am. In the morning. So
143 00:30:09.560 ⇒ 00:30:11.750 Robert Tseng: yeah, so I don’t.
144 00:30:12.182 ⇒ 00:30:24.710 Josh : My biggest thing right? Guys with the Qa. Is really simple. I don’t consider it a real bug until someone on our staff, including myself find something wrong.
145 00:30:25.000 ⇒ 00:30:41.950 Josh : If you guys find it before us like it’s like, Hey, it’s just part of the process. But, like, if our team, like myself, Adam Cutter mitash anyone, identify something before you guys, that’s where it’s like a code red, right? I mean, like, that’s where something like, Hey, you guys should know
146 00:30:42.220 ⇒ 00:30:46.099 Josh : that like, it’s gonna end up reflecting really badly on me.
147 00:30:46.300 ⇒ 00:31:02.740 Josh : because I’m taking ownership over this. So for me, I just really care about hey, you guys can make like, move fast, break stuff. It’s totally cool, but like, if something is broken as long as you find it, 1st identify it, call it out and fix it then no harm, no foul, cool.
148 00:31:03.690 ⇒ 00:31:31.266 Robert Tseng: Yeah, no, I hear you. I think. So there’s there’s like, you know, there’s a couple of adjustments that we need to make. I mean internal view reviewing the internal review process is not the problem here, and we we are able to deal with any like modeling or logical errors. I think. I mean this one. It’s like, okay, we approved the Pr yesterday. It looks fine testing. But then it just never made it to production. So that’s just our our platform, not fully doing what it’s supposed to. And
149 00:31:32.140 ⇒ 00:31:41.919 Robert Tseng: yeah, we’re we’re bringing. We’re setting up a new thing today to try to help with us. Help us with that. Yeah, I mean, we’re just. We’re just trying to to make this this smoother.
150 00:31:44.980 ⇒ 00:31:46.940 Josh : Got it. Okay, cool.
151 00:31:49.510 ⇒ 00:31:54.590 Robert Tseng: Okay, yeah. I mean, we’ll work around. But like, I guess.
152 00:31:56.370 ⇒ 00:31:59.079 Robert Tseng: Anyway, thank God, them a lot of that wish. So
153 00:32:00.810 ⇒ 00:32:15.170 Robert Tseng: yeah, this is the narrative I want to tell. If you guys could just review this, make sure I didn’t miss anything else. I think we already knew that there were some risks. And so I mean, I’ll write out one or 2 other things. I think these are the new things that we need to. We need to that were that were brought in.
154 00:32:16.230 ⇒ 00:32:19.999 Robert Tseng: I think we just got lucky it didn’t break earlier. So I think
155 00:32:20.880 ⇒ 00:32:23.949 Robert Tseng: with what we’re gonna set up with meta plane. And then
156 00:32:24.280 ⇒ 00:32:30.619 Robert Tseng: we need to make sure that this checklist is I got this. This needs to be. This needs to be tied up. So
157 00:32:33.650 ⇒ 00:32:34.230 Awaish Kumar: Whether.
158 00:32:34.230 ⇒ 00:32:35.530 Robert Tseng: Yeah, I guess.
159 00:32:37.530 ⇒ 00:32:40.210 Awaish Kumar: Include slack presentation.
160 00:32:41.960 ⇒ 00:32:42.760 Robert Tseng: What was that?
161 00:32:46.730 ⇒ 00:32:51.560 Awaish Kumar: Ci, CD, like we should include slack notification. It’s it’s a
162 00:32:52.590 ⇒ 00:32:55.830 Awaish Kumar: bit of action fails. We should know right that DVD.
163 00:32:55.830 ⇒ 00:32:56.190 Robert Tseng: PI.
164 00:32:56.655 ⇒ 00:32:58.049 Awaish Kumar: Has not been.
165 00:32:59.140 ⇒ 00:32:59.700 Awaish Kumar: We can.
166 00:32:59.700 ⇒ 00:33:22.599 Robert Tseng: I know we have a lot of the the slack notification channel. It is very noisy. I think we we can’t get around with. Yeah, we need to set up an on call rotation. Which I mean, we will do that on top of the slack notifications. But yeah, I don’t. I don’t. I don’t know if the slack notifications is enough, because the reality is we don’t have 24 7 coverage like that’s just how it is. So
167 00:33:22.700 ⇒ 00:33:32.659 Robert Tseng: I mean, this runs during a time, this, these these models run during a time when nobody is really online. Maybe Dave, a lot is the 1st to see it, because he’s the 1st to start.
168 00:33:33.321 ⇒ 00:33:35.280 Robert Tseng: But yeah, like, I think
169 00:33:35.530 ⇒ 00:33:55.470 Robert Tseng: there’s a i mean, like Mattesh saw something at 5 am. This morning like, sorry I’m not awake at that time, so if we have to, just maybe we run it earlier or like we do it, so that if anything does break in our in the Ci CD that we notice it before anybody else does. I think that’s that’s Josh’s point. So maybe we could talk about
170 00:33:55.700 ⇒ 00:33:58.409 Robert Tseng: any adjustments we need to make, so that we can
171 00:33:58.650 ⇒ 00:34:03.459 Robert Tseng: make sure somebody’s eyes are on it. As soon as it runs.
172 00:34:04.270 ⇒ 00:34:07.470 Robert Tseng: that. That seems to be like a basic thing that we need to shift
173 00:34:09.130 ⇒ 00:34:19.840 Robert Tseng: right? We can’t be running these at 5 Am. If we’re none of us are looking at it until, like, set 6 or 7 am. Because otherwise somebody else is gonna see it before us. If it doesn’t, if it doesn’t work.
174 00:34:23.389 ⇒ 00:34:31.210 Robert Tseng: But yeah, we can. We can make that decision later. I also don’t think it makes sense to do it too early. Because if we’re only refreshing this daily, then
175 00:34:31.840 ⇒ 00:34:42.790 Robert Tseng: well, yeah, sometimes people are okay with looking at data that’s fresh up until yesterday. But other people want to see it for today. So I think, we have to think about how we can
176 00:34:43.139 ⇒ 00:34:49.250 Robert Tseng: neat everyone’s needs. I, man.
177 00:34:49.780 ⇒ 00:34:50.300 Awaish Kumar: Yeah, but.
178 00:34:50.300 ⇒ 00:34:54.880 Robert Tseng: You know, we need go ahead.
179 00:34:56.989 ⇒ 00:34:59.619 Awaish Kumar: Yeah, right now. It’s it’s set up to run every hour.
180 00:34:59.969 ⇒ 00:35:01.139 Awaish Kumar: But that’s the.
181 00:35:01.980 ⇒ 00:35:13.069 Robert Tseng: Yeah, for our models are. Oh, right. But I’m talking about the Pdfs, right? Like the people aren’t querying like the reports every hour, like. I think the Pdf. Is what Natasha is pointing out and being like, Hey, something’s off
182 00:35:13.200 ⇒ 00:35:17.199 Robert Tseng: so, and that runs at what we set it up to do. 4 or 5 am. In the morning, I think.
183 00:35:17.980 ⇒ 00:35:23.060 Demilade Agboola: Yeah, that’s that’s the time. So about 5 Am. New Eastern time.
184 00:35:23.820 ⇒ 00:35:29.290 Demilade Agboola: Between 5 and 30, most most of the Pdfs are sent out.
185 00:35:29.960 ⇒ 00:35:42.280 Robert Tseng: Okay, then how about this? We’ll just say snapshot data. We’ll we’ll we’ll have an earlier cut off time. So we make sure that when it runs. Whoever’s the 1st on is we’re gonna be able to see it. So snapshots need to be
186 00:35:43.430 ⇒ 00:35:44.560 Robert Tseng: refresh.
187 00:35:44.800 ⇒ 00:35:54.669 Robert Tseng: That’s I’m just. I’m just gonna put numbers there, we can adjust it. But like, let’s let’s say we’re I mean, like, I don’t believe anybody’s looking at 7 at 2 am. E et
188 00:35:56.610 ⇒ 00:36:06.460 Robert Tseng: and then live well, or I guess, like tea bottles refresh, you know, hourly right?
189 00:36:07.130 ⇒ 00:36:11.189 Robert Tseng: So that way we will. Then we can set the. We can set the
190 00:36:12.440 ⇒ 00:36:16.480 Robert Tseng: like the expectation that. Okay, this data will will.
191 00:36:16.880 ⇒ 00:36:17.610 Robert Tseng: Thanks.
192 00:36:17.850 ⇒ 00:36:28.410 Robert Tseng: Yeah. It’s for, like, we direct people to only be looking at it up until yesterday. And then we’re also more accountable to like. Okay? Well, if if anything does.
193 00:36:28.550 ⇒ 00:36:33.370 Robert Tseng: Rick, we’ll be able to address it. During during that time. Do you understand? Does that make sense.
194 00:36:36.040 ⇒ 00:36:36.750 Awaish Kumar: Yes.
195 00:36:37.260 ⇒ 00:36:37.850 Robert Tseng: Okay.
196 00:36:39.040 ⇒ 00:36:43.570 Robert Tseng: Only promise a clean data
197 00:36:43.800 ⇒ 00:36:51.100 Robert Tseng: to yesterday. Hey? We can always refresh hourly. Rely on slack notifications.
198 00:36:53.620 ⇒ 00:37:00.330 Robert Tseng: Non call location to deal with.
199 00:37:02.770 ⇒ 00:37:17.149 Robert Tseng: Okay, so that’s that’s like, kind of our escalation process. So I’m just gonna call this escalation heuristics. And then I know you guys are bringing in Meta plane. I still fully understand. So I don’t know if I’ll be able to join that call when you when you talk about it, but
200 00:37:17.633 ⇒ 00:37:43.360 Robert Tseng: like I think it was before you join the call. The issue that I brought it is, people are always going to be checking our work, using, like the source of truth, that they think it doesn’t matter. I think the test that we set up on the Dbt side. Everything is a relative check. I kind of talk about it here, where we’re using like different tests to like signal when data is like 10% plus away from like what we expect it to be right. But you know, that’s not how
201 00:37:43.510 ⇒ 00:37:44.440 Robert Tseng: you know
202 00:37:44.860 ⇒ 00:37:52.979 Robert Tseng: Adam or Natasha is. Gonna look at it. They’re just gonna go in or cut it. They’re gonna go to bask, and they’re just gonna export something. And
203 00:37:53.780 ⇒ 00:38:18.530 Robert Tseng: whether or not that that’s like that. We just have to accept that. That’s how people are gonna queue. That’s how they’re gonna Qa or work. So we need to anticipate that and also have that set up so that we’re always consistently. We’re always answering the same question, why, our data is different from what they’re getting from the past platform. But if we know that already that we should, we should have that answer, can. It’s like, okay, maybe part of the slack notification that comes out is
204 00:38:18.530 ⇒ 00:38:31.330 Robert Tseng: okay. We have that basket is showing a hundred. We’re showing 95 as a reminder. Our bottle is not including dead dead like it just should be very clear, so that
205 00:38:31.570 ⇒ 00:38:39.360 Robert Tseng: if somebody does. If we anticipate somebody challenging the data in that way, we can. Just we can just answer that very quickly.
206 00:38:41.190 ⇒ 00:38:42.180 Robert Tseng: Is that fair.
207 00:38:44.010 ⇒ 00:38:44.780 Awaish Kumar: Yep.
208 00:38:44.970 ⇒ 00:38:45.720 Demilade Agboola: Yeah, that’s fair.
209 00:38:45.720 ⇒ 00:38:48.550 Robert Tseng: Okay, okay? So
210 00:38:49.470 ⇒ 00:38:57.850 Robert Tseng: I mean, I can kind of define, like, I mean, I can kind of summarize with the most common objections I hear? Just because I’ve gotten a bunch of objections this week.
211 00:38:57.990 ⇒ 00:38:59.460 Robert Tseng: Okay, so
212 00:39:04.240 ⇒ 00:39:06.535 Robert Tseng: a tool load?
213 00:39:10.420 ⇒ 00:39:11.370 Robert Tseng: that we want
214 00:39:20.070 ⇒ 00:39:21.100 Robert Tseng: other.
215 00:39:21.950 ⇒ 00:39:23.839 Robert Tseng: That’s good makes sense.
216 00:39:27.280 ⇒ 00:39:39.209 Robert Tseng: Okay. So the bass we’re gonna get J, 4, and there I guess the test will probably also at north be right. So for bass, this is what we’re gonna look at for orders plus revenue.
217 00:39:39.500 ⇒ 00:39:42.317 Robert Tseng: you know. They’re gonna look at ad spend
218 00:39:43.020 ⇒ 00:39:49.489 Robert Tseng: and attribution here, and and then they’re gonna look at like you know, attributed revenue or like channel
219 00:39:49.840 ⇒ 00:39:53.219 Robert Tseng: channel and and revenue.
220 00:39:53.480 ⇒ 00:39:58.549 Robert Tseng: So yeah, I think, we’re gonna yeah. Okay.
221 00:40:04.620 ⇒ 00:40:05.760 Robert Tseng: For now.
222 00:40:06.540 ⇒ 00:40:25.980 Robert Tseng: I mean, we definitely have access to these, and we well, we we literally get these in the warehouse, I mean, ask even to like, but maybe not like I think this could even just be like a daily platform export that we will just just go and like manually do like that doesn’t matter to me like I I think that’s just what we’re gonna have to do for now.
223 00:40:26.210 ⇒ 00:40:28.710 Robert Tseng: So maybe this will be
224 00:40:30.540 ⇒ 00:40:34.180 Robert Tseng: This will be manual. But then I think the rest will be fine.
225 00:40:34.620 ⇒ 00:40:35.370 Robert Tseng: Okay,
226 00:40:37.500 ⇒ 00:40:43.286 Robert Tseng: yeah, I mean, I can. I can go and rework this. I’ll I’ll summarize it. I’ll have gpt clean it up. But
227 00:40:44.150 ⇒ 00:40:45.030 Demilade Agboola: Quick question.
228 00:40:45.030 ⇒ 00:40:46.640 Robert Tseng: Help your guys. Go ahead.
229 00:40:46.880 ⇒ 00:40:53.500 Demilade Agboola: But just also thinking, can we also, since we’re trying to? So we do a lot of transformations within Basque.
230 00:40:53.870 ⇒ 00:40:59.550 Demilade Agboola: I’m sorry. Not within within Dbt, and then we push it out to bigquery, and then we have it into
231 00:41:00.234 ⇒ 00:41:18.290 Demilade Agboola: tableau. We could also create a tableau dash where we like literally have, like a yesterday or last 30 day comparison, or just do every day where we compare the raw numbers directly, so like after transformation, pre-transformation, like the raw numbers and how they match up.
232 00:41:18.450 ⇒ 00:41:25.480 Demilade Agboola: so that allows us to get a very quick view of what’s going on
233 00:41:26.350 ⇒ 00:41:33.289 Demilade Agboola: so like we could also have that as a Pdf that we send our internal channel or something. So we can just basically see, hey.
234 00:41:33.960 ⇒ 00:41:41.960 Demilade Agboola: the broad numbers in Basque yesterday are this so like before any of this transformations for anything. But these are our numbers right now.
235 00:41:45.410 ⇒ 00:41:52.059 Robert Tseng: Okay? Yeah. I mean, that’s that’s pretty much what I was trying to say with with this part.
236 00:41:52.060 ⇒ 00:41:56.260 Demilade Agboola: Yeah, yeah, I know. That’s I’m trying to see like the execution of it. That’s what I’m trying.
237 00:41:56.260 ⇒ 00:42:02.355 Robert Tseng: Okay, yeah, I mean, that’s what that’s what you I want you guys to do. I’m just giving you the the vision here. So
238 00:42:02.940 ⇒ 00:42:10.850 Robert Tseng: I mean, I don’t know if this is the dashboard that you had in mind, but even just like how products have like their uptime kind of like being like
239 00:42:11.480 ⇒ 00:42:12.350 Robert Tseng: I shoot.
240 00:42:12.520 ⇒ 00:42:16.750 Robert Tseng: I don’t know if this makes sense something that’s easy for the team to understand.
241 00:42:17.210 ⇒ 00:42:35.350 Robert Tseng: Maybe this is too much work, like, maybe we just kind of settle for this for now. But for all the different dashboards we have, like uptime issues right? And then, like just kind of able to give some some sort of, you know, updates here and like with a metric of like, how we’re doing
242 00:42:36.800 ⇒ 00:42:40.350 Robert Tseng: like, I think this would this, this would be helpful to the team like I
243 00:42:40.610 ⇒ 00:42:45.910 Robert Tseng: I don’t know if they would believe it. I think they would still be looking at the data. But so maybe this is not
244 00:42:46.110 ⇒ 00:42:51.245 Robert Tseng: fully like the best kind of view. But
245 00:42:52.000 ⇒ 00:42:53.959 Robert Tseng: I just want people to know that, like.
246 00:42:56.390 ⇒ 00:43:08.939 Robert Tseng: yeah, the perception is not, oh, yeah, like T, every something’s breaking every week. It’s like, No, actually, we’re up. Our uptime is 99.9 5% kind of thing. Like, I I want to to make sure that
247 00:43:09.140 ⇒ 00:43:17.650 Robert Tseng: the perception of our team’s reliability is grounded in some, you know, like this, this is the
248 00:43:18.180 ⇒ 00:43:28.660 Robert Tseng: yeah like that. It’s it’s probably true like I. I believe that our uptime is probably 99% plus. But I I just we don’t have a way to prove that right now with this.
249 00:43:28.980 ⇒ 00:43:37.859 Robert Tseng: It’s just he say he, he says, she said, like hearsay stuff that people are just throwing accusations around which are frustrating for us. Right now.
250 00:43:38.250 ⇒ 00:43:48.011 Robert Tseng: so I don’t know if in tableau we can spit something up like this. I thought elementary was supposed to be our solution to do that I don’t know if meta plane does this. But
251 00:43:49.210 ⇒ 00:43:51.769 Robert Tseng: yeah, I guess. Does this make sense.
252 00:43:53.891 ⇒ 00:43:56.640 Demilade Agboola: Yes, it does. We can try and see how to set it up.
253 00:43:57.210 ⇒ 00:43:57.490 Demilade Agboola: Okay.
254 00:43:58.050 ⇒ 00:44:04.580 Robert Tseng: I’ll just take a screenshot of that, and then maybe we’ll just get deliverables.
255 00:44:05.590 ⇒ 00:44:09.850 Robert Tseng: Okay, alright. So okay, if you guys could.
256 00:44:10.060 ⇒ 00:44:14.989 Robert Tseng: you know, when when you guys are meeting with Utam later on this stuff like, yeah, please.
257 00:44:17.350 ⇒ 00:44:24.699 Robert Tseng: yeah, like, I probably by end of day, I’d like to send just like a document towards that to them. So
258 00:44:25.080 ⇒ 00:44:28.919 Robert Tseng: if we could just have like a clear go ahead and like what we’re gonna do. And
259 00:44:29.380 ⇒ 00:44:30.869 Robert Tseng: like, yeah. Like.
260 00:44:31.000 ⇒ 00:44:35.459 Robert Tseng: if you need to include me on that call like, just mark me as optional, I’ll try to join
261 00:44:36.610 ⇒ 00:44:44.870 Robert Tseng: But yeah, I think this is this is our kick in the butt that we really need to get our Qa. Process cleaned up and and well documented.
262 00:44:50.660 ⇒ 00:44:52.200 Demilade Agboola: Okay. Sounds good.
263 00:44:52.520 ⇒ 00:44:53.210 Robert Tseng: We’re good.
264 00:44:54.660 ⇒ 00:44:56.080 Robert Tseng: Alright, thanks, guys.