Meeting Title: Brainforge Interview w- Zoran Date: 2026-05-06 Meeting participants: Mi, Zoran Selinger
WEBVTT
1 00:00:45.600 ⇒ 00:00:46.310 Zoran Selinger: Hi, it’s.
2 00:00:46.970 ⇒ 00:00:47.980 Mi: How are you?
3 00:00:48.820 ⇒ 00:00:50.670 Zoran Selinger: Yeah, I’m good, how are you?
4 00:00:50.900 ⇒ 00:00:53.669 Mi: Good, good! Thank you for your time today!
5 00:00:53.670 ⇒ 00:00:55.770 Zoran Selinger: Sure, sure, no problem, no problem.
6 00:00:55.980 ⇒ 00:00:58.180 Mi: Where… where are you based at?
7 00:00:58.360 ⇒ 00:00:59.699 Zoran Selinger: I’m in Croatia.
8 00:01:00.220 ⇒ 00:01:01.980 Mi: Oh, that’s nice.
9 00:01:02.560 ⇒ 00:01:03.030 Zoran Selinger: Yeah.
10 00:01:03.950 ⇒ 00:01:13.230 Zoran Selinger: Had an early start today, so been… been online, been working for… 10 hours already today.
11 00:01:13.230 ⇒ 00:01:16.480 Mi: Wow. Wow.
12 00:01:16.480 ⇒ 00:01:19.889 Zoran Selinger: I woke up kind of early, so I started immediately, and yeah.
13 00:01:20.010 ⇒ 00:01:20.570 Zoran Selinger: What?
14 00:01:20.570 ⇒ 00:01:22.249 Mi: What’s… what’s the time there?
15 00:01:23.150 ⇒ 00:01:26.719 Zoran Selinger: So now it’s, it’s, it’s 3.30 PM.
16 00:01:27.850 ⇒ 00:01:30.439 Zoran Selinger: Yeah, but I had a very, very early start.
17 00:01:30.910 ⇒ 00:01:31.350 Mi: Wow.
18 00:01:31.350 ⇒ 00:01:33.680 Zoran Selinger: station time, so…
19 00:01:33.680 ⇒ 00:01:36.589 Mi: I hope you can get some rest after this call.
20 00:01:36.590 ⇒ 00:01:41.950 Zoran Selinger: Yeah, yeah, I’ll be able to, I’ll be able to, no worries, no worries. How are you doing?
21 00:01:42.000 ⇒ 00:01:43.850 Mi: Good, good, good, yes.
22 00:01:43.850 ⇒ 00:01:44.900 Zoran Selinger: Where are you based?
23 00:01:44.900 ⇒ 00:01:46.260 Mi: New York.
24 00:01:46.570 ⇒ 00:01:51.600 Zoran Selinger: Oh, you’re in New York. Yeah, so I understand you… you met Robert recently, right?
25 00:01:51.600 ⇒ 00:01:56.720 Mi: Yes, we chatted on the phone, I think last week or something, yeah.
26 00:01:57.810 ⇒ 00:02:01.720 Mi: Yeah, so I learned a lot of, from him.
27 00:02:01.720 ⇒ 00:02:07.450 Zoran Selinger: Okay. Yeah, yeah, you can, you can. Yeah, so,
28 00:02:08.310 ⇒ 00:02:12.300 Zoran Selinger: So, I guess… yeah, so this is a technical part.
29 00:02:12.450 ⇒ 00:02:26.710 Zoran Selinger: Okay. This second interview is more of a technical part, but I’ll still like to hear more about, like, the intro, and let me, like, let me understand a little bit about what you’ve been doing so far.
30 00:02:26.890 ⇒ 00:02:31.149 Mi: Okay, of course. I’ll keep it short.
31 00:02:31.350 ⇒ 00:02:37.640 Mi: So, I have been working in marketing analytics, Oh my…
32 00:02:37.640 ⇒ 00:02:39.999 Zoran Selinger: I’m just closing the window keys are in front.
33 00:02:40.210 ⇒ 00:02:42.120 Mi: Okay.
34 00:02:42.420 ⇒ 00:02:46.710 Zoran Selinger: These are just up below my window, so they’re loud.
35 00:02:46.960 ⇒ 00:02:48.599 Mi: Yeah, I understand.
36 00:02:48.600 ⇒ 00:02:50.139 Zoran Selinger: Yeah, I’m working from home.
37 00:02:50.140 ⇒ 00:03:01.140 Mi: Yeah, okay, so I’ll start over. I’ve been spending my career on marketing analytics, since, 2013, something, or 2014.
38 00:03:01.220 ⇒ 00:03:23.509 Mi: I have experience with the different industries, like gaming, healthcare, and the tech, and the fintech, recently fintech. Having experience with B2C, B2B, I’ve been always supporting the growth slash marketing team at the… I also work, partner with,
39 00:03:23.510 ⇒ 00:03:30.860 Mi: product team, revenue team, and a sales team before, but I decided to stay with marketing.
40 00:03:30.860 ⇒ 00:03:31.570 Zoran Selinger: Okay.
41 00:03:31.570 ⇒ 00:03:33.820 Mi: That’s what I like the best, I think.
42 00:03:33.920 ⇒ 00:03:42.760 Mi: I had… earlier in my career, I was, focusing on technical, building
43 00:03:42.840 ⇒ 00:04:03.359 Mi: technical skills, like building models, dashboards, learning how to build a database, everything. And… but recently, I’ve been kind of transferring, transitioning into a role that connects, like, I would say, connected business, data insights to actual business impact
44 00:04:03.440 ⇒ 00:04:26.559 Mi: because along the years, I realized sometimes the data team, like us, we build some models, but the business doesn’t know how to use it. And sometimes the other team would feel like, they might be shy to ask to reach out for the data people for questions they have, things they need.
45 00:04:26.560 ⇒ 00:04:30.009 Mi: So I decided to become the bridge.
46 00:04:30.010 ⇒ 00:04:32.399 Mi: In between two teams.
47 00:04:32.400 ⇒ 00:04:49.660 Mi: So, so in my recent roles, I’m more, like, doing a lot of work, communication work, gathering everyone together. I try to understand what the marketing team needs, then translate it to the data team, and help the data team build, the projects.
48 00:04:49.660 ⇒ 00:04:57.989 Mi: Then, by my most recent job, with Camoon, it’s a… Camoon is a neobank, a super small Series A.
49 00:04:58.340 ⇒ 00:05:08.559 Mi: with, like, 30-ish people, so I was the person to do almost everything, even from scratch, connecting, API, you know, everything.
50 00:05:08.560 ⇒ 00:05:14.299 Zoran Selinger: I mean, yeah, when you’re in teams that small, then you are real, you’re playing multiple roles.
51 00:05:14.460 ⇒ 00:05:15.579 Zoran Selinger: At that point, that’s…
52 00:05:15.580 ⇒ 00:05:16.320 Mi: Yeah.
53 00:05:16.320 ⇒ 00:05:19.929 Zoran Selinger: Normal, but you also learn… you learn a lot, and you… you become.
54 00:05:19.930 ⇒ 00:05:20.690 Mi: Yes.
55 00:05:20.690 ⇒ 00:05:21.540 Zoran Selinger: that way.
56 00:05:21.720 ⇒ 00:05:33.150 Mi: Yeah, I love learning new things, so I was happy there. Yeah, thanks to AI, though, otherwise I wouldn’t be able to, you know, set up and get everything set up.
57 00:05:33.170 ⇒ 00:05:36.300 Zoran Selinger: I mean, of course, I mean, yeah…
58 00:05:37.080 ⇒ 00:05:45.220 Zoran Selinger: Yeah. You… yeah, and Robert probably told you that The point of, of,
59 00:05:46.460 ⇒ 00:06:02.139 Zoran Selinger: the company and the way we want to work, we want to become AI-native in basically everything we do. And yeah, we just started also building this, like, the Martech, MarTech side to.
60 00:06:03.180 ⇒ 00:06:11.060 Zoran Selinger: to be AI native. We, essentially, we want to get to a point where we are… we are…
61 00:06:11.680 ⇒ 00:06:14.950 Zoran Selinger: More in the commands.
62 00:06:15.870 ⇒ 00:06:22.260 Zoran Selinger: Inside a model, then we are in the interface of a marketing tool.
63 00:06:22.470 ⇒ 00:06:28.360 Zoran Selinger: Which is, yeah, this is something we strive for and what we work on.
64 00:06:28.410 ⇒ 00:06:44.680 Zoran Selinger: So that’s… there’s gonna be plenty of that here, and that’s… that’s, that’s accept… not only accepted, but very, very encouraged, in… in Brainforce. So that’s all… all good. AI is… is… is…
65 00:06:44.680 ⇒ 00:06:55.710 Zoran Selinger: fine and well with us, of course. Yeah, so you said you worked multiple roles there, so apart from… so you were setting up what, exactly?
66 00:06:56.270 ⇒ 00:07:12.100 Mi: I used Fivetrend to set up an API connection to some media channels like Apple Search. Then, also, I set up a website event tracking using Segment.
67 00:07:12.100 ⇒ 00:07:26.389 Mi: Excellent. Yeah, and connect data… we used Snowflake, so there was no direct connection between segment and Snowflake, so we used a tool called Groove, and I was able to connect everything and move data to Snowflake.
68 00:07:26.390 ⇒ 00:07:32.879 Mi: So we can build… so I use the data to build, attribution model, yeah.
69 00:07:32.880 ⇒ 00:07:43.929 Mi: I was like, yeah, I didn’t have engineer help for, like, a couple months, because no one on the team had experience with the web… web… with web.
70 00:07:43.930 ⇒ 00:07:44.500 Zoran Selinger: Yeah.
71 00:07:44.610 ⇒ 00:07:47.240 Mi: So… Oh, so…
72 00:07:47.240 ⇒ 00:07:53.790 Zoran Selinger: Yeah, I mean, like I said, that happens in small teams, that’s… Yeah. That’s exactly…
73 00:07:54.050 ⇒ 00:08:14.190 Mi: Yeah, thanks to Claude. So, yeah, then I also learned how to, manage a website. We use this tool called Webflow, to host our, company website, so I was able to get… learn how to use that tool, so I can build
74 00:08:14.190 ⇒ 00:08:16.210 Mi: Websites now.
75 00:08:16.210 ⇒ 00:08:18.219 Zoran Selinger: Sure, sure. I mean,
76 00:08:19.700 ⇒ 00:08:26.829 Zoran Selinger: If you do come on board, you’re probably gonna work on a client that has a website built on Webflow.
77 00:08:27.130 ⇒ 00:08:27.980 Mi: Okay.
78 00:08:28.010 ⇒ 00:08:32.989 Zoran Selinger: Yeah, because we already have, like, an idea of where we would like to.
79 00:08:33.309 ⇒ 00:08:45.709 Zoran Selinger: for you to jump in, so it’s gonna be a Webflow-based website. I mean, it’s not a huge deal, but it’s good to have an understanding of it, of course.
80 00:08:47.470 ⇒ 00:08:51.239 Zoran Selinger: Yeah, so you, you mentioned, you mentioned segment, right?
81 00:08:51.240 ⇒ 00:08:51.900 Mi: Yes.
82 00:08:52.340 ⇒ 00:08:56.039 Zoran Selinger: so, have you had any…
83 00:08:57.350 ⇒ 00:09:01.779 Zoran Selinger: So, what are some of the technical challenges that you, you…
84 00:09:01.900 ⇒ 00:09:14.469 Zoran Selinger: you kind of stumbled upon there. Have you… did you need to do a lot of debugging? How did that go? How did you plan? What needs to be tracked and all that? Tell me a little bit more about that.
85 00:09:15.200 ⇒ 00:09:31.809 Mi: So, honestly, when I got started, I did… I have… I only had a very basic idea of what I need. I need the campaign names, I need to be able to track all the campaign, who landed on the page from where, right? But since I didn’t get any help.
86 00:09:31.880 ⇒ 00:09:47.240 Mi: I had some vague, very vague memory from my previous company who used Segment before, so I know what I need, but with all the tools, I just asked a crawl, here are the tools I need.
87 00:09:47.240 ⇒ 00:09:56.990 Mi: how would I set up events tracking? Then Claude gave me some… also very, like, very good idea. It does not tell me, step by step what to.
88 00:09:56.990 ⇒ 00:09:57.690 Zoran Selinger: Yeah.
89 00:09:57.690 ⇒ 00:09:58.760 Mi: I had to…
90 00:09:58.900 ⇒ 00:10:09.169 Mi: contact segment. We also use Apps Fire, and I also have to contact Google Help, because we use Google Tag Manager to set.
91 00:10:09.170 ⇒ 00:10:09.880 Zoran Selinger: beholden.
92 00:10:09.880 ⇒ 00:10:29.439 Mi: Now to talk to a Webflow, team. I really had no idea of any of those things before, because I didn’t… I never… I was never the person to work directly with those partners. So then… so each of them taught me something.
93 00:10:30.500 ⇒ 00:10:39.289 Mi: Then with AI’s help, I was able to put everything together. Then once I put it together, I realized, how do I get… I don’t know how to get the data into Snowflake.
94 00:10:39.740 ⇒ 00:10:44.929 Mi: So… so then I talked, reached out to, Snowflake. I don’t know.
95 00:10:44.930 ⇒ 00:10:45.620 Zoran Selinger: I knew that.
96 00:10:45.620 ⇒ 00:11:10.000 Mi: Also, I talked to our internal engineer first, but they never… they had never done this before, so I had to talk to Segment, Snowflake, and also Gru, then finally figure out, here are, like, a few ways we can try to get the data into a Snowflake. So, we just tried one by one. The first few, I don’t even remember what we did.
97 00:11:10.000 ⇒ 00:11:28.959 Mi: But the… I think the third way, or the fourth way, whatever worked out, we were able to see the data in segment as a raw data, so we were able to track everything. So that’s where I stopped. But there are… so events, what I decided to track, of course, campaign.
98 00:11:29.200 ⇒ 00:11:42.320 Mi: creatives, the ad group, and the user’s user ID identifier. The best one we could use was, this, anonymous ID, using segment anonymous ID.
99 00:11:42.320 ⇒ 00:11:43.660 Zoran Selinger: segment’s anonymous ID, right?
100 00:11:43.660 ⇒ 00:11:50.939 Mi: Yeah, which, which, Segment said it’s their, kind of, like, their session ID, but expires, like, once a year also, so it’s.
101 00:11:50.940 ⇒ 00:11:51.659 Zoran Selinger: Yeah, yeah.
102 00:11:51.660 ⇒ 00:11:55.349 Mi: Yeah, so I decided to use that.
103 00:11:55.350 ⇒ 00:11:56.489 Zoran Selinger: They do their best.
104 00:11:56.490 ⇒ 00:12:08.570 Mi: Yeah, I know. I don’t remember when my old company actually used, segment to generate unique user ID, which worked out for us, too.
105 00:12:08.570 ⇒ 00:12:18.989 Mi: So that’s basically how everything worked. And once we had that, I worked with, then. By the end of summer, we hired a web…
106 00:12:18.990 ⇒ 00:12:37.319 Mi: developer, a front-end engineer who had a web development experience, so he… he actually helped me to connect a Northeim’s ID to our internal user ID. Yeah, if someone signs up or signs in, we were able to connect those IDs together. That’s the part I.
107 00:12:37.320 ⇒ 00:12:41.989 Zoran Selinger: How did… where did you do… do that? When did you set that up? Was this,
108 00:12:42.150 ⇒ 00:12:50.490 Zoran Selinger: a custom HTML tag in Google Tag Manager, or if you use, like, a segment feature?
109 00:12:51.430 ⇒ 00:12:52.340 Zoran Selinger: Upwork.
110 00:12:52.590 ⇒ 00:12:57.340 Mi: The way they explained to me was some database data, so we have our own.
111 00:12:57.340 ⇒ 00:13:06.579 Zoran Selinger: Sure, they were grabbing this data from a database, but where… so, did you only… Connect those two.
112 00:13:06.990 ⇒ 00:13:15.829 Zoran Selinger: in the… in the database, or somewhere before. So, in Google Tag Manager, or in Segment. Where did that merging happen?
113 00:13:16.110 ⇒ 00:13:16.780 Zoran Selinger: Merge…
114 00:13:16.780 ⇒ 00:13:23.010 Mi: Merging, I think, happened in the database. I’m not quite sure about that.
115 00:13:23.570 ⇒ 00:13:32.069 Mi: But when I set up the tracking, I set up… I did use Google Tag Manager to write some script, JavaScript, then implement.
116 00:13:32.070 ⇒ 00:13:35.679 Zoran Selinger: Basically, what I’m asking you is, did they push
117 00:13:35.980 ⇒ 00:13:39.800 Zoran Selinger: your internal user ID to the client.
118 00:13:39.980 ⇒ 00:13:46.159 Zoran Selinger: to the website, or did they save the segment’s anonymous ID into the database?
119 00:13:46.490 ⇒ 00:13:50.470 Zoran Selinger: Did it go in one direction or the other, if you remember?
120 00:13:50.630 ⇒ 00:13:57.500 Mi: I’m pretty sure there are both ways, but I… I don’t want to lie, but I don’t know…
121 00:13:57.500 ⇒ 00:13:58.020 Zoran Selinger: Yeah, sure.
122 00:13:58.060 ⇒ 00:14:05.219 Mi: exactly where it happened, but I know in the segment, we have data going both ways.
123 00:14:05.570 ⇒ 00:14:06.230 Zoran Selinger: So…
124 00:14:06.230 ⇒ 00:14:10.919 Mi: Yeah, that’s, that’s what I understood, yeah. Sure, sure.
125 00:14:10.920 ⇒ 00:14:20.230 Zoran Selinger: So, tell me, why did you use segment? Is that… was that tool… did they already have segment in place, or did you make that decision?
126 00:14:20.500 ⇒ 00:14:37.859 Mi: That’s the… we already had the tool in place, and I… I know from my previous company, we used Segment to set up… to track website events until the engineer developed the internal tool to do that. So…
127 00:14:38.670 ⇒ 00:14:55.340 Mi: I know engineers can do that, but it would take much longer time. There is always a trade-off of the speed and the perfection, right? Segment can do 90%… if Segment can do 90% of the job, I’d rather set it up so we can start collecting data.
128 00:14:55.340 ⇒ 00:15:09.540 Mi: Then later, if the engineer team decide to build their internal tool to track web events, then that’d be great. And if we have older data, we can merge to later system. So that’s, to me.
129 00:15:09.540 ⇒ 00:15:21.000 Mi: collecting data is, very important, and I would love to start collecting data as soon as possible, because I was also working on attribution model, I do need the data.
130 00:15:21.000 ⇒ 00:15:22.250 Zoran Selinger: Of course, of course. Yes.
131 00:15:22.250 ⇒ 00:15:28.229 Mi: So, our, Sokamu is the mobile app,
132 00:15:28.730 ⇒ 00:15:43.299 Mi: before me, the team was only looking at Apps Fire, the MMP data, so starting everything, entire attribution flow from install. If people don’t have install, if they’re just website visiting, they sign up on the web.
133 00:15:43.690 ⇒ 00:15:51.299 Mi: basically, we’re losing track on those users, so that’s why I want to start collecting data as soon as possible, yeah.
134 00:15:51.300 ⇒ 00:16:00.250 Zoran Selinger: Sure, sure. And in that company where, where you had segments set up, and then they transitioned.
135 00:16:00.540 ⇒ 00:16:06.640 Zoran Selinger: into… into developers-led, like, okay, explicit tracking.
136 00:16:06.750 ⇒ 00:16:15.819 Zoran Selinger: So, going off-segment into, like, a custom solution, do you remember what, what the decision was? Why, why that decision was,
137 00:16:16.660 ⇒ 00:16:21.810 Zoran Selinger: Was made. To go from segment into a custom tracking solution.
138 00:16:22.200 ⇒ 00:16:25.359 Zoran Selinger: Built entirely by… by developers.
139 00:16:25.590 ⇒ 00:16:27.839 Mi: Oh, you mean, like, my previous company?
140 00:16:27.840 ⇒ 00:16:28.550 Zoran Selinger: Yeah, yeah.
141 00:16:29.690 ⇒ 00:16:31.859 Zoran Selinger: Do you remember? Do you remember, maybe?
142 00:16:31.860 ⇒ 00:16:33.410 Mi: That was.
143 00:16:33.410 ⇒ 00:16:34.130 Zoran Selinger: remember.
144 00:16:34.130 ⇒ 00:16:50.749 Mi: Yeah, yeah, I remember the biggest reason was Segment was charging a lot of money to do the entire ecosystem work. So we decided to… so first of all, we decided to set up… start web
145 00:16:50.750 ⇒ 00:16:56.829 Mi: sign-up flow. So, just to test out if a website would work out for us.
146 00:16:56.830 ⇒ 00:17:14.320 Mi: Then later, we decided to, that was… the entire switch was after… after I… happened after I left, but I still have to tackle with my old co-workers, so I know they decided to switch completely, so they’re saving… they’re saving, like, really a lot of money.
147 00:17:14.319 ⇒ 00:17:20.820 Mi: And the… I know that company, the engineers, Blue Trust.
148 00:17:20.819 ⇒ 00:17:31.180 Mi: back-end data more, or database data. I’m not an engineer, but I only know the difference between front-end, back-end, and database.
149 00:17:31.180 ⇒ 00:17:46.910 Mi: So, the team decided to stay with back-end and database data instead of trusting, front-end data. So, but, but anyway, I think the most important thing… reason is for money, to save money.
150 00:17:46.910 ⇒ 00:17:49.840 Zoran Selinger: Perfectly legitimate reason, of course.
151 00:17:51.800 ⇒ 00:18:01.010 Zoran Selinger: Yeah, sometimes, sometimes the way it works, sometimes depending just on the number of events, sometimes depending on how much we pull or push.
152 00:18:01.130 ⇒ 00:18:12.439 Zoran Selinger: It’s just the pricing structure doesn’t work for that particular client, it’s not the right tool, and you have to do something else. That is fine, of course. So you…
153 00:18:13.030 ⇒ 00:18:22.169 Zoran Selinger: So, you were working on… on this, on this tracking, and you probably heard about, the server-side tracking.
154 00:18:22.380 ⇒ 00:18:23.780 Mi: The what? Sorry.
155 00:18:23.780 ⇒ 00:18:24.900 Zoran Selinger: server side.
156 00:18:24.900 ⇒ 00:18:26.190 Mi: A server’s side, yeah.
157 00:18:26.190 ⇒ 00:18:29.309 Zoran Selinger: Yeah, so did you, did you hear about that concept?
158 00:18:29.550 ⇒ 00:18:39.919 Zoran Selinger: And how would that differ from, just having, let’s say, tags, like a Facebook pixel, in a Google Tag Manager?
159 00:18:40.080 ⇒ 00:18:41.440 Zoran Selinger: container.
160 00:18:42.010 ⇒ 00:18:48.930 Mi: I only heard of, server-to-server that’s basically connected to, I mean.
161 00:18:49.040 ⇒ 00:18:56.859 Mi: to service APIs. I don’t know, I don’t know the correct terms, but together.
162 00:18:57.510 ⇒ 00:19:04.330 Mi: But… everything else, like, more engineering side, I’m not quite sure, yeah.
163 00:19:04.330 ⇒ 00:19:08.929 Zoran Selinger: Yeah, yeah. I mean, I have to… I have to ask you this more technical question, because
164 00:19:08.930 ⇒ 00:19:28.559 Zoran Selinger: Simply, that’s the… that’s the goal of this particular round, just to see where you are technically. Yeah, so don’t… don’t worry if you haven’t encountered any of these concepts, this is… that’s perfectly fine. So, you, when you had this Google Tag Manager container.
165 00:19:29.280 ⇒ 00:19:35.099 Zoran Selinger: You did not have an equivalent server-side Google Time Manager container. You only had one, right?
166 00:19:35.990 ⇒ 00:19:38.699 Mi: I only had one.
167 00:19:38.700 ⇒ 00:19:41.640 Zoran Selinger: One container, okay. Yeah. And…
168 00:19:41.850 ⇒ 00:19:57.719 Zoran Selinger: Were there many events? Which events were you tracking, in particular? Do you remember? Which one… which ones are the first ones you set up, and which ones were not as important when you were setting that up a little bit later, for example?
169 00:19:57.720 ⇒ 00:20:08.670 Mi: So, yes. So, that would go with the attribution model. Our attribution model was the last touch-based attribution.
170 00:20:08.950 ⇒ 00:20:09.510 Zoran Selinger: Okay.
171 00:20:09.510 ⇒ 00:20:18.910 Mi: That’s because most of the events, most of the install data we’re getting from Apps Fire are only last touch.
172 00:20:20.480 ⇒ 00:20:40.289 Mi: Okay, AppSuite only gives us the last touch before install, so just to match up, I was looking at the… all the website events I could track, but comparing the… I do compare web events to app install events, by… and by timestamp, then take the latest one.
173 00:20:41.540 ⇒ 00:20:58.259 Zoran Selinger: Right, right. Have you experimented with any other attribution models? Just to see how the value changes, how you assign the value towards each channel, advertising channel in particular? Have you explored that at all?
174 00:20:59.030 ⇒ 00:21:14.429 Mi: And Camus, I only took a look at the website because we… the only app install attribution we could get from Apps Fire was the last touch. So I was able to look at the website tracking data. I would say…
175 00:21:14.580 ⇒ 00:21:28.180 Mi: I would say, like, 90-ish percent of people would have only one touchpoint on websites, so, by that time, I didn’t think it was worth the effort to write.
176 00:21:28.180 ⇒ 00:21:28.680 Zoran Selinger: Yeah, yeah.
177 00:21:28.680 ⇒ 00:21:36.079 Mi: batch attribution. But at, in the current, my previous company, we…
178 00:21:36.260 ⇒ 00:21:46.209 Mi: did take a look at the… because we have all the touchbooks. Okay, currently, before, we were doing running, tool app.
179 00:21:46.310 ⇒ 00:22:04.600 Mi: campaigns. But later, we decided to switch almost 99% campaign to website, a web-to-app campaign, so we know exactly where people come from. And but still, I would say 80% of users had only one touchpoint. The other 20%, we took a look.
180 00:22:04.600 ⇒ 00:22:18.289 Mi: I would say if we run a multi-touch attribution, that would just give us, like, maybe about… I remember it was, like, a 6% lift. I did the math before, because,
181 00:22:18.290 ⇒ 00:22:21.419 Mi: Also, a lot of users, would have, like.
182 00:22:22.090 ⇒ 00:22:29.310 Mi: Google, or Google search, and non-paid search, and some random, some other
183 00:22:29.620 ⇒ 00:22:52.649 Mi: direct, I would say direct, visit. So, a lot of them don’t really have more than one touchpoint. So, by… by that time, we decided to… not to run a multi-touch attribution. But what we did was, instead of just, attribute one person to one channel, right, instead of just last touch or first touch of
184 00:22:52.800 ⇒ 00:22:58.019 Mi: forgot which one we went with. We decided to go with conversion event.
185 00:22:58.020 ⇒ 00:23:20.819 Mi: So, I have 3 conversion events if I installed sign up and set up direct deposit. So, I have 3 events. I want to see which one, which channel, which marketing events is driving me to convert, to install, which one is driving us to sign up, which one is driving me to, set up direct deposit. So, in that case.
186 00:23:20.820 ⇒ 00:23:40.349 Mi: some of us would have multi… different channels, so I might have three totally different channels, but I could also have only one Facebook channel for all the events. That helped a lot, and we also add in a lot more other data sources, like survey.
187 00:23:40.350 ⇒ 00:23:41.480 Mi: TV…
188 00:23:41.510 ⇒ 00:23:58.819 Mi: TV, we were able to get IP address, level data, from some provider. I know some other provider might not want to give you that data, so, we were lucky to have that, and which… then we combined everything together, and also promotion code. Promote code once you redeem a promotion.
189 00:23:58.940 ⇒ 00:24:07.849 Mi: We know where you are from, right? So, once we combined everything together, we were able to improve the attribution percentage by, like, 30%.
190 00:24:07.850 ⇒ 00:24:19.079 Zoran Selinger: how do you do the identities teaching in those cases? So you have the website, you have your segment anonymous ID, then you have your actual,
191 00:24:19.330 ⇒ 00:24:19.690 Mi: User.
192 00:24:19.690 ⇒ 00:24:36.979 Zoran Selinger: like a user ID from the bank itself, then you… you have an IP address eventually, so how do you reconcile identities? How do you… how do you link web interaction to a… to a TV IP address?
193 00:24:37.450 ⇒ 00:24:49.100 Mi: We’re just, so IP-wise, we’re just matching them, if we could match, but IP was not our first match key. What we did was that when you sign up.
194 00:24:49.140 ⇒ 00:25:01.769 Mi: For the app, you have to, you can sign up online, or you can download the app. Then, once… when you sign up, you’re required to input your phone number.
195 00:25:01.880 ⇒ 00:25:04.439 Mi: So, and the email address.
196 00:25:04.440 ⇒ 00:25:27.939 Mi: then we use that as the match key, we combo as a match key. We also… also, for install-first users, Apps Fire would give us a… an Apps Fire ID, which they say they already deduped. They say it’s a unique identifier we could use. So we also use that ID, and also segment would… would give us, some sort of,
197 00:25:27.990 ⇒ 00:25:38.269 Mi: user ID. They call it user ID. I… honestly, I don’t know what I do that is. Then we just put everything together, and we would dedupe.
198 00:25:38.440 ⇒ 00:25:49.439 Mi: Based on all the phone number and the other information. And we do understand a phone number could be used by multiple people, so we just…
199 00:25:49.440 ⇒ 00:25:52.170 Zoran Selinger: Now, more rarely than ever.
200 00:25:52.740 ⇒ 00:26:10.239 Mi: Yeah. And there were also one person… no one person could have multiple phone numbers. So, in that case, for… for the first case of phone numbers used by multiple people, we just take the one used that could… we could identify the last.
201 00:26:10.440 ⇒ 00:26:12.890 Mi: So, the last time used.
202 00:26:13.050 ⇒ 00:26:21.040 Mi: by the person. Because we also asked for email address. I know email address is not perfect, but, it’s a…
203 00:26:21.090 ⇒ 00:26:33.380 Mi: better than nothing. Then, for users, who have multiple phone numbers, in that case, we ask for email address and, what else? Apps FireID, so we
204 00:26:33.780 ⇒ 00:26:52.099 Mi: their first time install. If they install another time, within 30 days, Apps Fire would use the same ID. I think after 30 days, if they install again, they would get a different AppSfire ID. So, it was a lot of a mess.
205 00:26:52.100 ⇒ 00:26:56.280 Mi: And, I did have to do a lot of cleaning.
206 00:26:56.280 ⇒ 00:27:00.529 Mi: data cleaning work. It wasn’t… wasn’t… was never perfect.
207 00:27:00.530 ⇒ 00:27:06.460 Mi: But at the end of the day, that’s only for install and the sign-up.
208 00:27:06.460 ⇒ 00:27:12.210 Mi: Sign… when then we have a… if someone wants to set up a direct deposit.
209 00:27:12.210 ⇒ 00:27:27.450 Mi: or we’re user verified, we go through KYC process, we know their SSM, we know their address, so that wouldn’t be a problem. The only issue we would face was into the sign-up and the install.
210 00:27:27.530 ⇒ 00:27:38.429 Mi: Phase, but… but it’s… most… most of the time, I would say 90% of the time, so it’s okay, so, yeah. Yeah.
211 00:27:38.430 ⇒ 00:27:40.350 Zoran Selinger: Okay, okay, cool. Yeah.
212 00:27:40.480 ⇒ 00:27:43.740 Zoran Selinger: So, have you run any incrementality tests?
213 00:27:44.170 ⇒ 00:27:46.300 Mi: Yes, yes, like, all the time.
214 00:27:46.300 ⇒ 00:27:48.469 Zoran Selinger: Tell me a little bit about that.
215 00:27:48.680 ⇒ 00:27:50.750 Mi: So…
216 00:27:51.860 ⇒ 00:28:03.760 Mi: Where do I get started? Whenever we run a campaign, I’ve been pushing… so for Come Home, my previous company, my last company.
217 00:28:03.760 ⇒ 00:28:16.100 Mi: People didn’t know what incrementality is. They kind of have some idea, but they think… always think… people always overthink about it. Incrementality sounds super fancy and complicated, but…
218 00:28:16.770 ⇒ 00:28:29.920 Mi: just… I just have to let people know, even A-B testing, it’s an incrementality testing, right? You don’t have to overcomplicate it. So, over there, instead, I tried to convince
219 00:28:30.300 ⇒ 00:28:43.120 Mi: convince the team to run a test before, we decide to dump all the money on a single channel, or campaign, or some creative testing, because
220 00:28:43.360 ⇒ 00:28:57.240 Mi: we… we don’t… otherwise, we don’t know, right? It could be, like, wasted of money. I know people have smart ideas, but not every time’s gonna… it works out. So sometimes we run cra…
221 00:28:57.240 ⇒ 00:29:04.550 Mi: incrementality test, like, we ran some geo… not geo, I would call it impression tests.
222 00:29:04.550 ⇒ 00:29:15.059 Mi: Which, when, when Meta told… Meta rep told us that some of their clients, realized that just sending people,
223 00:29:15.670 ⇒ 00:29:21.120 Mi: multiple impressions with… within the short-term time.
224 00:29:21.120 ⇒ 00:29:43.979 Mi: works better than sending people one impression per month for a longer time. So we decided to test it out. We don’t know if it’s true. That, in the end of the day, that didn’t work out to us. It didn’t… it really didn’t work. But so the people… the team was a little bit upset about it, but I’m like, we’re not just trying to prove that the meta route.
225 00:29:43.980 ⇒ 00:29:46.139 Zoran Selinger: Were you the one to present the results?
226 00:29:46.140 ⇒ 00:29:47.179 Mi: Yeah, yeah, yeah.
227 00:29:47.180 ⇒ 00:29:56.640 Zoran Selinger: How did you do that? I guess this is a challenge, this is always a challenging bit, right? First of all, let me ask you, is it okay if you run a little bit over?
228 00:29:57.110 ⇒ 00:29:59.510 Mi: Oh yeah, I love that, I love that, yes, I love that.
229 00:29:59.510 ⇒ 00:30:00.610 Zoran Selinger: Cool,
230 00:30:01.590 ⇒ 00:30:09.839 Zoran Selinger: So that’s a challenging one, okay? Sometimes clients get really, really into something, and they’re sure it’s gonna work, and then…
231 00:30:10.000 ⇒ 00:30:13.049 Zoran Selinger: And then, you know, it’s not.
232 00:30:13.150 ⇒ 00:30:19.889 Zoran Selinger: You have the data, and now you have to communicate it, right? Yeah. In a nice way, and to explain
233 00:30:20.650 ⇒ 00:30:23.100 Zoran Selinger: This is how things are gonna go, like…
234 00:30:23.360 ⇒ 00:30:29.269 Zoran Selinger: this is not going to work, we have the data. How do you… how did you do that in this case?
235 00:30:29.490 ⇒ 00:30:35.820 Mi: So… Well, this is what I’ve done, which I found to work out for me, is…
236 00:30:36.070 ⇒ 00:30:42.610 Mi: Not to call them stupid. No, I’m kidding. I feel like, so I just,
237 00:30:43.260 ⇒ 00:31:00.669 Mi: present the result, say, here’s what we talked about before, here’s the goal which we tried to reach. But this is the test design we talk about would be… this kind of thing I would share before the test.
238 00:31:00.740 ⇒ 00:31:07.630 Mi: So, here’s how much money we spent, how long we ran the test, here’s the results. When the…
239 00:31:07.630 ⇒ 00:31:32.090 Mi: But the results does not look good, because… I wouldn’t say the result doesn’t look good. Here’s the results, and the idea we tested didn’t work out. SIMs didn’t work out. Here are some hypotheses. So, number one, this is… this idea doesn’t work. People just don’t like… users just don’t like it. Or, number one… number two is that there are some other factors that might be impacting this
240 00:31:32.090 ⇒ 00:31:32.730 Mi: test.
241 00:31:32.730 ⇒ 00:31:43.729 Mi: It really depends on the case, right? Sometimes we know something just went… just probably went wrong in the middle, then we have to examine our data.
242 00:31:43.730 ⇒ 00:31:54.540 Mi: if I… if we could find the cause of… I’ll let the team know. Here’s some… sometimes, we, by accident, we mailed the control… trunk control group.
243 00:31:54.540 ⇒ 00:32:07.959 Mi: Instead of Facebook campaign, right? But for some reason, we just send them some email. We’ll then, in this case, we’ll run the test again. If not, if that’s not the case, maybe I’ll just say.
244 00:32:08.500 ⇒ 00:32:26.399 Mi: it just didn’t work out. Maybe we could try something, test something else. Maybe the idea we are testing right here is too big, too vague, or maybe we can just, from a different angle, we can test again. Something like that. I…
245 00:32:26.530 ⇒ 00:32:34.940 Mi: the… the thing is, I don’t want to discourage people to share me their ideas. They’re the biggest.
246 00:32:34.940 ⇒ 00:32:36.020 Zoran Selinger: Of course, of course.
247 00:32:36.020 ⇒ 00:32:55.820 Mi: they know the… they know the business the best, so I… a lot of time, I learn new things from them all the time, so I feel like I would encourage them to still give me test ideas. If they don’t dis… they don’t agree, with the results, I would ask why.
248 00:32:55.820 ⇒ 00:33:11.570 Mi: for… because from their experience, their, instinct, instinct, here’s a… they think it will work, and I need to understand why they think it will work, so maybe the test design setup wasn’t correct. So…
249 00:33:11.570 ⇒ 00:33:11.990 Zoran Selinger: Yep.
250 00:33:11.990 ⇒ 00:33:15.730 Mi: So now, first of all, I have to go through the result, I have to go…
251 00:33:15.730 ⇒ 00:33:36.569 Mi: go through the data to make sure everything is correct, and then before I present the results to them. If it really doesn’t work out, I would encourage them to give me new ideas. We test something new. So far, I haven’t had any hard… no one has ever given me a hard time. People are very happy to learn new things.
252 00:33:36.600 ⇒ 00:33:43.359 Mi: And also, I don’t want them to spend a lot of money on the test from the beginning. Anyway…
253 00:33:43.360 ⇒ 00:33:44.950 Zoran Selinger: Yeah, yeah, so that, that helped.
254 00:33:46.180 ⇒ 00:33:50.369 Zoran Selinger: That helps if you haven’t spent a lot of money. It doesn’t work, it doesn’t work.
255 00:33:50.370 ⇒ 00:33:50.750 Mi: Yeah.
256 00:33:50.750 ⇒ 00:33:52.440 Zoran Selinger: Haven’t spent a lot of money.
257 00:33:53.920 ⇒ 00:33:57.180 Zoran Selinger: That’s fine, that’s fine.
258 00:33:57.180 ⇒ 00:34:02.260 Mi: So, what… I have… I also have some questions for you.
259 00:34:02.260 ⇒ 00:34:07.599 Zoran Selinger: Sure, sure, and yeah, that’s exactly what I wanted to say. That’s exactly what I wanted to say.
260 00:34:07.600 ⇒ 00:34:21.690 Mi: Yeah, Robert has mentioned that you’ve been working on some projects. Do you mind giving me some example of a project or a client you work with? Then I can get a better idea of how everything…
261 00:34:22.040 ⇒ 00:34:22.940 Zoran Selinger: Sure
262 00:34:23.489 ⇒ 00:34:41.989 Zoran Selinger: Yeah, so this is… so the client that you would probably be working on first is a client in telehealth, so this is, you… are you familiar with GLP-1 medication? That’s Ozempic and things like that, very, very popular right now.
263 00:34:42.360 ⇒ 00:34:43.540 Zoran Selinger: Right?
264 00:34:43.540 ⇒ 00:34:46.269 Mi: Oh, the one to help people lose weight?
265 00:34:46.270 ⇒ 00:34:46.870 Zoran Selinger: Yep.
266 00:34:46.870 ⇒ 00:34:48.589 Mi: Okay, I see.
267 00:34:48.590 ⇒ 00:34:53.730 Zoran Selinger: So it’s mostly that there are some other… so this is telehealth, right?
268 00:34:53.840 ⇒ 00:35:03.889 Zoran Selinger: Essentially essentially, you need… basically, it’s a virtual doctor’s appointment, so the floor…
269 00:35:04.200 ⇒ 00:35:09.230 Zoran Selinger: quite a lot of, quite a lot of questions, things like that. And yeah.
270 00:35:09.950 ⇒ 00:35:23.149 Zoran Selinger: some steps are kind of high friction as well. At some point, you have to, like, give a photo, stuff like that. So it can be… it can be difficult for people to go through.
271 00:35:23.160 ⇒ 00:35:31.410 Zoran Selinger: Through that process. So this is… that’s… and once they submit, they pay, the actual doctor.
272 00:35:31.410 ⇒ 00:35:45.829 Zoran Selinger: is looking over their application and approves the prescription, right? And only then they can go into an actual pharmacy or get the drug in the mail, right?
273 00:35:45.890 ⇒ 00:35:47.800 Zoran Selinger: So, that’s the…
274 00:35:47.990 ⇒ 00:36:03.269 Zoran Selinger: that’s the… the challenge. There’s quite a few of them, right? There’s… the competition is pretty high, right now, and they’re very, very popular. So that’s… that’s basically the game. We have a client that is…
275 00:36:03.720 ⇒ 00:36:10.009 Zoran Selinger: trying to be a multi-channel client, but they’re still heavily, heavily dependent on Google Ads.
276 00:36:11.130 ⇒ 00:36:14.130 Zoran Selinger: They are battling with, like, affiliate partners.
277 00:36:14.240 ⇒ 00:36:17.360 Zoran Selinger: Quite a bit, because of,
278 00:36:18.140 ⇒ 00:36:22.079 Zoran Selinger: They’re simply not converting as well as some other.
279 00:36:23.430 ⇒ 00:36:36.679 Zoran Selinger: And essentially, affiliate partners are trying to hustle them a lot, just to kind of, to… to give them, like, higher fees and all those things.
280 00:36:37.070 ⇒ 00:36:44.470 Zoran Selinger: because of allegedly lower conversion rates, right? So, yeah, they have a… they have this stack of…
281 00:36:45.030 ⇒ 00:36:49.500 Zoran Selinger: They have this stack of, you know, Google Tag Manager, which are
282 00:36:50.120 ⇒ 00:36:57.849 Zoran Selinger: now they’re thinking about moving away from it, and actually doing embedded tracking in…
283 00:36:58.090 ⇒ 00:37:14.180 Zoran Selinger: in the website. So, segment, mixed panel are in there, and of course, a warehouse that is sitting in BigQuery. So that is essentially the stack that, we would, need to manage.
284 00:37:15.030 ⇒ 00:37:18.320 Zoran Selinger: One is this technical side, of course.
285 00:37:18.590 ⇒ 00:37:24.059 Zoran Selinger: Where we actually have to understand what’s in segment, what’s in mixed panel.
286 00:37:24.090 ⇒ 00:37:30.630 Zoran Selinger: how data’s flowing, identities, teaching, all of those things. But yeah, there’s also a…
287 00:37:30.630 ⇒ 00:37:47.410 Zoran Selinger: quite… quite a big, reporting piece. And, you know, strategizing for… for… for the future of… or, okay, where do they need support? Like, there’s things, like, that are still outstanding. For example, they don’t have, like, like, symptoms cohorts.
288 00:37:47.550 ⇒ 00:37:55.510 Zoran Selinger: So, we want to understand… we want to understand, for example, how do people, after they, after they,
289 00:37:56.620 ⇒ 00:38:12.990 Zoran Selinger: reported some symptoms. How do they… what does that mean for their current treatment? Or do they cancel treatments right away, or do they finish the treatment that they paid for, and then… and then do not extend the treatment? Stuff like that, so…
290 00:38:12.990 ⇒ 00:38:21.359 Zoran Selinger: There’s obviously those, those higher level, higher level things. The goal is, of course.
291 00:38:21.700 ⇒ 00:38:26.729 Zoran Selinger: And this is very important to Brainforge, is this AI native piece.
292 00:38:27.000 ⇒ 00:38:37.049 Zoran Selinger: And that is, we want to be super, super efficient, using AI as much as possible. We are running, like, automated workflows through agents.
293 00:38:37.870 ⇒ 00:38:47.089 Zoran Selinger: but will, you know, do specific, specific things. So building and planning those tools is going to be a very important part of the job as well.
294 00:38:47.230 ⇒ 00:38:48.000 Zoran Selinger: Okay.
295 00:38:48.150 ⇒ 00:38:53.770 Zoran Selinger: So how do I get… Your… your favorite…
296 00:38:53.870 ⇒ 00:39:10.739 Zoran Selinger: AI stack to, you know, we need a new event. How do I now run a command in a model, in an agent, that will update the tracking plan automatically?
297 00:39:11.570 ⇒ 00:39:17.660 Zoran Selinger: that will… created triggers, variables, tags in GTM that will
298 00:39:17.880 ⇒ 00:39:36.730 Zoran Selinger: That will create all the necessary changes in segment, that will make changes to MixPanel if need be, amplitude, whatever, right? We wanna… this is what we wanna build. This is what we’re striving to do. We wanna become AI native, so this is… that basically means
299 00:39:36.880 ⇒ 00:39:47.450 Zoran Selinger: We want to try to get to a position where you’re not logging into segment interface, but we run a command in an agent that does it, right?
300 00:39:48.260 ⇒ 00:39:51.740 Zoran Selinger: So that’s… that’s basically the job.
301 00:39:51.990 ⇒ 00:39:56.260 Zoran Selinger: I think, plan for… for you,
302 00:39:56.570 ⇒ 00:39:58.740 Zoran Selinger: If… if you get on board.
303 00:39:59.640 ⇒ 00:40:02.040 Zoran Selinger: To see if you can be client-facing.
304 00:40:04.850 ⇒ 00:40:23.019 Zoran Selinger: So, yeah, we’ll discuss with the team, yeah. So, there is another piece. So, I have a take-home assignment for you. So, part of the technical is the take-home assignment. I did go a little bit overboard.
305 00:40:23.340 ⇒ 00:40:28.000 Zoran Selinger: So, it’s, it’s… There’s quite a few pieces to it.
306 00:40:29.700 ⇒ 00:40:36.339 Zoran Selinger: It’ll probably… Take 5 hours for it to do, but we don’t want it… we don’t want that.
307 00:40:37.210 ⇒ 00:40:40.180 Zoran Selinger: We want you to only invest 3.
308 00:40:40.400 ⇒ 00:40:41.370 Zoran Selinger: Okay. Okay.
309 00:40:41.670 ⇒ 00:40:43.750 Zoran Selinger: I’ll invest 3 hours.
310 00:40:44.310 ⇒ 00:40:47.499 Zoran Selinger: the first A, A portion.
311 00:40:47.850 ⇒ 00:40:57.679 Zoran Selinger: of the… of the assignment is the most important, so focus there. Whatever else you can get to, that’s just a nice-to-have. That’s fine.
312 00:40:57.790 ⇒ 00:41:04.840 Zoran Selinger: But the A portion, Focus on that, and see what you can do. See what you can do there.
313 00:41:06.910 ⇒ 00:41:24.719 Zoran Selinger: this is also, this is, like, this is my first interview, by the way, here. So I’m, like, I’m not super, super smooth with, like, interviewing, and so I’ll see, maybe I missed something, I might send you an email, we’ll see. So I’ll send you the assignment in an email.
314 00:41:24.980 ⇒ 00:41:33.499 Zoran Selinger: So we’ll, we’ll see, we’ll see what, what happens, what happens from, from there.
315 00:41:33.850 ⇒ 00:41:39.970 Mi: This is a great interview. I like this kind of mutual communication.
316 00:41:39.970 ⇒ 00:41:41.490 Zoran Selinger: So, having fallen.
317 00:41:41.490 ⇒ 00:41:42.200 Mi: You know?
318 00:41:42.200 ⇒ 00:41:48.469 Zoran Selinger: I mean, hopefully, we work together, so this is how we want to talk every day, essentially.
319 00:41:48.470 ⇒ 00:41:50.100 Mi: Yeah, that’d be great.
320 00:41:50.100 ⇒ 00:41:51.730 Zoran Selinger: Obviously, yeah, I wanna,
321 00:41:52.370 ⇒ 00:42:05.089 Zoran Selinger: we want to at least simulate, and this, I mean, it’s not a simulation, it’s a genuine conversation between two people that kind of work in the same vertical, right?
322 00:42:05.090 ⇒ 00:42:13.509 Zoran Selinger: Yeah. On the very similar positions. And yeah, if you come on board, we are very likely to work very close together.
323 00:42:14.170 ⇒ 00:42:16.500 Mi: Awesome, awesome, awesome.
324 00:42:16.500 ⇒ 00:42:21.570 Zoran Selinger: That’s the idea. Yes. Okay, well, thank you for your time.
325 00:42:21.570 ⇒ 00:42:23.270 Mi: Thank you!
326 00:42:23.270 ⇒ 00:42:27.769 Zoran Selinger: Yeah, so you can expect an email from me a little bit later.
327 00:42:27.990 ⇒ 00:42:33.210 Zoran Selinger: And yeah, I mean, Kaila will claw us in on the next steps.
328 00:42:33.390 ⇒ 00:42:35.809 Mi: Okay, thank you so much, so nice meeting you.
329 00:42:35.810 ⇒ 00:42:37.120 Zoran Selinger: Have a good rest of the day.
330 00:42:37.430 ⇒ 00:42:42.019 Mi: You too. Get some rest. Bye-bye. Bye.