Meeting Title: BF: Technical Screen & Final Interview Date: 2025-11-21 Meeting participants: Ujjwal Yadav, Samuel Roberts, Uttam Kumaran


WEBVTT

1 00:02:39.320 00:02:40.770 Ujjwal Yadav: Hey, Samuel.

2 00:02:40.910 00:02:42.250 Samuel Roberts: Hello, how are you?

3 00:02:44.160 00:02:45.350 Samuel Roberts: Can you hear me alright?

4 00:02:45.580 00:02:47.740 Ujjwal Yadav: Yeah, I can hear you. Can you see me?

5 00:02:48.370 00:02:50.170 Samuel Roberts: No, I can’t see you, but I can hear you.

6 00:02:50.540 00:02:51.959 Ujjwal Yadav: Okay, let me…

7 00:02:55.440 00:02:57.890 Samuel Roberts: There you are. How are you?

8 00:03:01.380 00:03:09.750 Samuel Roberts: Hello. Yeah. Yes, hi. Yeah, I think we’re just waiting on a couple more people, maybe? So, we’ll give it a minute or two here.

9 00:03:13.800 00:03:19.640 Samuel Roberts: Let me see if anyone said anything… Nope. Okay.

10 00:03:22.860 00:03:23.640 Samuel Roberts: Ugh.

11 00:03:26.120 00:03:28.039 Samuel Roberts: Now, where are you located?

12 00:03:28.540 00:03:29.809 Ujjwal Yadav: I’m from India.

13 00:03:30.320 00:03:41.820 Samuel Roberts: India? Okay. I was gonna say, the time difference with so many people, I’m never sure who’s… we have people all over the place, so… I’m not sure, it’s towards the end of my day here now, so I wasn’t sure where… what part of the day you’re in now, but it must be… yeah.

14 00:03:44.530 00:03:54.159 Ujjwal Yadav: I think it’s around, like, around 12.5 hours, I guess, so, if you’re living in San Francisco, but if in New York, it’s around, I guess, 10, 10.5 hours?

15 00:03:54.810 00:03:57.699 Samuel Roberts: Yeah, yeah, yeah, I’m on the… I’m in the Eastern time zone, so…

16 00:03:57.860 00:03:58.470 Ujjwal Yadav: Okay.

17 00:04:00.640 00:04:05.460 Samuel Roberts: Great. Yeah, I’ll see people are joining… haven’t…

18 00:04:09.030 00:04:10.190 Samuel Roberts: Let’s see…

19 00:04:14.190 00:04:14.970 Ujjwal Yadav: buddy.

20 00:04:19.829 00:04:26.259 Samuel Roberts: Alright, maybe it’s just us then, okay.

21 00:04:27.010 00:04:29.030 Ujjwal Yadav: Okay, it’s awesome, no worries.

22 00:04:29.930 00:04:31.749 Samuel Roberts: That’s fine. Okay.

23 00:04:33.590 00:04:35.010 Samuel Roberts: So, I,

24 00:04:35.430 00:04:44.180 Samuel Roberts: I’ve got your code, and I watched the loom and everything. I was hoping to just kind of talk through, the tech challenge a little bit.

25 00:04:44.430 00:04:45.719 Samuel Roberts: I mean, I just…

26 00:04:46.480 00:04:58.890 Samuel Roberts: you know, just, like, a conversation, nothing, nothing crazy. I’m not gonna be, like, hopefully, you know, tell me if I’m grilling you, but I don’t think it’s gonna be, like, a crazy kind of… you know, I don’t want to go too in-depth, because I want to kind of get a feel for just, like, you know, your understanding of everything, and…

27 00:04:59.360 00:05:02.439 Ujjwal Yadav: Whatnot, so let me, let me move over to there.

28 00:05:03.000 00:05:05.069 Samuel Roberts: And…

29 00:05:05.270 00:05:08.190 Ujjwal Yadav: I don’t know if you have it, do you happen to have it up and open, or…

30 00:05:08.990 00:05:11.299 Ujjwal Yadav: No, I have not opened it yet, let me open it.

31 00:05:11.840 00:05:12.760 Samuel Roberts: Okay, cool.

32 00:05:18.300 00:05:20.490 Ujjwal Yadav: Yeah, I hope so.

33 00:05:20.490 00:05:28.070 Samuel Roberts: Okay, yeah, I would just say, like, if you wouldn’t mind, like, sharing screen and walking through, the code, maybe we can just kind of step through a little bit, and you can…

34 00:05:28.250 00:05:34.000 Samuel Roberts: kind of… I’m just looking for, like, what’s the word? Like…

35 00:05:34.120 00:05:43.610 Samuel Roberts: not justification, but, like, the rationale for different parts of the code, and why you made certain decisions to where, and, you know, just kind of give me a high-level walkthrough, I guess, to start.

36 00:05:43.830 00:05:45.230 Ujjwal Yadav: Yep, sure.

37 00:05:47.530 00:05:49.799 Ujjwal Yadav: Okay, is my screen visible to you?

38 00:05:50.260 00:05:55.489 Ujjwal Yadav: Yes, I’m seeing, I’m seeing myself. Okay, okay. If you can’t see it then, okay.

39 00:05:57.530 00:05:59.380 Ujjwal Yadav: It’s… okay.

40 00:06:00.830 00:06:03.440 Ujjwal Yadav: Let’s move to the backend, main.py.

41 00:06:03.440 00:06:05.270 Samuel Roberts: Yep. That’s good.

42 00:06:10.750 00:06:12.210 Samuel Roberts: Yeah, okay.

43 00:06:13.230 00:06:18.070 Ujjwal Yadav: Okay, so, like, do you have any, like, question you want to start with, or shall I…

44 00:06:18.320 00:06:22.079 Ujjwal Yadav: Start with, like, From start.

45 00:06:23.620 00:06:27.270 Samuel Roberts: Yeah, I guess I would say if you could just kind of guide me through, like, what…

46 00:06:28.080 00:06:31.389 Samuel Roberts: when you… so I… let’s… let’s… let’s take a step back, maybe.

47 00:06:31.390 00:06:31.850 Ujjwal Yadav: Okay.

48 00:06:31.850 00:06:33.890 Samuel Roberts: You got the prompt, you got the,

49 00:06:34.110 00:06:42.409 Samuel Roberts: you know, the README and everything in the repo. I guess, what were your, kind of, first, how did you go about solving it? What was your first, kind of.

50 00:06:42.410 00:06:43.010 Ujjwal Yadav: Okay.

51 00:06:43.010 00:06:47.139 Samuel Roberts: What made you choose this kind of pattern to do it? That sort of stuff.

52 00:06:47.520 00:07:00.909 Ujjwal Yadav: Okay, so since, like, I have read about the assignment, I thought that the Python would be the best choice in this case, as we can use multiple libraries and everything here. Since it’s… I have a better support of few libraries.

53 00:07:01.160 00:07:19.459 Ujjwal Yadav: And I wanted to introduce multiple more things also, but due to the time constraint, I have included as much as possible, okay? So, for the high-level part, I have, like, made it kind of a production-based, so one is, like, a back-end folder, and next is, like, a front-end folder, okay?

54 00:07:19.710 00:07:32.810 Ujjwal Yadav: Let’s start with the front end, because it’s quite small, okay? So, front-end, I’ve only added, like, the simple Streamlit UI, as you have already seen that. Like, it will have, like, it will tell that if the API is connected or not through a health check API.

55 00:07:32.810 00:07:39.919 Ujjwal Yadav: And will give us a detail about that, and simple T2 upload buttons. I hope that you have seen the third loom also.

56 00:07:39.930 00:07:56.299 Ujjwal Yadav: But in that room, what I did was, I had few time remaining, so what I did is, I have added few… two new features into that. So basically, one of the features was to upload the CSV, like, we can also upload the… what we said, that CSV, forbidden list CSV also.

57 00:07:56.540 00:07:58.350 Samuel Roberts: Correct, yeah, I did see that. Yep.

58 00:07:58.550 00:08:07.540 Ujjwal Yadav: So, basically, I put it as an optional, so if you will upload it, it will use that. If not, it will use the fallback forbidden ingredient, like that, okay? Yep.

59 00:08:08.070 00:08:12.800 Ujjwal Yadav: Now, let’s move to the backend, okay? Sorry.

60 00:08:13.130 00:08:29.149 Ujjwal Yadav: So, for, like, high-level structure, let’s, what it does is, like, it has, like, a fast API, as I told you before, okay? Yep. And it has, like, two APIs. One is, like, health check API. So, what it does is, it will, like, check the health of the… if the backend is working or not, simply as that. Right.

61 00:08:29.360 00:08:38.579 Ujjwal Yadav: it actually needed. Next is our… another API. So, basically, I have written it in such a way that we get to know that it’s like a V1, then endpoint.

62 00:08:38.580 00:08:38.900 Samuel Roberts: Select that.

63 00:08:39.220 00:08:41.470 Ujjwal Yadav: To make it a bit production-like.

64 00:08:41.690 00:08:45.660 Ujjwal Yadav: So, it will have that, the main, evaluate.

65 00:08:46.170 00:08:58.379 Ujjwal Yadav: product API here, okay? It will, like, evaluate one product file, plus, optional CSV, as I told you, for any forbidden, this, ingredients, okay?

66 00:08:58.570 00:09:16.349 Ujjwal Yadav: And now, let’s move to the, LLM logic part in the backend. So basically, it’s inside the services part, okay? In the service, with all the LLM logic part is written. So basically, in AIService.py, this is triggered by evaluation service.

67 00:09:16.350 00:09:22.690 Ujjwal Yadav: this part, okay? Now, what it does is… Okay, let me…

68 00:09:23.260 00:09:26.709 Ujjwal Yadav: Okay, this is the main… okay, evaluation PY,

69 00:09:29.710 00:09:32.140 Ujjwal Yadav: I, I told with the cover, okay, hmm.

70 00:09:32.280 00:09:36.710 Ujjwal Yadav: So it will, like, have, like, multiple features, like, it will load the forbidden list.

71 00:09:36.850 00:09:44.949 Ujjwal Yadav: Of the ingredients, okay? It has multiple functions, like evaluate product, and it will call that function. Now, it will have two things, okay?

72 00:09:45.560 00:09:47.040 Ujjwal Yadav: like, C.

73 00:09:47.620 00:10:05.429 Ujjwal Yadav: since the forbidden list… if the item is present in the forbidden list, and in the document, or the text format, or in image, or any other things, what I did is, I have, made two routes now. One is that, of course, it will do the direct match. If it has found a direct match, let’s, like,

74 00:10:05.430 00:10:15.420 Ujjwal Yadav: Let’s say glucose, okay? If it finds a glucose, so it will automatically reject it, it won’t send to the LLM. That way, it will save our time, plus it will save our LLM cost, okay?

75 00:10:15.770 00:10:22.550 Ujjwal Yadav: Next, if it will, works, if it, like, passes that self-check, okay?

76 00:10:22.550 00:10:33.540 Ujjwal Yadav: So, what it will do is, it will, then send to the, Gemini as, like, a… like, you know, it might fail, so it will send the Gemini, it will check everything, it will even check the, what do we say?

77 00:10:33.540 00:10:42.600 Ujjwal Yadav: another name for it. Like, for example, it was Glucose here, but in the file, if it’s, like, Glucose D something something, I forgot the actual name, sorry about that.

78 00:10:42.600 00:10:43.380 Samuel Roberts: Okay.

79 00:10:43.380 00:10:53.690 Ujjwal Yadav: So it will find out that, and then, Gemini will return it the response in a structured way of the JSON that I have already set it in the…

80 00:10:54.280 00:10:57.919 Ujjwal Yadav: Mauders… iPhone… modders, you’re here?

81 00:10:58.280 00:11:05.129 Ujjwal Yadav: In schemas. So, it will return, result in a product name, status, and a reason string format, like that.

82 00:11:05.580 00:11:06.440 Samuel Roberts: Okay.

83 00:11:06.830 00:11:15.079 Samuel Roberts: So where, oh, sorry, take a step back. So where is that logic for the checking before the AI step?

84 00:11:15.240 00:11:16.959 Samuel Roberts: I missed that in there, I’m sorry.

85 00:11:17.250 00:11:20.690 Ujjwal Yadav: No worries, no worries. So, AI.Service is, like, our main…

86 00:11:20.890 00:11:26.190 Ujjwal Yadav: file part, okay? I have also added the max pre-retries and also the, delay.

87 00:11:26.580 00:11:34.709 Ujjwal Yadav: Sure. Like that. And for the delay part, I have added it as an exponential, because I have a free API of this, what we say, Gemini, so I don’t want to hit it again and again.

88 00:11:35.090 00:11:37.640 Samuel Roberts: Yep, yep, okay, okay, good, yep, thank you.

89 00:11:38.080 00:11:40.100 Samuel Roberts: Alright, sorry, continue now, yeah, my…

90 00:11:41.190 00:11:41.810 Ujjwal Yadav: Yep.

91 00:11:41.940 00:11:43.650 Ujjwal Yadav: So… sorry.

92 00:11:43.990 00:12:02.049 Ujjwal Yadav: So the main log… so basically, everything comes in this evaluation service.py initially, okay? So it will have one constructor, okay, where it is this… yeah, this is the constructor here. Okay, so basically, it’s, like, it loaded some CSV, like, load forbidden ingredient list, this,

93 00:12:02.510 00:12:03.290 Ujjwal Yadav: function.

94 00:12:03.290 00:12:03.720 Samuel Roberts: Yep.

95 00:12:03.760 00:12:07.280 Ujjwal Yadav: It’s function, like, it’s quite self-explanatory, okay? So…

96 00:12:07.280 00:12:07.760 Samuel Roberts: Yeah.

97 00:12:07.760 00:12:10.570 Ujjwal Yadav: Like, load the form unless if uploaded, okay?

98 00:12:10.730 00:12:29.900 Ujjwal Yadav: Then, let’s move to the next question, next part. It’s like, it will… it will have, like, multiple things. It will, like, parses, parse everything, it will extract and normalize this text from the file. It does the rule-based forbidden checklist from this, what you say, CSV. And then, if those passes, it will then

99 00:12:30.090 00:12:37.470 Ujjwal Yadav: sent to the LLM, as I told you before, okay? If it passes, then it will come to the AI service. So basically, all the AI logic part is done, here.

100 00:12:38.010 00:12:39.010 Ujjwal Yadav: Okay. Okay.

101 00:12:40.980 00:12:45.289 Ujjwal Yadav: Okay, so this one is the parsing part. So, basic parsing one is for, like.

102 00:12:45.460 00:12:51.679 Ujjwal Yadav: Either it’s from PDF or from image. For image, I’m using SRECOCR, okay?

103 00:12:52.020 00:12:52.659 Samuel Roberts: Saw that.

104 00:12:52.980 00:12:59.320 Ujjwal Yadav: And normal, like, for the text, it’s just doing the UTF-8, formatting part, okay?

105 00:13:01.080 00:13:04.279 Ujjwal Yadav: Yeah, and then move to the AI service, etc.

106 00:13:04.630 00:13:05.640 Ujjwal Yadav: Don’t forget, huh?

107 00:13:05.790 00:13:09.480 Ujjwal Yadav: So, it will, like, just send to the data to this, what we say.

108 00:13:12.050 00:13:26.459 Ujjwal Yadav: Gemini. For this, I have used… I’m using, like, 2.5 flash model. Okay. And for the prompt part, I have not written prompt in the hardcode… I have not hard-coded it inside the code. I have made another file for the prompt, and it’s there, okay?

109 00:13:26.660 00:13:29.199 Ujjwal Yadav: Let me show you the prompt. Sure.

110 00:13:30.100 00:13:31.040 Ujjwal Yadav: prompt.

111 00:13:33.460 00:13:35.240 Ujjwal Yadav: The prompt went…

112 00:13:39.080 00:13:41.700 Ujjwal Yadav: Yep. Okay, it was there, I phoned.

113 00:13:42.280 00:13:44.459 Ujjwal Yadav: Too many… too much code here, sorry.

114 00:13:44.460 00:13:45.480 Samuel Roberts: Yes, yes.

115 00:13:50.030 00:13:55.230 Ujjwal Yadav: So’s… Where’s the prompt went?

116 00:14:02.160 00:14:12.710 Ujjwal Yadav: P-R-O-M-P-T. Okay, yeah, this is the prompt, okay. Okay, it’s below the file, sorry. Yeah, I’ve done everything in the prompt, like, I’ve given the instructions to the,

117 00:14:13.130 00:14:27.210 Ujjwal Yadav: LLM, I have given the forbidden ingredient list and product text, and also tell that what it needs to do. And also, I have given it, like, an example response, like, do some… like, you have to reply something like this.

118 00:14:27.260 00:14:35.310 Ujjwal Yadav: Okay? Example response. So this is just a simple prompt. We can also, like, tweak it based on our, future needs, like that. Okay?

119 00:14:36.080 00:14:37.549 Samuel Roberts: Okay, yeah, thank you.

120 00:14:37.550 00:14:46.330 Ujjwal Yadav: So, basically, for… this is the main file, as I told you, so this is, like, the simple importing part. Then, I have, like, introduced this AI class.

121 00:14:46.790 00:14:56.159 Ujjwal Yadav: And then, like, somewhere between, like, here, you know, like, it’s loading the template prompt, and then evaluating, and all the retries and back-off part for the…

122 00:14:56.510 00:15:00.740 Ujjwal Yadav: Thing, and this is, like, Simple thing like that.

123 00:15:01.220 00:15:02.230 Samuel Roberts: Great. Okay.

124 00:15:02.830 00:15:03.510 Samuel Roberts: Okay.

125 00:15:03.660 00:15:11.619 Ujjwal Yadav: So, the flow is, like, Gemini returns some JSON as a text, and then this… evaluate, Jason?

126 00:15:12.580 00:15:14.180 Ujjwal Yadav: Evaluate…

127 00:15:17.970 00:15:18.910 Ujjwal Yadav: Jen.

128 00:15:19.170 00:15:24.570 Ujjwal Yadav: Yeah, evaluate one, yeah. Yeah, so basically it will evaluate… function is, yeah, evaluate response, this one.

129 00:15:24.700 00:15:44.109 Ujjwal Yadav: So it will, like, response… will check the validation through that model that I have already just made here, okay, and parse it and validate into our, like, Pythontec model. Then, if multiple retries fail, it will, like, again do the… this 3 retries, and with 2 second delay, exponentially, it will work. So, it works like this.

130 00:15:44.920 00:15:51.839 Samuel Roberts: So is that the same kind of, retry logic? If the… if it… if it doesn’t parse the JSON properly, what happens then?

131 00:15:53.920 00:15:56.679 Ujjwal Yadav: Can you repeat this part? Which… what did.

132 00:15:56.680 00:16:00.760 Samuel Roberts: Yeah, if for some reason the model doesn’t respond with the proper format.

133 00:16:01.480 00:16:04.260 Samuel Roberts: What’s the flow hap… what happens then?

134 00:16:04.880 00:16:08.780 Samuel Roberts: If, you know, if the model just responds with regular unstructured text or something.

135 00:16:12.060 00:16:16.750 Ujjwal Yadav: Okay, so if it fails, like, so it will, like, throw the error for the, what you say?

136 00:16:17.630 00:16:23.279 Ujjwal Yadav: if it doesn’t work, because LLM fails to generate on, like, a valid JSON, so the backend will, like.

137 00:16:23.430 00:16:28.490 Ujjwal Yadav: catch this error and retries up to 3 times, okay? So this is… this is what it’s doing.

138 00:16:28.490 00:16:30.609 Samuel Roberts: Same as if something else went wrong, is that basically…

139 00:16:30.610 00:16:36.050 Ujjwal Yadav: Yeah, of course it will definitely share the error for, like, here, in the UI.

140 00:16:36.570 00:16:37.160 Samuel Roberts: Okay.

141 00:16:37.630 00:16:49.760 Ujjwal Yadav: Like that. So, I thought, like, generally the LLM will, like, process the call, but if it fails, like, to give the, what we say, desired output, I think we should retry it, so…

142 00:16:49.940 00:16:51.279 Ujjwal Yadav: Yeah, okay, great.

143 00:16:54.590 00:16:55.360 Samuel Roberts: Okay.

144 00:16:55.590 00:16:56.740 Samuel Roberts: It’s a busy, yeah.

145 00:16:56.740 00:17:02.780 Ujjwal Yadav: Giving some… any bullshit answers, it will, like, at least give us… a correct thing.

146 00:17:03.030 00:17:04.250 Samuel Roberts: Right, right.

147 00:17:04.250 00:17:06.890 Ujjwal Yadav: Yeah, I didn’t prevent crashes and stuff like that, so that’s why.

148 00:17:09.089 00:17:14.860 Ujjwal Yadav: So, yeah, as I told you before, like, I have already decided, like, structured the output here, no? Like.

149 00:17:14.869 00:17:15.759 Samuel Roberts: Of course, yeah.

150 00:17:16.010 00:17:28.980 Ujjwal Yadav: So it’s, like, it’s, like, a very rare case. I don’t think, like, it will give that way. It will always give it in this thing, because I’ve written it two times, like, in the one… in the schema, and next is in the prompt also. So, yeah.

151 00:17:29.120 00:17:29.690 Samuel Roberts: Welcome.

152 00:17:30.970 00:17:31.710 Samuel Roberts: Great.

153 00:17:32.070 00:17:33.110 Samuel Roberts: Hey, Tom.

154 00:17:34.690 00:17:35.360 Uttam Kumaran: Yay!

155 00:17:36.020 00:17:37.370 Uttam Kumaran: Hey, nice to meet you.

156 00:17:38.200 00:17:39.160 Ujjwal Yadav: Hi, Tom.

157 00:17:39.800 00:17:40.910 Uttam Kumaran: Hey, how are ya?

158 00:17:41.380 00:17:42.529 Ujjwal Yadav: I’m doing fine.

159 00:17:43.620 00:17:49.140 Uttam Kumaran: Yeah, I guess my question was gonna be maybe a little bit about, like…

160 00:17:49.260 00:17:54.400 Uttam Kumaran: If you’ve built anything, similar to this, maybe you can walk through

161 00:17:54.850 00:18:00.589 Uttam Kumaran: like, a past project you worked on that was similar to this, and, like, I think more of what I’m interested also is, like.

162 00:18:00.950 00:18:12.510 Uttam Kumaran: how you… how you, in the past, you worked with a team, like, do you guys typically… you working with… within a repo, working with another team, with, like, GitHub and things like that? Maybe you can talk through a past project and, like, kind of the team structure.

163 00:18:14.030 00:18:17.729 Ujjwal Yadav: Okay, so the thing is, I have worked in, like, various teams. I have worked from, like, a big…

164 00:18:17.850 00:18:27.390 Ujjwal Yadav: a mid-sized organization to a very small startup at Automy. So, I have experience with both the things, and for the past projects,

165 00:18:27.790 00:18:39.620 Ujjwal Yadav: like, I have done various types of projects, okay? So, let me start with this, recent project that I’ve built. It worked in the Autominee. It’s like a real-time voice AI backend, and.

166 00:18:39.620 00:18:50.489 Uttam Kumaran: Yeah, maybe also, while you’re talking, maybe I can actually also have you just, do, like, a visual diagram of it, because I think that that could be helpful. Maybe I’ll just send you, like, a…

167 00:18:50.630 00:18:52.509 Uttam Kumaran: Big Jamboard, if you wanna, like.

168 00:18:52.620 00:18:54.150 Uttam Kumaran: Try to do it that way.

169 00:18:54.820 00:18:56.219 Ujjwal Yadav: Okay, okay, I got…

170 00:18:56.220 00:18:57.710 Uttam Kumaran: If you have a whiteboard, you could do it.

171 00:18:58.070 00:19:00.120 Ujjwal Yadav: Yeah, I have, like, Xia later on.

172 00:19:00.510 00:19:01.020 Samuel Roberts: Sure.

173 00:19:01.020 00:19:02.100 Uttam Kumaran: Oh, yeah, yeah, perfect.

174 00:19:05.170 00:19:11.710 Ujjwal Yadav: So, okay, I had drawn a few things here, okay. So, this is, like, one of the smallest, like.

175 00:19:12.520 00:19:26.420 Ujjwal Yadav: you can say high-level designing. So basically, it’s like a backend service, okay? And what it does is, it will, like, connect the inbound and outbound calls from the user to the OpenAI’s real-time, web sockets, okay?

176 00:19:26.490 00:19:45.720 Ujjwal Yadav: So here, what I did is, I have, like, connected the WebSocket to your WebSocket with our backend, and then it will open a new WebSocket into the OpenAI real-time backend, okay? So it’s like a kind of a bridge, but our backend has, like, multiple features, or you can say the tools, basically. So those tools include, like.

177 00:19:49.560 00:19:50.530 Ujjwal Yadav: Okasari.

178 00:19:52.950 00:19:54.070 Ujjwal Yadav: Here, sorry.

179 00:19:54.390 00:19:59.910 Ujjwal Yadav: So it’s like a multiple tools, like, end call functionality, like, it will end the call, it will have, like,

180 00:20:00.150 00:20:10.370 Ujjwal Yadav: it can do the call transfers, and also has another feature, it can, like, look up into a vector DB for the answers. So basically, we do not need to, like.

181 00:20:10.370 00:20:22.110 Ujjwal Yadav: write everything into the prompt, as it will, like, increase the cost and everything, and it is not ideal. So what I did is that if the company has too much data or something like that, I have attached another assistant to it.

182 00:20:23.740 00:20:34.869 Ujjwal Yadav: So I have, like, attached another assistant to it. So what it will do is, it will, like, call that function to retrieve all the information related to that

183 00:20:35.520 00:20:45.079 Ujjwal Yadav: Basically, to that company, okay? And then it will, verify everything into it, and then, give that response back to the user.

184 00:20:45.380 00:21:01.209 Ujjwal Yadav: So this is, like, a high-level flow of it, and one of the, like, most better, like, feature I’ve worked on it is the end call feature. So basically, on, like, on paper, it seems quite simple, like, yeah, we just need to close that WebSocket, right? We just, if we…

185 00:21:01.210 00:21:13.700 Ujjwal Yadav: and want to end the call, we need to end the web… like, close the WebSocket. So, what I did is, so, basically, if we do that thing, there will be… there’s a problem that it will immediately call that function and end the call.

186 00:21:13.810 00:21:23.049 Ujjwal Yadav: Now, the problem is that, let’s say, the AI want to say some sentence, like, okay, nice meeting you, have a nice day, or something like this. If it’s like a…

187 00:21:23.620 00:21:27.189 Ujjwal Yadav: big sentence. What it will do is…

188 00:21:27.690 00:21:32.100 Ujjwal Yadav: it will stop in between. Like, it will say, okay, and the call will get cut.

189 00:21:32.630 00:21:38.069 Ujjwal Yadav: So, this is, like, a bit of a clanky, voice AI. It was not very smooth operation.

190 00:21:38.970 00:21:52.920 Ujjwal Yadav: So, to tackle it out, what I did is, so, for the last sentence, whenever the, it’s gonna say the last sentence, or all the sentence, basically, we are, catching it inside a, what you say?

191 00:21:53.170 00:22:05.770 Ujjwal Yadav: array, and then, since Voice AI speaks in very constant speed, I calculated that time, how much time it will require to complete this sentence, and then I will call that function, okay?

192 00:22:06.300 00:22:20.470 Ujjwal Yadav: Now, this specific, this thing, is not working only in the case of the end call, but working for all other functions also, like call transfer. So, it will not immediately call transfer until and unless it completes its whole sentence, something like that.

193 00:22:20.580 00:22:23.270 Ujjwal Yadav: Except for the case of the knowledge-based call.

194 00:22:23.450 00:22:36.090 Ujjwal Yadav: So, I have not done this thing into the knowledge base one, because, like, we need the answer immediately. So, till the time AI finishes speaking to the user, what it will do is, it will

195 00:22:36.210 00:22:39.129 Ujjwal Yadav: Already, like, go to the new assistant.

196 00:22:39.370 00:22:41.490 Ujjwal Yadav: Fetch all the detail, and come back.

197 00:22:41.730 00:22:53.759 Ujjwal Yadav: So, that’s why I did it like that. So it’s a bit, you can say, smooth and everything. So, this is like a whole, you can say, technical overview of it.

198 00:22:54.080 00:22:55.450 Ujjwal Yadav: Of this project.

199 00:22:56.780 00:22:57.560 Samuel Roberts: Okay.

200 00:22:58.760 00:22:59.789 Ujjwal Yadav: Okay, you’re here, okay.

201 00:23:00.390 00:23:15.409 Ujjwal Yadav: And, next, the things I have done in the autumn meeting is, like, since, about the question that you asked for, like, we have, like, a single repo for this backend, and we, like, create PRs and every stuff like that to work on it.

202 00:23:15.410 00:23:22.499 Ujjwal Yadav: Basically, there were, like, only a few devs there, so only 2 or, two devs were there for this specific backend.

203 00:23:22.500 00:23:32.519 Ujjwal Yadav: So only we do all the work here. So we only, generally, what we do is, we have, like, a base code, we create our own PR, and try to merge it here, like that.

204 00:23:32.720 00:23:40.179 Ujjwal Yadav: And also, it’s like a multi-tenant based, and so it’s not like it’s working for a single client. It works for, like.

205 00:23:40.180 00:23:40.700 Samuel Roberts: Like…

206 00:23:40.900 00:23:42.990 Ujjwal Yadav: multiple,

207 00:23:43.580 00:23:57.539 Ujjwal Yadav: like, you can host multiple people here at the same time. Like, I have created a config file, so it will have a each config for each client, with their own ID, numbers, their assistant IDs, and stuff like that.

208 00:24:03.640 00:24:07.260 Samuel Roberts: Okay, and… Okay. So, can you scroll back up a little bit to the diagram?

209 00:24:07.780 00:24:08.719 Ujjwal Yadav: Okay, yeah, sure.

210 00:24:09.150 00:24:13.090 Samuel Roberts: Yeah, thank you. Okay. So, you said there were a few other devs working as well?

211 00:24:14.670 00:24:28.719 Ujjwal Yadav: Yeah, so basically, once I architectured this, so, like, like, currently it has, like, 25, you can say voice AIs, okay? So, if someone wants to add their another 26 voice AI, they just need to, like.

212 00:24:28.880 00:24:31.500 Ujjwal Yadav: add a… as I told you, I made it a multi-.

213 00:24:31.500 00:24:32.680 Samuel Roberts: They just need to…

214 00:24:32.680 00:24:36.280 Ujjwal Yadav: their own… Config of that new client.

215 00:24:36.450 00:24:42.220 Ujjwal Yadav: There. Okay. Yeah, last thing I forgot to mention, that the… it has, like.

216 00:24:42.410 00:24:50.559 Ujjwal Yadav: after the call gets ended, it has, like, a processing of all the call, okay? And it will send that to the CRM, to go high level, basically.

217 00:24:52.370 00:24:56.669 Uttam Kumaran: And was that all done in, like, what environments? All done in Python or in JS?

218 00:24:56.670 00:24:57.510 Samuel Roberts: Question again.

219 00:24:57.510 00:25:10.879 Ujjwal Yadav: Okay, so basically, we did it specifically in the Node.js. There are, like, two reasons for it. So, basically, when it was launched, so, at that time, it was launched only in the Node.js.

220 00:25:11.450 00:25:15.510 Ujjwal Yadav: And secondly, it’s much faster than Python, so that was our, like.

221 00:25:15.890 00:25:21.679 Ujjwal Yadav: reason for it, using. Because when it was launched, only that time I started working on it.

222 00:25:26.280 00:25:26.790 Uttam Kumaran: Okay.

223 00:25:27.570 00:25:32.609 Uttam Kumaran: So let’s say you were going to add another integration, right? Let’s say you had to,

224 00:25:33.050 00:25:44.209 Uttam Kumaran: to the backend, in addition to the OpenAI, you needed to pull, like, transcripts from a Postgres instance. Can you walk us through, like, the architecture of, like, how you would think about

225 00:25:44.530 00:25:45.580 Uttam Kumaran: Doing that?

226 00:25:48.750 00:25:56.440 Ujjwal Yadav: transcripts from Postgres. So, see, I don’t think, like, we need any transcript to tell any user.

227 00:25:58.420 00:26:05.010 Ujjwal Yadav: So, what I would generally do is, like, you want to say that thing, like, the transcript to the user, or…

228 00:26:05.260 00:26:06.750 Ujjwal Yadav: Anything else?

229 00:26:06.750 00:26:22.280 Uttam Kumaran: as part of… so you’re… you’re using OpenAI real-time, but let’s say you need to pull past transcripts related to that phone call. So you need to… you need to pull in… the same person’s calling multiple times, you want the AI to reference its past phone calls. So talk to me about how you would…

230 00:26:23.230 00:26:31.459 Uttam Kumaran: connect… how you would leverage, say, there’s data in a Postgres instance, like, if you could walk through the process of, like, integrating that data

231 00:26:31.880 00:26:33.900 Uttam Kumaran: That’s for the functions you would need.

232 00:26:34.080 00:26:34.870 Uttam Kumaran: Things like that.

233 00:26:35.510 00:26:36.410 Ujjwal Yadav: Okay.

234 00:26:36.740 00:26:38.529 Ujjwal Yadav: Let me think on it.

235 00:26:38.780 00:26:41.389 Ujjwal Yadav: To design, how would I design that?

236 00:26:43.430 00:26:45.300 Ujjwal Yadav: Let me clean up this first.

237 00:27:02.030 00:27:03.400 Ujjwal Yadav: Books, Chris.

238 00:27:03.800 00:27:06.319 Uttam Kumaran: We would have a postcards movie here.

239 00:27:06.960 00:27:07.800 Ujjwal Yadav: PG.

240 00:27:11.960 00:27:13.150 Ujjwal Yadav: Oh, goodness.

241 00:27:14.110 00:27:16.129 Ujjwal Yadav: Let’s create a new function call.

242 00:27:17.860 00:27:18.740 Ujjwal Yadav: practice.

243 00:27:22.500 00:27:25.990 Ujjwal Yadav: So, it’ll call that function here.

244 00:27:27.200 00:27:30.160 Ujjwal Yadav: Tender, tender, user.

245 00:27:34.640 00:27:45.849 Ujjwal Yadav: So, like… So, to do this thing, what I would do is, like, I would create a new function, so if, like,

246 00:27:46.040 00:27:58.380 Ujjwal Yadav: like, whenever the call comes in, so it will have a specific user ID, like, client ID, as I told you before. So, based on that ID, what it will do is, once the call gets connected, it will, fetch all the responses.

247 00:27:59.260 00:28:02.010 Ujjwal Yadav: In that PG vector, in the, like…

248 00:28:02.210 00:28:10.060 Ujjwal Yadav: PG storage. So basically, it will have, like, a data stored according… based on the IDs of the client ID, okay?

249 00:28:10.980 00:28:16.490 Ujjwal Yadav: And then it will have the user ID and transcript, everything, with the time, okay?

250 00:28:23.650 00:28:28.110 Ujjwal Yadav: And then we can, like, get the data based on that.

251 00:28:29.200 00:28:44.029 Ujjwal Yadav: But if they call it from, like, another different number, that ID might not work. So for that part, what we can do is, we can add, instead of ID, we can use the phone number, basically. So, we can get all the transcript.

252 00:28:44.230 00:28:46.630 Ujjwal Yadav: Which are linked to this phone number.

253 00:28:47.150 00:28:48.020 Ujjwal Yadav: Okay.

254 00:28:50.040 00:28:50.910 Ujjwal Yadav: Hello?

255 00:28:51.690 00:28:53.890 Samuel Roberts: Yes, yes, sorry, we’re here. We were just listening again.

256 00:28:53.890 00:28:55.749 Ujjwal Yadav: So yeah, I just, I, like, improvised.

257 00:28:55.750 00:28:56.740 Uttam Kumaran: Sorry, we’re here, we’re here.

258 00:28:57.900 00:29:07.679 Ujjwal Yadav: Yeah, I have, like, I want to improvise it right now, so… it will work on the base of the phone number. Look, basically, we will save the transcript based on the phone number, okay?

259 00:29:07.980 00:29:15.090 Ujjwal Yadav: And then, we will retrieve that, all the, like, history, and, like, send that to the…

260 00:29:15.350 00:29:25.500 Ujjwal Yadav: LLM beforehand. So we can use that. So we will, like, use their number to fetch when the backend receives the call immediately. It will… this function will trigger at that time.

261 00:29:26.000 00:29:28.690 Ujjwal Yadav: So, that’s how it’s… I will design it.

262 00:29:30.220 00:29:30.980 Samuel Roberts: Okay.

263 00:29:31.150 00:29:37.419 Ujjwal Yadav: Plus, also, like, and when the call will get end, we will generate the summary, and we can store it

264 00:29:37.640 00:29:44.189 Ujjwal Yadav: Also here. So instead of, like, if the transcript is too long, we can, instead of sending whole transcript, we can send a summary also.

265 00:29:44.900 00:29:52.540 Ujjwal Yadav: That will be, like, it will, like, better for, like, tokenization and everything, like, it will use less tokens.

266 00:29:52.760 00:29:54.849 Ujjwal Yadav: Less expensive calls, so, yeah.

267 00:29:56.230 00:29:56.770 Uttam Kumaran: Great.

268 00:29:59.430 00:30:05.780 Uttam Kumaran: Yeah, I guess my next question is sort of, like, tell me a little bit about, where you’re…

269 00:30:05.990 00:30:12.919 Uttam Kumaran: like, what you’re trying to do, even in… within AI or engineering, but also, like, how you’re trying to progress, like…

270 00:30:13.090 00:30:16.030 Uttam Kumaran: What do you… what do you… what sort of, like.

271 00:30:16.280 00:30:21.399 Uttam Kumaran: I’m sort of interested in why you’re looking for a new role, and, like, what do you want out of your next role?

272 00:30:23.290 00:30:28.660 Ujjwal Yadav: Okay, so the thing is, see, I want to work more into the AI, that’s, like,

273 00:30:29.290 00:30:35.409 Ujjwal Yadav: more into a technical part, so that’s what I’m looking for in, like, my future work.

274 00:30:35.720 00:30:36.950 Ujjwal Yadav: So, yeah.

275 00:30:37.420 00:30:39.120 Ujjwal Yadav: That’s, like, a smooth…

276 00:30:39.290 00:30:50.990 Ujjwal Yadav: that’s my, like, motivation. Like, mostly the thing is, I don’t, like, it’s not like I don’t work for any other kind of things. I prefer, like, working on, like, different, technologies and working with, like.

277 00:30:51.130 00:30:57.129 Ujjwal Yadav: multiple backends and AI. So this is, like, currently I’m, like, exploring the

278 00:30:57.240 00:31:01.750 Ujjwal Yadav: AI domain more, and I want to, like, grow myself in that, so…

279 00:31:02.640 00:31:15.009 Uttam Kumaran: I guess, tell me about, like, you know, the last place you’re working, or we’re currently working, like, why… why don’t you think they… like, why not go… why not do that there? I guess, like, tell me about why you’re looking externally for your next role.

280 00:31:16.210 00:31:20.860 Ujjwal Yadav: Okay, so, basically, in my current role, there’s no, like.

281 00:31:21.550 00:31:27.519 Ujjwal Yadav: like, a language barrier here, quite a lot. So basically, the… in the autuming, there’s, like.

282 00:31:28.020 00:31:31.259 Ujjwal Yadav: mostly the clients are from the Poland, okay?

283 00:31:31.420 00:31:32.069 Uttam Kumaran: Oh, okay.

284 00:31:32.400 00:31:51.799 Ujjwal Yadav: Yeah, so we have, like, a language barrier here, and secondly, the only… like, initially the team consists of many non-Polish people, along with the clients and the projects from the UK and, US, but recently the… it has changed, and most of the things they are focusing on the Polish market.

285 00:31:51.800 00:32:00.400 Ujjwal Yadav: Plus, the only devs remaining are two not… two, which are, like, non-Polish, so… One reason is this.

286 00:32:01.040 00:32:09.689 Ujjwal Yadav: So it’s like a bit of hassle with talking with the clients, like, we have to translate everything, and then make a loom, and also sometimes dub it into it, so…

287 00:32:10.050 00:32:13.709 Uttam Kumaran: Oh, okay, okay. It’s crazy.

288 00:32:13.710 00:32:15.909 Ujjwal Yadav: They don’t understand before.

289 00:32:15.910 00:32:19.470 Uttam Kumaran: How did you get into that… how did you get into… into that business?

290 00:32:20.390 00:32:24.019 Ujjwal Yadav: So basically, like, you are talking about how I joined Automy.

291 00:32:24.560 00:32:26.670 Uttam Kumaran: Yeah, like, what was your background before that?

292 00:32:26.880 00:32:30.740 Samuel Roberts: Okay, so before that, I have worked for, like, an Indian startup.

293 00:32:30.840 00:32:43.799 Ujjwal Yadav: basically Indian company, basically, so I worked as, like, a software developer born here, but my major work was, like, a normally front-end, back-end work, but I wanted to work in the AI field, and when I got the opportunity, I, like.

294 00:32:44.170 00:32:47.080 Ujjwal Yadav: Came to the… This otomy.

295 00:32:47.930 00:32:48.420 Uttam Kumaran: Okay.

296 00:32:49.200 00:32:53.720 Uttam Kumaran: What’s, like, the work culture at Automy, like…

297 00:32:54.070 00:33:00.969 Uttam Kumaran: are they, like… is it very heads down? Are you, like, always in meetings? Like, what is it like working there?

298 00:33:01.430 00:33:23.989 Ujjwal Yadav: It’s like a mix of both, actually. So, like, as I told you, I was a developer there, so I had, like, two roles there, working. So I have to do work as a client base also, and also do the work as a developer on the back end, and handling their, like, new clients. For example, some people are there only for the CRM part and stuff like that. They don’t know enough of the coding, so they just give me

299 00:33:23.990 00:33:25.769 Ujjwal Yadav: The details about that.

300 00:33:25.790 00:33:44.599 Ujjwal Yadav: their new, client, and I, like, add those things, their voice AI and stuff like that, or any other custom functions they actually require. So these are, like, a few of the functions that I told you, but, like, let’s take an example. One of our clients requires the ongoing call calculation, and you know, like, AI is not good at calculating everything.

301 00:33:44.630 00:33:51.090 Ujjwal Yadav: So, instead for that, I have, like, added a new function for it. It will, like, calculate the cost of the dams, basically.

302 00:33:51.340 00:33:57.340 Ujjwal Yadav: Basically, they sell dams, like, a home barriers, something like that, so… Like that.

303 00:33:59.070 00:33:59.670 Uttam Kumaran: Okay.

304 00:34:02.870 00:34:14.009 Uttam Kumaran: Yeah, I mean, to give you a sense of, like, of here, like, the biggest piece is, like, we’re… we’re a team, right? So we have a kind of a few people, and probably are working a lot slower

305 00:34:14.150 00:34:17.779 Uttam Kumaran: Then, kind of Automate, but most of our stuff…

306 00:34:17.880 00:34:26.870 Uttam Kumaran: like, has to go through, like, engineering rigor. And so, we’re starting to spend more time thinking about the architecture, thinking about how we’re building a real platform.

307 00:34:26.909 00:34:46.609 Uttam Kumaran: And so, definitely, I think there will be probably some change with that. You’ll be working with, like, a larger team, but also, like, I think there’s definitely a lot that you can learn from everybody about how to do things, and also, like, feel that you’re not just, like, getting asked by someone who has no idea. Most of the people in the company are engineers.

308 00:34:46.610 00:34:55.779 Uttam Kumaran: Including me and Sam, so nobody is, like, surprised at the work we’re doing, or has no idea. Like, I’m still in cursor every day doing stuff.

309 00:34:55.830 00:35:06.539 Uttam Kumaran: And so everybody is… is doing that. I think the second piece also is… is, yeah, I guess I… I’m… I’m… what I… when I ask, like, where… what you want to do more, it’s more interested in, like.

310 00:35:06.540 00:35:17.960 Uttam Kumaran: Do you end up wanting to lead a team? Do you want to do more architecture work? Like, how do you see your growth? You know, I totally hear you on, I want to do more in AI, and certainly, but…

311 00:35:17.960 00:35:32.649 Uttam Kumaran: Fundamentally, like, the work that we do in AI is engineering work, and so I’m interested in hearing, okay, how do people grow? Like, okay, I want to go deeper into architecture, into security, into scalability, into

312 00:35:32.650 00:35:38.710 Uttam Kumaran: I want to start doing more analytics. Okay, maybe I actually want to help on the architecture side. I want to lead a team.

313 00:35:38.710 00:35:53.719 Uttam Kumaran: Right, so I’m trying to find out how I can turn folks… how I can give you more leverage, right? Because everybody here grows, and so nobody does something now in six months, or we’re going to be growing. And so that’s kind of more I’m curious in, like, where you think

314 00:35:53.780 00:35:57.020 Uttam Kumaran: You know, you might want to be, you know, later down the line.

315 00:35:57.940 00:36:05.860 Ujjwal Yadav: So, what I’m thinking is, like, I want to architecture everything and scalability part, I want to focus on that initially.

316 00:36:05.920 00:36:20.149 Ujjwal Yadav: And, like, learn a bit more about, like, how to make scalable systems and stuff like that. So, I’m, like, more interested and, like, want to work into it and want to grow as, like, I can say, architecturing scalability part.

317 00:36:20.290 00:36:26.569 Ujjwal Yadav: So it’s, like, quite interesting for me, and, like, I think I have still need to learn quite much more here.

318 00:36:27.710 00:36:31.859 Uttam Kumaran: Yeah, like, it seems… it’s clear that you can ship features. I think what… what…

319 00:36:31.950 00:36:41.309 Uttam Kumaran: You know, for us, it’s important that, like, you’re doing things that are scalable, you’re doing things within, like, a platform or repository that are, like, things are reasonable, modular.

320 00:36:41.340 00:36:58.320 Uttam Kumaran: like, we do a lot of documentation, for example, is really important, and things like that, but also it’s still moving fast, so I think that’s great. Yeah, I feel like that’s kind of, like, most of the questions I had. I appreciate you taking the time. I don’t know, Sam, if you had any other questions.

321 00:36:58.550 00:37:04.269 Samuel Roberts: No, I think we got into it, pretty well. I was curious about the thinking behind some of the code earlier, but we got into that already.

322 00:37:04.270 00:37:04.790 Uttam Kumaran: Cool.

323 00:37:05.550 00:37:06.090 Samuel Roberts: Yeah.

324 00:37:07.190 00:37:13.790 Uttam Kumaran: Okay, cool, so maybe, as well, maybe me and Sam will… we’ll refresh after this, and then we’ll definitely be in touch over email.

325 00:37:14.540 00:37:17.170 Ujjwal Yadav: Yeah, sure. Okay. Thanks, Tom. Thanks for your time.

326 00:37:17.170 00:37:29.200 Uttam Kumaran: Thank you so much for taking the time, and I appreciate… I know it’s… I know it’s, it’s not, it takes a bunch of hours to ship this, and then you went a little bit above and beyond, so I appreciate it. Yeah. No, it’s really, really helpful.

327 00:37:29.640 00:37:41.469 Ujjwal Yadav: Yeah. Yeah, it was nice working on this project, plus talking with Sam. It was nice talking with you, too. And I really appreciate that you, like, asking me the thing, like, what I want to do, so it was quite a…

328 00:37:41.610 00:37:45.919 Ujjwal Yadav: Nice talk, you can say. It’s not… it doesn’t seem like an interview, this was a nice thing.

329 00:37:46.200 00:37:57.109 Uttam Kumaran: Cool, yeah, no, you’d be surprised. I never… nobody in my career ever asked me what I wanted to do, so I try to do that now. You know, so I appreciate it.

330 00:37:57.640 00:38:00.500 Ujjwal Yadav: Yeah, I really like that idea, yeah.

331 00:38:00.500 00:38:01.480 Uttam Kumaran: Okay.

332 00:38:01.740 00:38:02.500 Uttam Kumaran: Cool.

333 00:38:02.500 00:38:04.340 Ujjwal Yadav: I hope you’ll talk to you soon. Bye-bye.

334 00:38:04.390 00:38:05.620 Samuel Roberts: Yeah, thanks so much. Bye.

335 00:38:05.620 00:38:06.410 Uttam Kumaran: Alright, thank you.