Dhananjay Pandit

Hi. Uh, my name is Dananjay Pandit. Um, I have been a senior software engineer, um, working at Axon for about five and a half years now. Um, and my background, um, I have done my undergraduate studies from the University of Mumbai, and my graduate studies for my master's in software engineering from Arizona State University. Um, and like I said, I've been working with Axon for about five and a half years now. I started off as a software engineer, and I've been promoted to a senior software engineer while working at Axon. A little bit of other introduction would be, um, um, I we currently I am currently on the records management team, uh, which helps build a cloud based report writings offer for law enforcement officers. And I am currently looking for a new job since I'm moving, uh, from the United States back to Mumbai, India to be closer to family. Um, and so that's why I'm looking for new job opportunities, which are either based in and around Mumbai or, uh, which are completely beautiful. What else? Yeah. Uh, that's about it. My technology stack in terms of, uh, brief introduction is I have used Golang, um, extensively at, uh, Axon for the last 5 5 and a half years. Um, all of my undergraduate and graduate studies used Python as my primary language. Uh, so I'm very familiar with that. And I have recently finished the Azure Cloud course on Udemy. We use Azure pretty heavily here at Axon, um, so I knew about some of the features, uh, with some hands on experience. However, doing the course has helped me get a lot more insights into some of the specifics, um, that have the internals and the workings of Microsoft Azure. So that's a brief introduction about myself.

Uh, so which bill concurrency patterns would be most beneficial for handling multiple video streams simultaneously? Um, I don't have too much experience with, uh, video streaming specifically. Um, ours is a cloud based application with no video streaming involved. However, I do have experience with 2, uh, goal line concepts, which I feel would be, uh, pretty relevant for this answer. I'm gonna increase the height of this so that I am slightly more in view. There you go. Uh, alright. So first things first. So I think the, uh, I have used gRPC, um, as our API with Go, and I think the streaming feature of gRPC would be really beneficial when paired with the Go concurrency patterns. Um, so I would say it would be really nice if we can have an API which would stream responses, I e, uh, video streams back to our clients In terms of which go concurrency patterns would we we would be using, um, I I I feel like channels would be a great, um, answer to that. Uh, we would be able to do error handling very easily. Uh, we would be able to defer error handling, uh, in case that nothing comes up. You don't have to actively keep on checking, um, and the channel can let us know if it finds, um, anything that's concerning of this work. So I feel like that would be a really interesting pattern to use for video streaming. GRPC streams combined with channels in Golang would be really powerful, and I feel like it would be able to do the job. Any issues that I see, not necessarily. I cannot think of anything right now, um, but that could also be because I don't have super close knowledge of the workings of video streaming specifically. Uh, but, yeah, I think that would be my answer. GRPC streaming APIs, uh, coupled with Golang channels.

How would you leverage AWS Step Functions in the orchestration of media processing workflows? So I I don't know, uh, specifically of the AWS step functions, but I'm assuming that they're very, very similar to Microsoft Azure's, uh, function apps. Um, so I think it would be really beneficial when for orchestration. I think that that's the main question. Uh, and whenever we want to do any sort of orchestration, uh, within our application where first you wanna do this, then you wanna do that, then you wanna do that, whatever you have a workflow, so to speak, uh, that you need to build into your application, um, that is when the step functions are really gonna be beneficial. Um, likewise with function apps, where you want something to be transient. You don't want it to live for a very long time. It is a step uh, to a larger means. Um, so you would want it to do that step and then just go away. Um, so I think that is where I would use any well, AWS Lambda, for example. But I think the step functions give us an additional benefit on top of it that you can orchestrate your workflows as part of it. So you can say, first do this, then do that. If something goes wrong, you can have fallbacks, uh, in place so that you don't completely adder out, so you can gracefully head handle adders as part of, uh, a step function as well. So I would use it for orchestration mainly around things that need to be transient, uh, however, still need to be tied to a singular workflow. Um, cron jobs or something like that is something that immediately comes to mind. So, yeah, if you upload let's say, it's you upload a new media content to, um, our servers, um, step functions could be a great way to ingest those. Uh, so step 1 would be to support the whole uploading of the entire file. Uh, step 2 can be, uh, where you extract the metadata out of those files. And then step 3 can be, like, elastic search ingestion for, um, those, uh, metadata for that video. Um, so, yeah, that's that you can orchestrate those things using AWS steps functions.

Oh, in Kubernetes, how would you ensure zero downtime deployments for a live video streaming service? Zero downtime. That's definitely, um, a challenge. Uh, I would say there we could that I would have to ask follow-up questions to this. What does it mean, um, to have a zero downtime deployment? Are you just talking about deployment specifically? Because in that case, that is a much focused answer. Um, I'm gonna assume that that is what is being asked since the question says so that we're just specifically saying 0 downtime deployments. So there's a couple of ways to do things. Uh, one way to do it would be and I wouldn't recommend it this way because we cannot guarantee that it would be completely zero downtime. Um, one way to do it would be gracefully terminating the connections and um, the pods that, uh, are gonna go away as part of the deployment and then spinning up new pods in its place after one of them has, uh, gracefully shut down. Uh, by graceful shutdown, I mean, the connections being handled by uh, the connections and the requests being handled by the pods that are going away, uh, should be shut down, uh, gracefully. Whatever requests that are in the midst of being served need to be completed, uh, before it shuts down, And then new pods will be spun up in its place, and those would be, uh, responsible for handling the traffic. We in this instance, it's really hard to maintain zero downtime because we never know, uh, if, uh, if one of the pods is running a long running process. Right? Like, if something if an if a pod is having WebSocket connection, if it's a really long lived connection to the client, we cannot just wait for the pod to, uh, you know, finish, uh, its connection because it's gonna be a while before, uh, it does that. So I think the the other approach that I really like is, uh, blue green blue green deployments or canary deployments through Kubernetes, where we could spin up the entire pods, uh, cluster, um, with the new version as a new deployment, um, and not have it serve any traffic. Um, the old so we would have 2 deployments, uh, ready. 1 would be the old deployment. 1 would be the new deployment. The old deployment would still be responsible for handling a 100% of, um, the traffic. And slowly, we can start cutting over a small percentage of traffic over to the new deployment. So let's say 10% of the traffic gets cut over to the new deployment. Then if everything goes well, then we can slowly start scaling up. Um, this is, of course, under the assumption that the deployment that we're making, the new deployment, doesn't immediately need to go out to all customers because this is a relatively slow moving process. So you would go to 10, then you would maybe go 25, 50, 75, a 100. And so we would have to have checks and balances in place at every one of those checkpoints to ensure that, uh, things are live and the new deployment is serving traffic without any major errors.

Describe a method to ensure data consistency during stateful application upgrades in Kubernetes. Stateful application upgrades. Um, I can't say I've ever worked with a stateful application in a professional setting, meaning where it involves a deployment and everything. So I'm not in the best position to answer this question, um, but let me try to think about it if something else comes to mind. We could have data stored onto the Kubernetes pods itself, uh, either through a config map or a volume mount or something like that, and have data be consistent across, uh, upgrades in that sense, uh, where the application itself, uh, would know, uh, what data it had in referenced in the past. But again, I haven't used it, uh, so I don't wanna make, uh, an answer up, uh, for something that I'm not, uh, intimately familiar with.

What Python data structures and algorithms would be most efficient for processing live video stream metadata? Processing live video stream metadata. Uh, so it's a live feed. So it's continuously coming in. Um, and I think the the first one that comes to mind would be a queue. Uh, right? So you would have something that comes in. Uh, it's 1st in, 1st out. Uh, comes if it comes in first, it's the first one to get processed, and we want to sort of maintain that order. So we would want to use a queue in that sense to make sure that we are streaming, uh, and processing the stream metadata in the right order. So queues would be a great example for the Python data structure, um, for processing live video stream metadata. In terms of algorithms, um, anything that can leverage the queue data structure would be a good algorithm for processing video stream metadata. Uh, again, I don't have professional experience with live video streaming. Uh, so I don't know, um, specifically which algorithm would be beneficial.

Looking at this Kubernetes deployment, yaml, this configuration is causing intermittent issues with the application's availability. Identify why this might be problematic and what could be done to mitigate the issue. App version, apps v one. Deployment, video back end. Replicas 2. Video back end. Video back end. Oh, um, so immediately, I'm looking at the limits and the request. I'm seeing that the limit is the exact same for both memory and CPU, um, as of the request. So whenever there's a spike, um, in traffic coming to this particular, uh, video back end application, it is unable to scale up because we have limited the, uh, memory and CPU to its initial request of 256 megabytes and 0.5, uh, of the CPU cores. So it does not have any room to scale up and request additional resources. Um, so unless we have something in place like a horizontal pod autoscaler, it's not gonna be able to scale up, um, its resources or its replica count because we're just saying that there's 2 replicas, uh, for this particular application. So I would recommend in this instance to have a higher memory limit, uh, depending on how much memory and CPU we're planning to consume, I would say at least we should go up to, like, 512, the double, uh, memory, 512 megabytes, and double the CPU as well as in terms of 1 instead of 0.5. So at least there is some buffer between the memory and the CPU request and the memory and CPU limits. Uh, and that should help the, uh, issues that we have been having with intermittent issues with this application's availability. Uh, so yeah.

How would you use Golang's built in testing package to create a load test for a video streaming API? Uh, testing package to create load test. Um, so it would effectively we our test would have to act as an end to end test with spawning of multiple different client connections to a video streaming API. Um, and we can do that through concurrency control in Go. So we would spin up however many, like, x number of, uh, new clients as part of the, uh, testing package and then try to connect all of those clients at the exact same time, uh, do the video streaming API and see if it is able to gracefully handle all of the connections, uh, look at the latency and all of those numbers as part of, uh, those clients that we have respond, and see if it's able to gracefully error handle and all of those types of things as part of the, uh, testing package. Uh, we can assert that it doesn't error out. We can assert that, um, uh, the latency is within acceptable metrics. And so, yeah, concurrently spawn multiple different client connection, uh, to the video streaming API as part of the load test and, uh, see if our API gracefully handles the situation or if it dies completely.

What Terraform modules would be instrumental in scaling a media streaming platform on AWS? I don't have, uh, hands on experience with Terraform, so I don't know what Terraform modules would be instrumental. So I'm gonna skip this answer.

Elaborate on the advantages of using Terraform when managing large scale Kubernetes clusters for media services. I would say so I know a little bit about Terraform. We do use it, um, in my current company. However, Terraform as an application is more used by our site reliability engineers, um, as opposed to our actual developers. So which is why I don't have hands on experience, uh, using Terraform. Uh, however, the benefit of using Terraform is basically to manage our infrastructure. So I I understand Terraform as infrastructure as code. So you can similar to how you can write application as a code in Kubernetes, where you can define the services, the config maps, and the secrets, and then have that be, um, deployed to Kubernetes in a declarative, um, uh, fashion. You do the same thing with Terraform and your infrastructure where you can stand up, um, servers or you can request for additional resources, uh, through Terraform. And the benefit of doing that is it's, uh, it can go through a code review pipeline, uh, so people can approve or deny changes as part of Terraform, so you're not willy nilly killing additional servers or creating new servers and stuff like that. It can go through an approval workflow. It can be version controlled, so you can put it into Git, uh, which also means that you can have CICD be part of Terraform, um, and then have that go and scale up servers or scale down servers or the infrastructure as required. So that's those are the advantages that I see. Um, so Terraform would be at a layer lower than Kubernetes where we're talking about the actual infrastructure for Terraform, and Kubernetes would be managing the services that get deployed on the infrastructure set up by Terraform.