Janani Sivasubramanian

Could you help me understand more about your background by giving a brief introduction of yourself? Okay. So uh, I'm a Snowflake developer and, uh, in currently working in Tata consultancy services. So my role is for, uh, in development, uh, in the Snowflake technology where I will be where I will be writing, uh, SQL queries and, uh, fine tuning optimization in that SQL queries, data modeling. And, uh, and, also, I will get the required inputs from the data analyst from the mapping sheets, and then I will be proceeding it with the, uh, query we'll start with the query as for the, uh, as for the, uh, reports they want. So my, uh, my role is for a Snowflake, uh, SQL things. In that Snowflake, uh, dev, there are, uh, different environments. And in each environment, uh, we will be selecting each roles and each data warehouses, uh, uh, where we can, uh, we'll start with each attribute and, um, uh, given by the data analyst. So what we will do is, like, we will start, uh, writing the, uh, data, uh, attribute each attribute, and we'll first analyze. Before starting with the code, we will first analyze the table frequency, table load frequency, whether it is in the, uh, correctly, uh, whether it has been correctly loaded, uh, whether all the tables are correctly loaded means correctly loaded means, uh, load loads, uh, each. The table's nature, we will analyze it. And after analyzing, we will start with the code attributes. And in the code attributes, we will be, uh, uh, we will be selecting the rep which are the attributes required for that particular report. So we will be selecting that, and, uh, obviously, we will be requiring more, uh, tables every all the attributes can't be present in a single table. So we will be using fine tuning the queries, and we will be using many tables. And, uh, in that many tables, we will be, uh, fine tuning it, and we will be using as a uh, as of now, we did we are just giving the UDF. Uh, we are creating the UDF, uh, with which parameters has, and we will just, uh, give the invoking query to the API team, and they will process it. And, uh, so in our query from the Snowflake site, after we are running it and the query, uh, it will take only milliseconds to run it. Uh, it won't even go for a seconds, and, uh, we have done many reports, uh, like, as our team and as a player, uh, individual player, I have done, uh, 2 to 3 reports, uh, big reports. And then that I have, uh, I have been, uh, trained, uh, like, done well in my part. So I have also got, uh, client appreciation as well as, uh, my team lead has, uh, personally motivated me and, uh, always encouraging me. And, uh, and also after the completion of the reports, they have appraised, uh, me. And, uh, yeah, I'm, uh, Yeah. As of now, that's it. Thank you.

How, how would you leverage snowflakes time travel and zero copy cloning features to enhance data recovery and testing? How would you leverage snowflakes time travel and zero copy cloning features to enhance data recovery and testing? So cloning concept, zero copy cloning means, first of all, cloning, it's not like a new concept, which is like, even it starts with a good clone in that also. So as of in snowflakes, zero copy cloning features to enhance the data recovery and testing is data duplicate, like, time travel and zero copy. So zero copy cloning means, which can clone a database seamlessly with a simple command. It's very similar to pointers and which acts as a reference to another variable, instead of actually being that variable. So when a command is given to clone a database, it does the following like, it will create the new database and all objects underneath the databases are created, one could expect the data to be held as a replica, but it's so what's so when as a user when I'm cloning the particular object or the particular table or the particular attribute, the flow, the cloud service, like it simply fetches the data from the actual source. Like it makes the data as up to date as possible, like latest timestamp and the latest dates. So it but the snowflake does it everything with the metadata. So this time travel and zero copy cloning features are mostly done in the service layer. Advantages of zero copy cloning and time travel features to enhance data recovery, almost no additional storage costs, and no, no waiting time also required and no need of any administrative efforts like to clone the process instantly promotes a corrected fixed data to the production. So Transcribed by https://otter.ai

How would you design a resilient data pipeline in Azure Data Factory to handle intermittent source data availability? Okay. So how would you design the receiving data pipeline in Azure Data Factory to handle intermittent source data availability? So, uh, in Azure, uh, we can ingest the data from both on premises and, uh, cloud data source and transform or process the data by using existing compute services such as, uh, Hadoop and, uh, where the results are published in the on premise or the cloud for business intelligence, uh, uh, which is known as the Azure Data Factory. And Azure Data Factory, it's basically a a data integration service, uh, where, uh, for it works in the cloud for orchestering, the automating automating the data movement and the data transformation. Uh, it does not store any data itself. It allows you to create data driven workflows to or orchestrate the, uh, movement of the data, uh, between supported data stores and then process the data, uh, using the compute services in other region or in an on premise environment. So it also allows you to monitor and manage the workflow using both programmatic as well as in the, uh, UI mechanisms. Uh, it supports the diar data. It also supports the data migrations, uh, carrying out data integration process and, uh, integrating data from different ERP systems and, uh, uh, loading it into Azure Synapse for reporting. So first, uh, if we need to connect and collect the data, then we should, uh, transform and enrich like, once it is present in the centralized data store in the cloud, it is transformed using computer, uh, services, uh, for the Azure data lake analytics. And then, uh, the transform data from the cloud to on premises source like SQL server or keep in your cloud storage sources, uh, for the conception by the BI and, uh, for, uh, other tools like, uh, Power BI, uh, tools, which we which way they can, uh, get us input. So data factory copying the data from a source data store to a sync data store. So Azure, it supports the data source, source or sync data stores like Azure Block Storage or Azure Cosmos DB and Azure Azure Azure Azure BlockStore Block Storage and the Cosmos, uh, DB. And uh, Data Factory, uh, it supports the transformation activities such as high map reduce park, and that can be handled to pipelines either individually or chained with other activities. Uh, area supported data source for data transformation activities. Uh, so, like, moving data from uh, from from one source to sync or that, uh, with, Uh, it's it can be done using this, uh, uh, this required tools.

What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? What steps would you take to migrate an existing data model to Snowflake ensuring minimal downtime? Okay. Upload.

Can you implement an auditing system? Can you implement an auditing system in Snowflake to track historical changes of critical datasets? So if we need to, uh, yes, we can implement an auditing system in Snowflake to track the historical changes of critical datasets. So if we need to, uh, trace or track a particular operation performed, uh, like, given by me or given by the user or vice versa, then the, uh, then, uh, it can be achieved by querying the, uh, Snowflake account usage query history view, uh, uh, that records the history of the queries executed in the account. We can design our audit logs through the, uh, data available in this view. So, uh, like, by giving the role use the role account admin, and you can give the, uh, start time query, and, uh, we can use the start time as greater than this current date, uh, 4th April and less than the start time, 30th, uh, like, uh, 5th April, like, uh, uh, or or the 31st 30th April. And, uh, we will keep, like, uh, use query test like this warehouse should be. And, uh, if we give order by the start time, and then we can apply, uh, like, different sets of filters on the query text column to search for the different comments that the users must have executed. So the redemption of the data in this view is, uh, like, uh, uh, for per month. Like, one one this now for in this example from 4th April to 30th, that means this irrespective days. So Can you implement an API by providing the robust features. So auditing and logging. So Snowflake Snowflake actually maintains a detailed audit logs for all API activities, uh, includes, uh, including the data access and modification operations. These logs records who performed the actions, what actions were taken, and when they are gonna audit audit audit audit records can be used for compliance, security, and troubleshooting purposes, like for DDL and DML. So for Snowflake logs, DDL operation performed through AP such as creating or altering tables, views, data structures, and schema evaluation on for the DML, like, uh, insert, update, delete. Uh, these are the DML, uh, for the data changes and the access, uh, through the APIs. So users and applications are, um, for this, uh, like, audit policies will be helpful. So Snowflake allows administrators to define the audit policies to specify which types of API activities should be audited or, uh, which, uh, or which policies can be configured to capture specific actions, users, and, uh, object the access controls.

Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands. Describe how you would configure Azure Data Factory to handle dynamic scaling based on workload demands.

Based on the provider Snowflake SQL snippet, can you explain what the issue is with current stored procedure and how it might affect data processing? Okay. Okay. Based on the provided small flex sequence, but can you explain what the issue is with the current stored procedure and how it might affect data processing? Create or replace the procedure. Update order status. Begin update order. Set status. Discharge the status. Place the missing commit statement. Okay. So It's like creating the 2 sort procedures, uh, where we will begin, and then, uh, finally, we are dispatching it. So, uh, Snowflake SQL super. Can you explain what the issue is with the current stored procedure and how it might affect the data processing? Actually, uh, in, uh, set statements and where statement and we have given the set status as equal to dispatch. Commit is executed after operations like insert or delete and update. So, uh, the syntax itself, uh, is using the For this particular store, the procedure set status, wire status, missing, commit statement. Uh, so, uh, uh, it's missing Missing the commit statement. So for each transactions, the commit is, uh, uh, in the in in d v side, if we need, uh, if we are giving the update and if we are giving set and we are giving the bad, so, uh, like, we should insert the we should get the inputs and we should, uh, after that that commit transaction, that that print statement, that should be there. And because of that is the issue, and it might, uh, it might it just it might not run because of the syntax error. And, Or it might be rolled back or okay.

Suppose you come across a section within a CICD pipeline configuration using Azure Data Factory or Azure Synapse, identify the possible oversight and and its potential impact on the deployment process. So Stages build jobs, build and test steps, echo building and testing, and display name build and test jobs deploy to, uh, prod condition succeeded steps. Uh, echo deploying to production. Display name deploy to production. No explicit environment specified for the deployment job. So if you come across this section within a c a CD, actually, in this, uh, the jobs, uh, the job name they have given building and test script echo, building and testing, and the display name, it will be coming as build and test. And, uh, it's the stage isn't the deployed. Uh, so after build deploy stage is there, so job deploy to prod. Actually, it's, uh, it's moving to production environment and that, uh, condition, it is succeeded. So as of now, it's good to go out for the production environment and script echo deploying to production and then deploy to, uh, display name deploy to production. So no explicit environment specific for the uh, deployment job. What I should do? Suppose I come across the section within a CICD using Azure Data Factory, identify the possible oversight. Identify the possible oversight and its potential impact on the it will be deployed successfully.

Would you build a machine learning data pipeline in Snowflake? How would you build a machine learning data pipeline in Snowflake and ensure it is updatable as new data becomes available. Um, so as of mission learning, uh, we will see, uh, mission learning in first we will, uh, see the, uh, that, uh, Billman's equation. So according to the Billman's equation, we should value the state. Uh, we we should know the value of the state, and that value of the state should be, um, uh, equal to the, uh, maximum for that particular state, uh, for, uh, maximum of the reward point of the, uh, for that particular action. Uh, reward, uh, reward given for particular state and action, and then it will be, uh, given with a discount, which is the comma into the value of the prime state, which is an existing. So, uh, according to the Bellman's equation, um, and how the mission learning data actually, this Billman equation, it's coming from how the mission is learning. Like, how that, uh, um, how it does refinement learning by doing with it's doing its reinforcement learning. In that reinforcement learning, it is do it is, um, uh, getting updated, uh, uh, like, it's learning from its own mistakes. It's not preprogrammed. It's learning from its own mistake. So, uh, so in in coming in while doing that mission learning data pipeline in Snowflake and, uh, so so the the data pipeline like, it it might have many, uh, source tables or customer interactions or external, uh, data feeds. And using the Kafka connector for the Snowflake, the Kafka connector facilitates seamless integration, uh, between, uh, the Snowflake and, uh, it's enabling the real data streaming. Real time data, uh, it's, uh, it's seriously no. It's it's a very, uh, huge impact. And, uh, so enabling the real time by connecting that Kafka and the Snowflake real time data streaming. So, uh, so Snowflake seems to capture, uh, and it changes from the incoming data stream. For example, streams can be created to capture the policy updates or claim submission and customer interaction. This seems to continuously track and stores the, uh, changes ensuring the real time data synchronization. And after that, the transformation and the, uh, enrichment will happen where, uh, where, uh, the whether the users can apply transformations or enrichment operations, uh, in this, uh, uh, to this capture data. Uh, for instance, uh, you, uh, you can interest the customer interaction with the demographic data or claims data with the geolocation and then data processing with task automates the data processing activities based on the, uh, predefined schedules or the conditions and the real time, uh, data loading with smoke pipe. Uh, it's continuously automatically integrating the transform data from the streams, uh, into target tables within the Uh, so analytics and visualization for that, we will be using Power BI tool. And in that, uh, this actually, in Snowflake itself, we can see the, uh, by giving the graph, we can see how, uh, is, uh, like, how it can be, uh, like, applying machine learning algorithms, building predictive models, and generating the visualizations, uh, using Tableau or Power BI. And the dashboards are paid and monitored, and alerts are given also. Uh, continuous monitoring and alerting are crucial to ensure the performance, uh, for the, uh, for the reliability of the data pipeline.

Applied DevOps practice to improve collaboration and reduce the lead time in snowflake data operations. So creating the isolated environments or the schema changes or the table selection like new features like schema changes means which can be expensive or operationally since they require developers to like us to code changes with the update and the database schema. So using the test data to validate the feature changes is far from ideal even if the schema is the same between the production and the pre-production stages. So with production data is so costly and time consuming so that it can result in the schedule delays or compromise the product quality. So scaling the production environments but will try before pre-production have been smaller in scale due to costs associated with the procuring or standing up and managing the production scales. So instantly create any number of isolated environments reduce the schema change with the variant data types and rapidly seed the pre-production environment with production data and it has two ways to fasten up the environment with the production data secure data sharing is used when environments are on separate snowflake accounts and zero copy cloning is used when the environments are on the same account. And instantly scale environments to run jobs quickly and cost effectively where the scaling issues are easily overcome with the snowflakes pre-second pricing structure customer pay only for the time needed to run the job and no matter the cluster size. So develop teams can like they can scale the production environment to run a big process in a fraction of the time and scale it down when the process ends. So easily roll back with the snowflake time travel. So they keep like to handle the errors in the rollbacks in the CI CD process with time travel enabled object like tables or schemas. It can be deleted or easily restored and access programmatically at a point like particularly within 19 day within the 90 days having to manage and maintain costly backups. So beyond this feature snowflake data cloud makes DevOps like using simple language like it makes easy by using simple language like Python SQL and Node.js and near zero maintenance because it's a cloud based it's a completely software as a service provider and it's real time integration with the external services using custom like external services like custom service that are stored and executed outside the snowflake.

Optimize the data retrieval time in Snowflake while dealing with the large semi structured j, uh, JSON datasets using Snowflake SQL. Optimize data retrieval time in Snowflake while dealing with large semi structured JSON datasets using Snowflake's equal. We can optimize it by using the copy into operation. Although if it is a very, uh, like, um, um, like, optimizing the large and restructure j JSON datasets using the, uh, using the Snowflake SQL, uh, like, uh, it's if it is getting loaded into a bulk section destination, uh, we can use copy into operation and, uh, which can be loaded into the Snowflake table using a create table command. And, uh, and after seeing that create table, uh, then, uh, specifies the variant column in which the JSON data resides. And, uh, if the path to the RGB element in the JSON, uh, like, in the JSON or cast the data to the appropriate type. And, uh, a view could manually be created as the essential hides this complexity, um, from the end user communicator. So to automate the creation, uh, such as a view, we will need to know 2, uh, key pieces of information. Uh, one is the path to each element and the other one is the data type of each element. And, uh, uh, it turns out we can leverage a natural join, uh, to flatten a subquery to get the information about the individual elements in the JSON document. So so build the query and then run the query, loop through the returned elements, build the view, and then, uh, construct the DDL or the DM construct the DDL first and then run the DDL to create the view. Then it turns out the snowflakes, uh, procedure, s piece are the snowflake stored procedures are perfect for the use. Yes.