Monday 9:00 AM - 5:00 PM · Room 342

Deep Dive: Building a framework for orchestration in Azure Data Factory

Finding the balance between cost, efficiency and performance for cloud-based ETL processes can be a tricky proposition, and a good ETL framework will help you get there.

In this all-day session we will take a deep dive into what the components of such a framework may look like in Azure Data Factory, and why orchestration can be a good option if you're trying to minimize cost. We will also do an in-depth review of an actual framework I've developed and use today, which you can use as a starting point for your own efforts.

If you are currently using Azure Data Factory and feel like you're recreating the wheel all the time, or if you're planning to move from SSIS to ADF in the future and would like some ideas on how to create an ETL framework, this will be the perfect training day for you.

Monday 9:00 AM - 5:00 PM · Room 348

Introduction to Microsoft Fabric for the Data Professional

Join us for a full day of exploration into Microsoft Fabric, the revolutionary Software as a Service (SaaS) product that’s changing the game in Business Intelligence and Modern Analytics. For the last 3 years, Microsoft has been building this groundbreaking technology, which offers the ability to ingest and link to existing data without duplication. With Azure Synapse Data Engineering, Data Factory, Data Science, Data Warehousing, Realtime Analytics, and Power BI all at your fingertips, the possibilities are endless. In this pre-conference event, we’ll guide you through a full build scenario, showing you how to link existing data, and utilize OneLake. Join us and discover the power of Microsoft Fabric!

Tuesday 9:00 AM - 5:00 PM · Room 342

ETL, KQL, and RTI: Harnessing Data in Motion with Microsoft Fabric

Real-time Intelligence in Microsoft Fabric brings us into the next generation of data movement and event-driven architecture. As the arms race of cloud data platforms continues, make sure you know the weapons at your disposal so you and your company are fully equipped for the future - because the future is not far away. While you may be familiar with RTI’s streaming data capabilities, did you know it can be an integral part of your ETL architecture? While you may have heard of KQL, did you know it can do data visualization? While you may have heard of RTI, did you know it can help you monitor your entire data estate?

Join RTI product group members and Microsoft Data Platform MVPs for a full day of learning. Takeaways include:

• Loading a full medallion architecture using RTI components
• Use case by use case breakdown of the components of RTI and how customers are using them today across their data estate
• Data visualization and trend analysis with KQL
• Monitoring your entire data estate using Real-time Intelligence in Microsoft Fabric

Tuesday 1:30 PM - 5:00 PM · Room 340-341

Maximize efficiency in your data-driven projects by leveraging AI and Copilot technologies.

In the Total Economic Impact™ of Microsoft Fabric study, the composite organization based on data from four companies using Fabric showed an increase in data engineer and data scientist productivity of up to 50%, plus a 15% increase in business analyst productivity.

In this workshop, we will explore the transformative potential of various copilots, AI services within Microsoft Fabric. It covers live demos and hands on experience on Microsoft Fabric Copilots, AI Skills, Fabric integration with AI and GenAI services.

Participants will thoroughly explore the Copilot maturity framework and learn strategies to overcome adoption challenges, enabling customers to maximize the benefits of Copilots.

Wednesday 8:30 AM - 9:40 AM · Room 340-341

Microsoft Fabric: Ultimate Data Security for Robust Data Warehousing

This session delves into the advanced security features of Microsoft Fabric, specifically designed for data warehousing. Attendees will learn how to classify and protect sensitive data using sensitivity labels and policies in Microsoft Purview.

The session will cover the implementation of Row-Level Security (RLS) to restrict access at the row level, ensuring users only see data relevant to their roles. Additionally, it will explore Column-Level Security (CLS) to secure specific columns in datasets, providing an extra layer of protection for sensitive information.

Participants will also discover how to use data masking techniques to obfuscate sensitive data, making it unreadable to unauthorized users while maintaining usability for authorized personnel. The session will emphasize aligning data warehousing practices with regulatory requirements such as GDPR, HIPAA, and CCPA, leveraging Microsoft Fabric’s compliance tools and features.

Join us to gain a comprehensive understanding of how to secure your data warehouse end-to-end within Microsoft Fabric.

Wednesday 8:30 AM - 9:40 AM · Room 344

Microsoft Fabric: Lessons from Year 1

As of November 2023, Microsoft made the announcement that Microsoft Fabric was Generally Available. In the time since, many organizations jumped into the exciting world of this emerging technology.

Any fabric, technical or not, needs to be well maintained to retain its initial quality and usefulness. In the same way, your approach to implementing Microsoft Fabric could either set yourself up for a successful, long-term implementation, or one that needs a rebuild in 1-2 years.

In this session, I'll walk through features of Fabric that have lived up to the hyped labels and work as prescribed. I'll also provide demos challenges I've faced with the product and how I've worked through them. Finally, I'll recommend routine activities that should be performed on any Fabric environment to keep it nice and clean.

(If this wasn't enough Fabric references and you need more, check out this session for even more puns!)

Wednesday 10:20 AM - 11:30 AM · Room 320-321

Fabric Data Factory: What's New and Roadmap

In this session, you will learn the exciting product innovations and roadmap for Fabric Data Factory. You will learn how Fabric Data Factory provides industry-leading data movement, transformation and orchestration capabilities. You will learn how AI-powered development experiences will enable you to be more productive and effectively build data integration solutions.

Wednesday 10:20 AM - 11:30 AM · Room 348

Getting started with SQL database in Fabric

Microsoft Fabric, the unified data platform, now includes an operational database solution and it is SQL! Come learn all the fundamentals of how this solution is the same and different from other SQL deployment options. We will also show you the value of using SQL in the Fabric ecosystem including developer experiences, automation, monitoring, and integration for AI applications.

Wednesday 2:00 PM - 3:10 PM · Room 320-321

Building a framework for orchestration in Azure Data Factory

Finding the balance between cost, efficiency and performance for cloud-based ETL processes can be a tricky proposition, and a good ETL framework will help you get there.

In this session we will look at what the components of such a framework may look like in Azure Data Factory, and why orchestration can be a good option if you're trying to minimize cost. We will also walk through an actual framework I've developed and use today, which you can use as a starting point for your own efforts.

If you are currently using Azure Data Factory and feel like you're recreating the wheel all the time, or if you're planning to move from SSIS to ADF in the future and would like some ideas on how to create an ETL framework, this will be the perfect session for you.

Wednesday 2:00 PM - 3:10 PM · Room 344

PowerBI, DirectQuery and SQL Server. It is a good choice?

You will learn best practices, tips and tricks on how to successfully use SQL databases (OnPremise,IaaS, PaaS,SQL Managed Instance) with PowerBI on production environments.

How improve the performance, using for example, Read-Scale, HyperScale or Synapse, partitioning, ColumnStoreIndexes, Indexed views, etc..

How to monitor and diagnostic your database and find out issues with Query Data Store. These learnings are fruit of Microsoft CSS support cases, and customer field engagements.

Thursday 8:25 AM - 9:35 AM · Room 345-346

Harness the Power of Microsoft Fabric and Notebooks

Microsoft Fabric gives us a notebook experience unlike any other previous Microsoft product. The power of notebooks is immense. Sure you can use data from your Lakehouse, Data Warehouse, or OneLake but what about PYODBC? Can we connect to a relational database without a Data Pipeline or a Data Flow? Could we just download a file from Kaggle or Github and start using Data Wrangler? Could we use Beautiful Soup to scrape data, load it into a Pandas data frame, and begin working with it? Can we invoke OpenAI models using GPT to glean new insights into our data? Yes, Yes, Yes, Yes, and Yes. Yes we can, and in this session you will learn how to harness the power of Microsoft Fabric Notebooks.

Thursday 12:40 PM - 1:50 PM · Room 320-321

Choosing the Right Data Store--An Overview of Azure Data Platform Choices

There are several different data platform solutions for use within your application. Selecting the right option can make the difference between a well-performing application and a poorly performing one; not to mention the cost aspect of choosing the wrong solution.

In this session we'll look at the options of Azure SQL Database, Azure SQL Database Managed Instance, and Cosmos DB to see when these are all going to be the correct option, and when these aren't going to be the right option; both from a price and performance perspective.

Thursday 2:00 PM - 3:10 PM · Room 342

Data Processing Architecture: Key Design Principles & Considerations

In the era of big data, the design of data processing architecture is crucial for efficient data management and analysis. This presentation explores the fundamental principles and considerations essential for constructing robust data processing systems. Key design principles such as scalability, reliability, security, and flexibility are examined in detail.

The architecture's ability to handle varying data flows, ensure data integrity, and maintain security across multiple stages is emphasized. Additionally, the presentation discusses various architectural patterns, including data warehouses, data lakes, and data flow pipelines, highlighting their respective use cases and benefits.

Furthermore, the presentation contrasts traditional data processing architecture with the emerging concept of data mesh. While traditional architectures focus on centralized data processing and transformation, data mesh advocates for a decentralized approach, promoting domain-oriented data ownership and self-serve data infrastructure.

This comparison underscores the shift from monolithic data management to a more flexible and scalable architecture, addressing the diverse needs of modern data-driven organizations.

By adhering to these principles and considerations, data engineers can create systems that not only meet current data processing needs but are also adaptable to future technological advancements and data requirements.

Thursday 2:00 PM - 3:10 PM · Room 340-341

Working with OAuth 2.0 APIs in Azure Data Factory

Working with APIs can be tricky, and even more so when it's an OAuth 2.0 API. Add to that an ETL platform and automation, and you now have a perfect storm that's pretty difficult to navigate.

This session is about my journey with OAuth 2.0 APIs while trying to extract my own financial data, how I struggled with the authorization flow and how it finally started making sense.

We'll talk about what an OAuth 2.0 API is, and why they are so difficult to deal with when your tool of choice is an automated ETL platform. After that we'll take a closer look at the steps to develop an ADF pipeline that extracts data from an OAuth 2.0 API, and review some tools that can help you throughout the development process.

Friday 10:20 AM - 11:30 AM · Room 344

Harnessing the Power of Data Mesh: A Deep Dive into the Four Pillars of Modern Data Architecture

Abstract: In today's data-driven world, traditional data management approaches often struggle to keep pace with the growing complexity and scale of enterprise data. Enter Data Mesh, a revolutionary paradigm that redefines how organizations handle data. This session will explore the four foundational pillars of Data Mesh architecture: Domain-Oriented Decentralization, Data as a Product, Self-Serve Data Infrastructure, and Federated Computational Governance.

We'll delve into how Domain-Oriented Decentralization empowers teams to own and manage their data, fostering agility and innovation. Discover the concept of Data as a Product, where data is treated with the same rigor and care as any other product, ensuring quality and usability. Learn about Self-Serve Data Infrastructure, which democratizes data access and tooling, enabling teams to work independently and efficiently. Finally, understand the importance of Federated Computational Governance in maintaining security, compliance, and consistency across decentralized data environments.

Join us to gain insights into implementing Data Mesh in your organization, transforming your data strategy, and unlocking the full potential of your data assets.

Friday 12:30 PM - 1:40 PM · Room 344

Build a Robust App with Fabric SQL Database, GraphQL API, and User Data Functions

Discover how to harness the full potential of Fabric to build powerful modern data-driven applications. In this session participants will learn how to use the API for GraphQL and User Data Functions over SQL databases and other data sources in the Microsoft Fabric Platform to build modern “CRUD” (Create, Read Update and Delete) data APIs that you can immediately call from an application. APIs for GraphQL enable you to effortlessly connect your Fabric data with your applications, with only a few clicks. If you have more complex business logic for your data, you can implement it with Python User Data Functions. With these two powerful building blocks and a SaaS-ified, developer-friendly experience, your app will be up and running in no time!