Databricks

In this 2-hour hands-on course, Andreas Kretz walks through how to build end-to-end data pipelines on Databricks using Spark Declarative Pipelines and Lakeflow Designer.

Along the way, you'll:
✔️ Learn the fundamentals of Delta Lake and Unity Catalog
✔️ Build a Bronze → Silver → Gold pipeline using the medallion architecture
✔️ See how declarative pipelines handle dependencies and orchestration automatically
✔️ Build pipelines visually with Lakeflow Designer
✔️ Explore how Databricks Assistant and Genie can accelerate development

youtube.com/watch?v=nrfP2_…

A cloud agent may outlive the request, worker, container or deployment that started it.

This reference implementation pairs Temporal with Lakebase Postgres so recorded work survives worker replacement, evidence and review state stay queryable, and human decisions can arrive days later. Unity Catalog remains the governed policy source through synced tables.

See how to build it:
databricks.com/blog/build-dur…

Off-the-shelf industry data models give you a starting point. They don’t capture the specific language, rules and relationships that define your business.

Vibe Data Modeling takes a different approach. The open-source agent helps teams build, validate and evolve business-specific data models using roughly 250 modeling rules, while keeping data modelers and business stakeholders in the loop.

In this video, see how teams can start with 40 industry models as a baseline, then iterate toward models that better reflect how their business actually works.
youtube.com/watch?v=z2K-6o…

A new @Replit integration with Databricks accelerates the development of business apps and deployment on live, governed enterprise data.

A four-step setup guide walks Databricks admins, Replit org admins, and end users through going from zero to a governed production app in 15 minutes.

Get started: databricks.com/blog/how-build…

Workday Data Connect federation is now in Beta in Unity Catalog.

Unify people and finance data from @Workday with the rest of your enterprise data in Databricks and govern it.

Teams on Workday Data Cloud get zero-copy, read-only access to the HR and finance tables Workday shares.

Databricks runs the queries, Unity Catalog governs access, lineage, and auditing, and teams can combine current people and financial data with the rest of their enterprise data for analytics and AI, including Genie-powered natural-language exploration, workforce analytics, and financial planning on trusted, governed data. databricks.com/blog/announcin…

Anthropic's latest model, Claude Haiku 5.5, is now being made available, as a Day 0 launch, on Databricks!

It is the cheapest, fastest, and most capable small model Anthropic has ever released. On Databricks' OfficeQA Pro V1 benchmark, Claude Haiku 5.5 delivers ~15% higher quality than Haiku 4.5 at a fraction of the cost.

Run Haiku 5.5 & 60+ other models on the data that already sits in Databricks and power your agentic apps and agents. Unity Gateway takes care of governing, monitoring, and securing every call as you scale.

Try Claude Haiku 5.5 → dbricks.co/3TQncpg

What if AI could figure out the right business context without waiting for a fully built ontology?

Genie Ontology does that by inferring relevant context across your data and assets to answer open-ended questions.

Advancing Analytics’ @MrSiWhiteley digs into how OntoRank chooses what to trust, why certified assets still matter, and what that means for data governance.

Watch the full video: youtube.com/watch?v=voZXeH…

How do you stop an agent from overspending or taking one risky action too many?

Omnigent policies can control both cost and behavior by checking actions before they happen. They can set spending limits, restrict tool usage, and track accumulated risk across a session so the next action can require approval once a threshold is reached.

🔗 Learn more in the blog: databricks.com/blog/contextua…

🔗 GitHub repo for this series: github.com/viktoriasemaan…

⚽ @adidas knows that the best insights don't just answer questions, but help you ask better ones.

With Databricks Genie, adidas helps analysts spend less time crunching data and more time uncovering insights, helping teams make smarter decisions and ultimately design better products.

Hear more about how adidas’ product innovation runs on data.

Databricks is bringing databases up to AI speed.

“Lakebase, a ground-up rearchitecture of databases, can spin them up and shut them down at AI speeds, facilitating the multitude of small, fast workloads of AI-driven development.”

Excited to see Lakebase in @FastCompany's Next Big Things in Tech issue. dbricks.co/4zfhoov

IP Functions are now GA on Databricks!

Every firewall, load balancer, and application server generates records keyed on IP addresses. Analyzing that data in SQL used to mean regex hacks, custom UDFs, and brittle bitwise math.

Now you can parse, validate, and join IPv4 and IPv6 addresses and CIDR blocks with built-in SQL functions. They're optimized in Photon.

⚡ Up to 3.1x faster, 6.4x cheaper CIDR joins in benchmarks versus another leading cloud data warehouse.

IP Functions are a critical building block for our Security Lakehouse vision: threat detection, investigation, and network analytics on one governed copy of your data.

Learn more: databricks.com/blog/ip-functi…

Your data team deserves more than dashboards.

📕 The Big Book of Data Warehousing and BI is a practical guide to building the foundation for faster time-to-insight, stronger governance and AI/BI readiness on the Databricks Lakehouse.

Inside:
• Reference architecture and code samples
• Migration strategies and dimensional modeling
• Performance and cost optimization at scale
• Governance with Unity Catalog
• BI integrations and ways to extend BI with AI

Download the eBook: databricks.com/resources/eboo…

Genie One just got a major set of updates across context, data access, collaboration, and automation. 🧞

Here's what you can do now:
• Ask questions with business-aware context built in. Genie Ontology is enabled by default
• Apply your org's data conventions to every prompt with workspace instructions
• Upload Word docs, images, CSVs, spreadsheets, and PDFs directly into your workflow
• Query Unity Catalog tables inside Genie with schema preview and one-click access requests
• Automate recurring work with scheduled tasks that reference past runs for consistent outputs

@YoussefMrini walks through all of it in under 3 minutes:
youtube.com/watch?v=hRayT1…

Ever tried getting Claude Code and Codex to actually work together without playing copy-paste between terminals?

@leonvz takes Omnigent, our new open-source meta-harness, for a spin to show how multiple coding agents can share sessions, rules and security policies in one system.

He demos forking work across agents, multi-agent review and debate with Debby, and splitting implementation across subagents with Polly.

Watch the full walkthrough:
youtube.com/watch?v=gYtQ1L…

What changes when more of your Postgres working set stays in DRAM instead of falling through to slower cache and storage layers?

For fixed-size Lakebase Postgres computes with CU >= 80, we now set shared buffers to 75% of DRAM and disable the local file cache, keeping hot pages in the lowest-latency access path.

In production, large endpoints have seen up to 2x throughput, fewer reads from the storage layer and lower latency after the change.

See how we’re improving Lakebase Postgres compute caching:
databricks.com/blog/improving…

Coding tasks aren’t equally difficult, so why send every one to the same model?

Smart Routing evaluates each task separately and selects the lowest-cost model capable of doing the job, helping balance model quality, latency and cost.

In this demo, Omnigent splits an app build across planning, backend and frontend work, routes each task to a different model, and runs some of them in parallel.

Learn more: bit.ly/smart-routingai

GitHub repo for this series: github.com/viktoriasemaan…

How much of your AI workload is really just making a decision?

ai_decide is a new Databricks AI Function built for fast, structured decisions over governed data. It classifies, scores and chooses what happens next in a fraction of a second, with lower latency and cost than an LLM on similar tasks.

Use it for model routing, document processing, agent evaluations and real-time app logic 🐍 databricks.com/blog/introduci…

Agentic apps are putting new pressure on the data stack.

Join Databricks Co-founder and Chief Architect @rxin to explore why Postgres is the foundation for a new data processing architecture, and how LTAP lets transactions and analytics read from the same copy of data.

Learn how to:
- Bring agent memory, application state and historical data under the same permissions
- Remove sync intervals as a bottleneck in the agent memory loop
- Maintain compatibility with existing Postgres drivers, extensions and ORMs

You’ll also see the architecture in action through demos and enterprise patterns.

Live November 10–12. Register now: databricks.com/resources/webi…

Two new frontier models just landed on Databricks.

@OpenAI GPT-6.1 Sol and @SpaceXAI Grok 4.7 are both live today.

GPT-6.1 Sol leads the cost-quality Pareto frontier on OfficeQA Pro v2. Grok 4.7 reaches the frontier at enterprise document parsing. Two benchmark leaders, available the day they ship.

Frontier models are shipping faster than ever. Unity Gateway keeps you right there with them. It picks the right model for the right task, governs every call, and controls costs across GPT-6.1 Sol, Grok 4.7, and 60+ other frontier and open models already on Databricks.

Try GPT-6.1 Sol and Grok 4.7 today: docs.databricks.com/aws/en/release…

AI doesn’t have an intelligence problem. It has a context problem.

When business context is scattered across dashboards, documents, tickets and chats, users struggle to get fast, trusted answers and data teams stay stuck fielding ad hoc requests.

See what it takes to close that gap with a unified context layer that helps AI deliver trusted answers, act autonomously and give teams back time. Go under the hood of Genie Ontology and see the approach in action with a product demo.

Watch on-demand: databricks.com/resources/webi…