# LearnSQL > A free, read-only learning site by CPE Today: sample databases, a REST API, an MCP server and long guides on SQL, DAX and Power Query M. Guides are plain Markdown. Start with the agents file. ## Start here - [Agents file](https://learnsql.cpetoday.com/agents.md): how to connect, query rules, which guide to read - [All guides in one file](https://learnsql.cpetoday.com/llms-full.txt): every guide, concatenated - [API documentation](https://learnsql.cpetoday.com/docs): REST API reference - [MCP server setup](https://learnsql.cpetoday.com/ai/connect): connect an AI assistant ## Start here - [How databases work, for finance professionals](https://learnsql.cpetoday.com/analyze/how-databases-work.md): Tables, keys and relationships explained with invoices and ledgers, why an ERP is not one big spreadsheet, and how to read a schema. - [From Excel to SQL: what you already know](https://learnsql.cpetoday.com/analyze/excel-to-sql.md): VLOOKUP is a join, SUMIFS is GROUP BY, a filter is WHERE: translate the Excel skills you use every day into SQL, side by side. ## SQL - [SQL essentials](https://learnsql.cpetoday.com/analyze/sql-essentials.md): SELECT, WHERE, joins, GROUP BY, CASE, dates, subqueries and CTEs, with the SQL Server and MariaDB differences. - [SQL deep end](https://learnsql.cpetoday.com/analyze/sql-deep-end.md): Window functions, recursive queries, ROLLUP, deduplication and gaps-and-islands, indexes and execution plans. ## Power Query - [Power Query editor: clean data without code](https://learnsql.cpetoday.com/analyze/power-query-editor.md): The editor, applied steps, the common clicks for finance data, and what each click writes in M. - [M basics: queries, values, lists and records](https://learnsql.cpetoday.com/analyze/power-query-m-basics.md): What a query is, the let and in structure, types and values, lists, records and each. - [M tables: rows, columns, joins and groups](https://learnsql.cpetoday.com/analyze/power-query-m-tables.md): Select, add and change columns, merge, group, pivot and unpivot, plus text, date and number functions. - [M functions and patterns](https://learnsql.cpetoday.com/analyze/power-query-m-patterns.md): Your own functions, if and try, running totals, a date table, latest row per group and a complete worked example. - [M performance, errors and when to use M](https://learnsql.cpetoday.com/analyze/power-query-m-performance.md): Query folding, the errors you will meet and how to fix them, and when to use M, SQL or DAX. ## Data modeling - [Data modeling for finance](https://learnsql.cpetoday.com/analyze/data-modeling-for-finance.md): Facts and dimensions for a general ledger, budget against actual at different grains, chart of accounts hierarchies and the mistakes that give wrong totals. - [Relationships and the data model](https://learnsql.cpetoday.com/analyze/relationships.md): Star schemas, RELATED, role-playing dates, many-to-many and security. - [Model the LearnSQL databases](https://learnsql.cpetoday.com/analyze/learnsql-models.md): Connection details and a star-schema recipe for each sample database. - [Data structures and Power BI limits](https://learnsql.cpetoday.com/analyze/data-structures.md): Which relational structures exist, which sample database teaches each, and what Power BI and Excel cannot model or load. ## Power BI and DAX - [Power BI Desktop how-to](https://learnsql.cpetoday.com/analyze/power-bi-desktop.md): Model view, DAX query view, parameters, security, publishing and PBIP projects. - [Power Pivot in Excel](https://learnsql.cpetoday.com/analyze/power-pivot.md): The Excel Data Model: load data, relate tables, write measures, PivotTables and cube formulas. - [DAX fundamentals](https://learnsql.cpetoday.com/analyze/fundamentals.md): Measures versus columns, row and filter context, VAR and RETURN, and your first measures. - [CALCULATE and filter context](https://learnsql.cpetoday.com/analyze/calculate.md): The most important function in DAX: add, remove and keep filters. - [Time intelligence](https://learnsql.cpetoday.com/analyze/time-intelligence.md): Date tables, year to date, prior year, rolling 12 months and fiscal calendars. - [Iterators and table functions](https://learnsql.cpetoday.com/analyze/iterators.md): SUMX, RANKX, TOPN, ADDCOLUMNS and when to use each. - [Pattern cookbook](https://learnsql.cpetoday.com/analyze/patterns.md): Percent of total, ranking, Pareto, new versus returning customers, budget variance, calculation groups. - [Report recipes](https://learnsql.cpetoday.com/analyze/report-recipes.md): The standard sales, finance and operations report layouts and the measures behind them. - [Performance and troubleshooting](https://learnsql.cpetoday.com/analyze/performance.md): Make DAX fast, read common error messages, and debug a wrong number. - [Function reference](https://learnsql.cpetoday.com/analyze/functions.md): A quick reference of the functions you will use, grouped by purpose. - [DAX deep end](https://learnsql.cpetoday.com/analyze/dax-deep-end.md): How the formula and storage engines work, context transition, how CALCULATE evaluates, query plans and why a measure is slow. ## Finance playbooks - [SQL for accountants](https://learnsql.cpetoday.com/analyze/sql-for-accountants.md): Accounting questions answered in SQL and checked on real sample data: aging, trial balance, year over year, duplicates, accruals and fund balances. - [DAX for financial reporting](https://learnsql.cpetoday.com/analyze/dax-financial-reporting.md): Model a general ledger for DAX and build income statement, balance sheet, budget variance and ratio measures with correct signs and fiscal calendars. - [Power Query for accounting data](https://learnsql.cpetoday.com/analyze/power-query-accounting-data-prep.md): Fix the import problems finance teams meet: bank exports, trial balances, monthly folders, account mapping and data quality checks, with M you can paste. - [Power BI for finance teams: ledger to dashboard](https://learnsql.cpetoday.com/analyze/power-bi-for-finance.md): A start-to-finish build: connect to the ledger, shape it, model it, write the measures and lay out a monthly finance pack. - [Audit and controls analytics in SQL](https://learnsql.cpetoday.com/analyze/audit-analytics-sql.md): Journal entry testing, duplicate payments, vendor master checks, Benford's law and cutoff tests, each with SQL run on real sample data. ## Sample database guides - [CPA firm practice management (cpafirm)](https://learnsql.cpetoday.com/analyze/cpa-firm.md): Clients, engagements, tax returns, time, billing, trust and a general ledger for a small-business CPA firm, with realization, WIP and AR aging queries. - [Nonprofit donors, grants and funds (nonprofit)](https://learnsql.cpetoday.com/analyze/nonprofit.md): Donor management, pledges, grants and proposals, fund accounting, payroll allocation and volunteers for a regional nonprofit. - [Power Query playground (pqlab)](https://learnsql.cpetoday.com/analyze/power-query-lab.md): 86 tables built for practicing Power Query: a retail backbone plus deliberately messy tables (now with geospatial exercises), a technique index and M recipes. - [IoT and smart home (smarthome)](https://learnsql.cpetoday.com/analyze/smart-home.md): One connected family home: 83 devices, 13 months of sensor readings, solar, battery and EV energy, automations, security, bills, trips and a floor plan. - [City government operations (bigbear)](https://learnsql.cpetoday.com/analyze/big-bear.md): A small resort city: 311 requests, snow operations, dispatch, permits and code enforcement, a water utility, the lodging tax, parcels and fund accounting, on a map and a July-June fiscal year. - [Retail data warehouse (retailwh)](https://learnsql.cpetoday.com/analyze/retail-warehouse.md): Star and snowflake schemas, slowly changing customers, price history, inventory and order-pipeline snapshots, a factless table, multi-currency and budgets, in one retail warehouse. - [Big data: manufacturer orders (mfgorders)](https://learnsql.cpetoday.com/analyze/big-orders.md): 25 million order lines from a manufacturer to wholesalers and retailers, stored as compressed columnstore on SQL Server and computed on the fly on MariaDB; plus a bill of materials. - [Utility field operations (fieldops)](https://learnsql.cpetoday.com/analyze/field-operations.md): A water and electric utility: assets, work orders with labor and materials, vendors with purchase orders and invoices, customer billing and service interruptions, all keyed. - [Maps and fiscal calendars](https://learnsql.cpetoday.com/analyze/maps-and-fiscal.md): Addresses with latitude and longitude, reference geography, and a fiscal calendar in every lab database: map it, chart it by fiscal period. ## Function reference Every function page is listed in https://learnsql.cpetoday.com/llms.txt. Each language has one Markdown index: - [DAX function index](https://learnsql.cpetoday.com/functions/dax/md): 174 functions in 15 categories - [Power Query M function index](https://learnsql.cpetoday.com/functions/m/md): 226 functions in 19 categories ## Tools Free, in-browser tools (no sign-in, nothing stored on the server). Each page explains the tool and links to the functions it uses: - [Chart lab: turn a SQL query into a chart](https://learnsql.cpetoday.com/tools/chart-lab): Pick a business question or write your own SELECT, run it on the sample databases, and see the result as a chart, a table and the SQL. - [Date table generator](https://learnsql.cpetoday.com/tools/date-table): Build a date table with calendar and fiscal columns, week numbers and working-day flags as Power Query M, DAX, T-SQL or MariaDB SQL. - [DAX formatter and formula checker](https://learnsql.cpetoday.com/tools/dax-formatter): Format a DAX formula into a standard layout and check it for common mistakes, all in your browser. - [Visual SQL query builder](https://learnsql.cpetoday.com/tools/sql-builder): Pick a database and tables, add joins, columns, filters and totals, and get the SQL for SQL Server or MariaDB, with a plain-English explanation and a live run. - [Data stories: six charts from real business data](https://learnsql.cpetoday.com/tools/data-stories): Six short stories, each with a live chart built from a real query on the sample data and the SQL underneath. - [Measure builder](https://learnsql.cpetoday.com/tools/measure-builder): Pick a business question, map its columns to your model, and get the DAX measure, an equivalent T-SQL query and a Power Query step. - [Power Query M explainer](https://learnsql.cpetoday.com/tools/m-explainer): Paste a Power Query M query and get every step explained in plain English, with reference links, folding hints and a re-indent button. - [SQL recipes for accounting and finance](https://learnsql.cpetoday.com/tools/sql-recipes): Ready-to-run SQL for month-end close, receivables, payables, cash, margin, budgets, controls, payroll, nonprofit and inventory, on live sample data. - [SQL formatter and query outline](https://learnsql.cpetoday.com/tools/sql-formatter): Format a SQL Server or MariaDB query, see in plain words what it does, and get a list of common mistakes to check. ## Optional - [Sample databases](https://learnsql.cpetoday.com/databases): the databases the guides use - [Sitemap](https://learnsql.cpetoday.com/sitemap.xml) --- # How databases work, for finance professionals Your billing system, your bank feed and your general ledger are all databases. QuickBooks, Sage and most ERP systems keep their data in tables, and every report you run is a question asked of those tables. This guide explains the parts you touch every week: tables, keys, relationships, journal entries, transactions, indexes and views. The examples use a CPA firm's invoices, payments and general ledger from the `cpafirm` sample database, because the shapes are the same as in a small business or a controller's ledger. Every query was run against the LearnSQL sample data. Each block says which database it runs on. The queries use SQL Server (T-SQL), and the MariaDB differences are listed where they matter. For the query syntax itself, read [SQL essentials](/analyze/sql-essentials) next. ## A table is a list with fixed columns A table holds one kind of thing. Each row is one item, and each column is one field of that item. The `invoices` table has one row per invoice, and every row has the same columns. Three real rows look like this: | invoice_number | client | issue_date | due_date | status | total_amount | |---|---|---|---|---|---| | INV-2025-03477 | C-10001 Kim, Jennifer | 2025-10-08 | 2025-11-07 | Paid | 475.00 | | INV-2026-03847 | C-10207 Morris, Maria & Joseph | 2026-10-21 | 2026-11-20 | Sent | 945.00 | | INV-2026-03848 | C-10171 Young, Paul | 2026-10-22 | 2026-11-21 | Sent | 868.75 | Every column has a data type, and the type decides what the database lets you do with it: - **Whole numbers** (`client_id`, `invoice_id`) are for identifiers and counts. - **Text** (`invoice_number`, `status`) is for names and codes. - **Decimals** (`total_amount`) keep money exact to the cent. Store money as a decimal, not a floating-point number, which can drift by fractions of a cent. - **Dates** (`issue_date`) sort, subtract and filter correctly. A date stored as text does not. In text, `10/02/2026` sorts before `09/15/2026`, and the database cannot count the days between two text values. ## Keys: how rows point at each other A **primary key** is the column that identifies one row. In `invoices` it is `invoice_id`. The invoice number is unique too, but the database uses the id, because a number a person types can be corrected later and every reference would have to change with it. A **foreign key** is a column that holds another table's primary key. `invoices.client_id` holds the `client_id` of a row in `clients`. The database can enforce the link, so an invoice cannot point at a client who does not exist. The next query takes one invoice and follows its two foreign keys to the client and the invoice lines. Runs on `cpafirm`. ```sql SELECT i.invoice_number, c.client_name, i.total_amount, il.line_number, il.description, il.amount FROM invoices AS i JOIN clients AS c ON c.client_id = i.client_id JOIN invoice_lines AS il ON il.invoice_id = i.invoice_id WHERE i.invoice_number = 'INV-2026-03848' ORDER BY il.line_number; ``` **What it returns.** Two rows, one per line. The invoice total is 868.75, and the two lines add to it: 525.00 for a 1040 preparation and 343.75 for two hours of tax planning. The client name appears on both rows because it comes from one row in `clients`. The database did not copy it. ## Why an ERP is many tables and not one big sheet Picture the whole billing history in one spreadsheet. Each invoice row repeats the client's name, street address, city and email. Then the trouble starts: - A client moves offices. You must change every invoice row for that client. Miss one, and the same client appears at two addresses. - The firm renames the account "Accounts Receivable" to "Trade Receivables". If the account name is typed into every journal line, the old name is in tens of thousands of lines, and the report splits into two accounts. This problem is called **normalization**: store each fact once, in the table that owns it, and let other tables refer to it by key. The address lives in `clients` once. Each invoice holds only `client_id`. The journal lines hold only `gl_account_id`, and the account's name lives in `gl_accounts`. > Diagram: Left, one wide sheet that repeats the client name and address on every invoice row. Right, two linked tables where the client's address is stored once in the clients table and each invoice holds only the client_id. The next query counts journal lines by account name. The names come from one table, so a rename in `gl_accounts` changes every report that uses it. Runs on `cpafirm`. ```sql SELECT TOP 5 a.account_name, COUNT(*) AS journal_lines FROM gl_journal_lines AS l JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id GROUP BY a.account_name ORDER BY journal_lines DESC; ``` **What it returns.** Accounts Receivable has 42,354 lines, Operating Cash has 20,292, and Bookkeeping Revenue has 14,240. Each name is typed once in `gl_accounts`. MariaDB uses `LIMIT 5` in place of `TOP 5`. ## Relationships and how many rows match **Cardinality** is how many rows sit on each side of a relationship: - **One to many.** One client has many invoices. One invoice has many lines. This is the common case, and it is drawn with a "1" at the one side and a crow's-foot "many" mark at the other. - **Many to many.** One payment can pay several invoices, and one invoice can be paid by several payments. A pair of tables cannot hold that directly. The database uses a third table, a **bridge**, with one row per payment and invoice pair. In `cpafirm` that table is `payment_applications`. A useful check is an **orphan** test: find child rows whose parent is missing. A foreign key should prevent them, but a check you run yourself proves it. This query returns the invoices whose client is not in `clients`. Runs on `cpafirm`. ```sql SELECT COUNT(*) AS invoices_without_a_client FROM invoices AS i WHERE NOT EXISTS (SELECT 1 FROM clients AS c WHERE c.client_id = i.client_id); ``` **What it returns.** 0. Every invoice has a client. An orphan count above zero is a data problem to chase before you trust any total. > Exercise: How many invoices were paid by two or more payment applications? Return a single count. ## Journal entries: a header and lines that balance A journal entry has a **header** (date, source, memo) and **lines** (account, debit or credit). Each line is one side of the entry. The rule is simple: the debits and the credits in one entry are equal. When the billing system bills an invoice, it writes a journal entry with `source_type = 'Invoice'` and `source_id` set to the invoice. The invoice is the bill. The journal lines are where the ledger balance lives. Here is the entry for INV-2026-03848. Runs on `cpafirm`. ```sql SELECT a.account_number, a.account_name, l.debit, l.credit FROM gl_journal_entries AS e JOIN gl_journal_lines AS l ON l.journal_entry_id = e.journal_entry_id JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE e.source_type = 'Invoice' AND e.source_id = (SELECT invoice_id FROM invoices WHERE invoice_number = 'INV-2026-03848') ORDER BY l.journal_line_id; ``` **What it returns.** Three lines. Accounts Receivable (1200) is debited 868.75. Tax Preparation Revenue (4100) is credited 525.00, and Tax Planning Revenue (4200) is credited 343.75. The debit equals the two credits. > Diagram: A journal entry with a header line and three lines. Accounts Receivable is debited 868.75; Tax Preparation Revenue is credited 525.00 and Tax Planning Revenue is credited 343.75. Totals are equal. Two rules follow for any query you write: 1. **A balance is debits minus credits.** Assets and expenses are normally debit balances. Liabilities, equity and revenue are normally credit balances. 2. **Sum the journal lines, not the invoices, for ledger numbers.** An invoice total and a ledger total can differ by timing, adjustments or write-offs. Use the lines for anything that must agree with the books. The next control runs over every entry in the journal. It counts entries whose debits and credits differ. Runs on `cpafirm`. ```sql SELECT COUNT(*) AS entries_checked, SUM(CASE WHEN x.total_debit <> x.total_credit THEN 1 ELSE 0 END) AS unbalanced, SUM(x.total_debit) AS all_debits, SUM(x.total_credit) AS all_credits FROM (SELECT e.journal_entry_id, SUM(l.debit) AS total_debit, SUM(l.credit) AS total_credit FROM gl_journal_entries AS e JOIN gl_journal_lines AS l ON l.journal_entry_id = e.journal_entry_id GROUP BY e.journal_entry_id) AS x; ``` **What it returns.** 42,549 entries, none unbalanced, and total debits equal to total credits at 46,902,772.68. Run this control before any other ledger work. If it fails, nothing built on the journal can be trusted. > Exercise: How many journal entries have each number of lines? Show the number of lines and the number of entries, fewest lines first. ## Transactions: all of the entry, or none of it A **transaction** groups several writes into one unit. Either every write is saved, or none is. Posting an invoice means writing the invoice header, its lines and the journal entry. If the system fails after the header and before the lines, a transaction undoes the header as well. The database never keeps half an entry. > Diagram: Three writes in order: invoice header, invoice lines, journal entry. If all three succeed the transaction commits and all are kept. If the second write fails the transaction rolls back and none are kept. This is why a healthy ledger always balances. The software writes an entry as one unit, and the database refuses to leave a partial one behind. In SQL Server the statements are `BEGIN TRANSACTION`, `COMMIT` and `ROLLBACK`. MariaDB accepts `START TRANSACTION` as well as `BEGIN`. When you read a report, you read committed data. A query never sees half of an entry that is still being written. ## Read-only access is safe for reporting Give analysts read-only access to the database. They can run any `SELECT`, but they cannot change a balance, a status or an address. A mistake in a read-only query can give a wrong number, and you can fix it by running the query again. > **Watch out.** A mistake in an `UPDATE` changes the books. The LearnSQL sample databases work the same way. The sample API accepts `SELECT` statements only, so every query in this guide reads the data and writes nothing. Keep posting rights with a few people who need them, and reporting rights with everyone who builds reports. ## Indexes: how the database finds rows quickly An **index** is a sorted list that points at rows, like the index at the back of a book. Without one, the database reads every row to find the ones you asked for, which is a table scan. With an index on `client_id`, it jumps straight to one client's invoices. > Diagram: Left, without an index the database reads every invoice row to find the matches. Right, an index on client_id is a sorted list that jumps straight to the rows for one client. The database keeps the index up to date for you. The cost is that every insert or update must also update the index, so a table with many indexes writes a little more slowly. The next query lists the indexes on `invoices`. Runs on `cpafirm`. ```sql SELECT t.name AS table_name, i.name AS index_name, i.type_desc, i.is_unique FROM sys.indexes AS i JOIN sys.tables AS t ON t.object_id = i.object_id WHERE t.name = 'invoices' AND i.name IS NOT NULL ORDER BY i.name; ``` **What it returns.** The primary key (`pk_invoices`), an index on `client_id`, an index on `issue_date`, an index on `status`, and a unique index on `invoice_number`. The unique index does two jobs: it speeds up lookups by number, and it stops the same number from being entered twice. **Practical rule.** Filter on the raw column. Write `issue_date >= '2026-01-01' AND issue_date < '2027-01-01'` rather than `YEAR(issue_date) = 2026`. Wrapping the column in a function usually stops the database from using the index on it. MariaDB shows the same index list with `SHOW INDEX FROM invoices`. ## Views: a saved query with a name A **view** is a saved `SELECT` that you can query as if it were a table. `vw_ar_aging` in `cpafirm` builds the receivables aging: each client's open balance split into buckets by how many days past due it is. The next query asks which clients owe the most. Runs on `cpafirm`. ```sql SELECT TOP 5 client_name, current_amt, days_1_30, days_31_60, days_61_90, days_over_90, total_due FROM vw_ar_aging ORDER BY total_due DESC; ``` **What it returns.** Sammamish Tire & Brake LLC owes 8,895.00, of which 5,535.00 is 1 to 30 days past due. Cedar Freight Corp. owes 8,747.88, of which 337.88 is more than 90 days past due. Each row shows how late the money is, not only how much. A view is only as correct as its definition, so check it against the books. The next query adds up the view and then reads account 1200, Accounts Receivable, from the general ledger. Runs on `cpafirm`. ```sql SELECT (SELECT SUM(total_due) FROM vw_ar_aging) AS open_per_aging_view, (SELECT SUM(l.debit - l.credit) FROM gl_journal_lines AS l JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE a.account_number = 1200) AS receivable_balance_in_gl; ``` **What it returns.** Both numbers are 566,429.12. The subledger (the aging by client) agrees with the general ledger to the cent. > **Audit tip.** When they do not agree, look for a missing payment, an adjustment that was posted to the wrong account, or a view that filters out rows. ## Reading a schema diagram A **schema** is the map of tables, columns and keys. Most database tools draw it as an **entity relationship diagram** (ERD). Each box is a table. The top line names the table, and the lines below list its key columns. A line between two boxes is a relationship. The end with the "many" mark belongs to the child table, which holds the foreign key. > Diagram: Entity relationship diagram of the invoice tables: clients to invoices to invoice lines, invoice lines to engagements, engagements to services, and services to service lines, with the key columns named in each box. Read an ERD in this order: 1. **Find the primary keys.** Each table's own key is the one that identifies its rows. 2. **Follow the foreign keys.** The column marked FK points at another table's primary key. 3. **Read the marks at each end.** A "1" at one end and a "many" mark at the other means one parent to many children. You can list the keys in the database itself. This query returns the primary and foreign keys for `invoices` and `invoice_lines`. Runs on `cpafirm`. ```sql SELECT tc.TABLE_NAME, tc.CONSTRAINT_TYPE, kcu.COLUMN_NAME FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS kcu ON kcu.CONSTRAINT_NAME = tc.CONSTRAINT_NAME WHERE tc.TABLE_NAME IN ('invoices', 'invoice_lines') AND tc.CONSTRAINT_TYPE IN ('PRIMARY KEY', 'FOREIGN KEY') ORDER BY tc.TABLE_NAME, tc.CONSTRAINT_TYPE, kcu.COLUMN_NAME; ``` **What it returns.** For `invoices`, the primary key `invoice_id` and foreign keys to `client_id`, `billing_contact_id` and `issued_by`. For `invoice_lines`, the primary key `invoice_line_id` and foreign keys to `invoice_id` and `engagement_id`. The foreign keys are the answer to "how do I join this table?" ## Where the LearnSQL sample databases fit Each sample database teaches a different shape. Pick the one that matches the work you do. | Database | Shape | Use it to practice | |---|---|---| | `cpafirm` | Accounting: clients, invoices, payments, time entries, a general ledger | Receivables, billing, ledger tie-outs | | `nonprofit` | Fund accounting: a general ledger by fund, budgets, grants | Budget against actual by fund and account | | `qbo` | QuickBooks Online-style tables in the `QuickBooksOnline` schema (Invoices, JournalEntries, Accounts) | Extracts from a small business ledger | | `qbd` | QuickBooks Desktop-style tables in the `QuickBooks` schema | Desktop-style exports and aging | | `sage100` | Sage 100 table names (AR_Customer, AP_InvoiceHeader) in `dbo` | Real ERP table naming | | `contoso` | A star schema: `FactSales` with dimension tables | Power BI measures and model design | | `retailwh` | A star-schema warehouse with a monthly budget and snapshots | Budget against actual, and the modeling guide | `cpafirm`, `nonprofit`, `sage100` and `retailwh` keep their tables in `dbo`, so `invoices` and `dbo.invoices` are the same table. `qbo` and `contoso` use their own schema names, which is why the queries on them write `QuickBooksOnline.Invoices` or `contoso.FactSales`. ## Try it Run these on the databases named in each line. Read the output before you change anything. 1. **Runs on `cpafirm`.** List the columns of the `payments` table, in order. ```sql SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'payments' ORDER BY ORDINAL_POSITION; ``` 2. **Runs on `cpafirm`.** How did clients pay in fiscal 2026 (July 2025 to June 2026)? Check the payment method that brought in the most cash. ```sql SELECT payment_method, COUNT(*) AS payments, SUM(amount) AS received FROM payments WHERE fiscal_year = 2026 GROUP BY payment_method ORDER BY received DESC; ``` 3. **Runs on `cpafirm`.** Is any invoice missing its lines? The answer should be 0. If it is not, the invoice totals cannot be checked against the detail. ```sql SELECT COUNT(*) AS invoices_without_lines FROM invoices AS i WHERE NOT EXISTS (SELECT 1 FROM invoice_lines AS il WHERE il.invoice_id = i.invoice_id); ``` Next, read [SQL for accountants](/analyze/sql-for-accountants) for the close queries that build on these tables, and [SQL essentials](/analyze/sql-essentials) for the syntax behind every example here. --- # From Excel to SQL: what you already know If you can build a pivot table, you already know most of SQL. Every query here is a task you do in Excel today, written the way a database reads it. Nothing is new except the syntax. The examples use the `cpafirm` sample database: a CPA firm's invoices, payments, invoice lines and general ledger. These are the tables you would otherwise export to a sheet. Each block says which database it runs on, and every result was checked against the sample data before it was published. The queries use SQL Server (T-SQL), with MariaDB notes where the syntax changes. New to the database idea? Read [How databases work](/analyze/how-databases-work) first. ## The cheat sheet | In Excel | In SQL | Section | |---|---|---| | AutoFilter, criteria | `WHERE` | [Filter](#filter-where) | | Sort A to Z | `ORDER BY` | [Sort](#sort-order-by) | | Data, Remove Duplicates | `DISTINCT` | [Duplicates](#remove-duplicates-distinct) | | `VLOOKUP`, `XLOOKUP`, `INDEX`/`MATCH` | `JOIN` | [Lookups](#vlookup-xlookup-and-index-match-join) | | `#N/A` when nothing matches | `LEFT JOIN` and `NULL` | [No match](#na-left-join-and-null) | | `SUMIFS`, `COUNTIFS` | `GROUP BY` with `SUM` and `COUNT` | [Summing](#sumifs-and-countifs-group-by) | | PivotTable | `GROUP BY`, with `CASE` for the columns | [Pivot](#pivottable-group-by-and-case) | | `IF`, `IFS` | `CASE WHEN` | [IF](#if-and-ifs-case) | | `IFERROR` | `COALESCE` and `NULLIF` | [IFERROR](#iferror-coalesce-and-nullif) | | `LEFT`, `MID`, `TRIM`, `&` | `LEFT`, `SUBSTRING`, `TRIM`, `CONCAT` | [Text](#text-left-substring-trim-concat) | | `EOMONTH`, `YEAR`, date subtraction | `EOMONTH`, `YEAR`, `DATEDIFF` | [Dates](#dates-eomonth-year-datediff) | | Running total with `$B$2:B2` | `SUM() OVER` | [Running totals](#running-totals-sum-over) | | `RANK.EQ` | `RANK() OVER` | [Rank](#rank-rank-over) | | Data, Subtotal | `GROUP BY ROLLUP` | [Subtotals](#subtotals-group-by-rollup) | ## A sheet is a table A sheet with a header row is already a table. The headers are the column names, each row is one record, and a formula bar is a query. The database adds three things a sheet does not have. Every column has a fixed data type, so a date is always a date. The table can hold far more rows than a sheet. And the query reads only what it asks for, so it does not open the whole file. The sample tables are bigger than a comfortable spreadsheet. `invoices` has 21,282 rows. `payments` has 20,222. The journal, `gl_journal_lines`, has 93,613 lines. An Excel worksheet holds 1,048,576 rows, so all of these fit on one sheet, but a sheet of that size is slow to filter, sort and recalculate. The database has no such limit to hit. ## Filter: WHERE In Excel you set an AutoFilter: Status equals "Partial", and Total is at least 1,000. In SQL the same criteria go in `WHERE`, and `AND` joins them. Runs on `cpafirm`. ```sql SELECT invoice_number, client_id, issue_date, status, total_amount FROM invoices WHERE status = 'Partial' AND total_amount >= 1000 ORDER BY total_amount DESC; ``` **What it returns.** The largest partly paid invoice is INV-2024-02815 for 7,905.00, then INV-2023-02833 for 5,740.00 and INV-2025-04157 for 4,700.00. The filter is the same one you would set by hand. | In Excel | In SQL | |---|---| | Text equals "Partial" | `status = 'Partial'` | | Number at least 1,000 | `total_amount >= 1000` | | Begins with "INV-2026" | `invoice_number LIKE 'INV-2026%'` | | Contains "Trust" | `payment_method LIKE '%Trust%'` | | Is not blank | `client_id IS NOT NULL` | Note the last row. > **Watch out.** In SQL a blank cell is `NULL`, and `= NULL` never matches. Test for it with `IS NULL` or `IS NOT NULL`. > Exercise: List the invoices with the status 'Partial' and a total of at least 1,000. Show the invoice number, client id, issue date and total, sorted by invoice number. ## Sort: ORDER BY `ORDER BY` is Sort A to Z. Add `DESC` for largest first. Without it, the database returns rows in no particular order. A sheet keeps the order you left it in, but a query has no memory of your last sort. Always put an `ORDER BY` on a result you will read or copy. ## Remove duplicates: DISTINCT Excel's Remove Duplicates keeps one row per value in the columns you choose. `SELECT DISTINCT` does the same. If you list two columns, the database keeps one row per unique pair, which is the same as choosing both columns in Excel. To count how many different clients paid in fiscal 2026 (July 2025 to June 2026), remove duplicates and count what is left. Runs on `cpafirm`. ```sql SELECT COUNT(*) AS clients_who_paid FROM (SELECT DISTINCT client_id FROM payments WHERE fiscal_year = 2026) AS paid; ``` **What it returns.** 753 distinct clients with at least one payment in the year. A row count of payments would be much larger, because most clients pay several times. ## VLOOKUP, XLOOKUP and INDEX-MATCH: JOIN Your lookup formula reads a key from one sheet, finds the matching row on another, and returns a column from it. In SQL this is a `JOIN`. The join condition is the lookup key, and the columns you select can come from either table. Here is the invoice register with the client's name, for the largest invoices of fiscal 2026. The Excel version would be `=VLOOKUP(B2, Clients!A:C, 2, FALSE)` in a helper column. Runs on `cpafirm`. ```sql SELECT TOP 5 i.invoice_number, i.issue_date, i.total_amount, c.client_name FROM invoices AS i JOIN clients AS c ON c.client_id = i.client_id WHERE i.fiscal_year = 2026 ORDER BY i.total_amount DESC; ``` **What it returns.** Five invoices, each with the name of its client. The client name is stored once in `clients`, and the join brings it across. MariaDB uses `LIMIT 5` in place of `TOP 5`. One difference matters. `VLOOKUP` returns the first match. A join returns every match. If the lookup table has the same key twice, the join repeats the invoice. `clients.client_id` is the primary key, so each invoice finds exactly one client. Check your lookup keys for duplicates before you join. > Diagram: Two tables matched on client_id. The clients table holds client 171 Young, Paul and client 1 Kim, Jennifer. The invoices table holds INV-2026-03848 for client 171 and INV-2025-03477 for client 1. The result adds the client name to each invoice. ## #N/A: LEFT JOIN and NULL When `VLOOKUP` finds nothing, Excel shows `#N/A`. > **Watch out.** An inner join drops the row instead, so the item disappears from the report without a warning. A `LEFT JOIN` keeps every row from the left table and fills the right side with `NULL`, the database word for "no value". The next query lists four clients and what each was billed in fiscal 2026. Two of them have no invoices that year. Runs on `cpafirm`. ```sql SELECT c.client_name, COUNT(i.invoice_id) AS invoices_fy2026, SUM(i.total_amount) AS billed_fy2026 FROM clients AS c LEFT JOIN invoices AS i ON i.client_id = c.client_id AND i.fiscal_year = 2026 WHERE c.client_id IN (1, 2, 3, 4) GROUP BY c.client_name ORDER BY c.client_name; ``` **What it returns.** Kim, Jennifer: 2 invoices, 950.00. Martin, Maria & Edward: 1 invoice, 600.00. Smith, Angela & Anthony: 0 invoices, and `billed_fy2026` is `NULL`. Wright, Jeffrey: 1 invoice, 1,580.00. The client stays in the list, which is what you want for a report of who was billed and who was not. > Diagram: Four clients on the left. Three have matching fiscal 2026 invoices and show a count. Smith, Angela and Anthony has no match, so its invoice count is blank, shown as NULL, and the client still appears in the result. **Put the year in the `ON` clause.** If you move `AND i.fiscal_year = 2026` into `WHERE`, the rows with no invoices are removed and the `LEFT JOIN` behaves like an inner join. This is the most common mistake in lookup-style reports. Across the whole client list, 239 clients had no invoice at all in fiscal 2026. That is a question for the partner, not a lookup error. ## SUMIFS and COUNTIFS: GROUP BY `SUMIFS` adds the amounts that match the criteria. In SQL, you name the criteria as the groups and let `SUM` add up each group. `GROUP BY` replaces the list of formulas you would otherwise copy down a column. Here is fiscal 2026 billing by service line. The amounts sit on `invoice_lines`, and the service line is three links away, so the query walks them in order: line to engagement, engagement to service, and service to service line. That walk is the chain of helper columns you would add in Excel. Runs on `cpafirm`. ```sql SELECT sl.line_name AS service_line, SUM(il.amount) AS revenue, COUNT(*) AS invoice_lines FROM invoice_lines AS il JOIN invoices AS i ON i.invoice_id = il.invoice_id JOIN engagements AS e ON e.engagement_id = il.engagement_id JOIN services AS s ON s.service_id = e.service_id JOIN service_lines AS sl ON sl.service_line_id = s.service_line_id WHERE i.fiscal_year = 2026 GROUP BY sl.line_name ORDER BY revenue DESC; ``` **What it returns.** Six rows. Bookkeeping is 1,395,332.50, Tax Preparation 1,103,875.00, Payroll 494,480.00, Assurance 440,800.00, Advisory 349,833.75 and Tax Planning 135,356.25. The six add to 3,919,677.50, which is exactly the fiscal 2026 invoice total in `invoices`. A total that agrees with the source is a control, and you should check for one every time. > Diagram: Many invoice lines, each tagged with a service line, collapse into one row per service line. Bookkeeping sums to 1,395,332.50, Tax Preparation to 1,103,875.00 and Payroll to 494,480.00. `COUNTIFS` works the same way. Put every criterion in `WHERE` and count the rows. This query counts invoices by status and totals each group. Runs on `cpafirm`. ```sql SELECT status, COUNT(*) AS invoices, SUM(total_amount) AS amount FROM invoices GROUP BY status ORDER BY invoices DESC; ``` **What it returns.** Paid 20,330 invoices for 16,257,058.75. Sent 393 for 373,747.50. Written Off 294 for 223,037.50. Partial 136 for 140,213.75. Overdue 129 for 129,780.00. ## PivotTable: GROUP BY and CASE A PivotTable puts one field down the rows, another across the columns, and sums a value into each cell. SQL does the row side with `GROUP BY`. For the columns, you write one `CASE` expression for each column you want. The query below pivots fiscal 2026 billing by service line and fiscal quarter. Runs on `cpafirm`. ```sql SELECT sl.line_name AS service_line, SUM(CASE WHEN i.fiscal_quarter = 1 THEN il.amount ELSE 0 END) AS q1, SUM(CASE WHEN i.fiscal_quarter = 2 THEN il.amount ELSE 0 END) AS q2, SUM(CASE WHEN i.fiscal_quarter = 3 THEN il.amount ELSE 0 END) AS q3, SUM(CASE WHEN i.fiscal_quarter = 4 THEN il.amount ELSE 0 END) AS q4, SUM(il.amount) AS fiscal_year_total FROM invoice_lines AS il JOIN invoices AS i ON i.invoice_id = il.invoice_id JOIN engagements AS e ON e.engagement_id = il.engagement_id JOIN services AS s ON s.service_id = e.service_id JOIN service_lines AS sl ON sl.service_line_id = s.service_line_id WHERE i.fiscal_year = 2026 GROUP BY sl.line_name ORDER BY fiscal_year_total DESC; ``` **What it returns.** One row per service line, with the four quarters side by side. Across each row, q1 to q4 add up to `fiscal_year_total`. Check that on every pivot you build. A pivot with a quarter column missing drops that quarter's revenue, and the total will not match the source. The difference from Excel: a PivotTable adds a new column when a new quarter appears in the data. The SQL column list is fixed, so a new column needs a new line. For a fixed report, that is a feature, because the layout cannot change under a reader. > Exercise: Pivot the invoices by status for the whole file: one row per status with the number of invoices and the total amount rounded to cents. Largest total first. ## IF and IFS: CASE `IF` and `IFS` choose a label or a value from a list of conditions. `CASE WHEN` is the same idea. It checks each condition in order and returns the value for the first one that is true, the same as `IFS`. `ELSE` is the final `TRUE` branch. The example classifies each payment against its invoice due date, for payments made in fiscal 2026. Runs on `cpafirm`. ```sql WITH timed AS ( SELECT CASE WHEN p.payment_date <= i.due_date THEN 'On time' WHEN DATEDIFF(day, i.due_date, p.payment_date) <= 30 THEN '1 to 30 days late' ELSE 'More than 30 days late' END AS timing FROM payment_applications AS pa JOIN payments AS p ON p.payment_id = pa.payment_id JOIN invoices AS i ON i.invoice_id = pa.invoice_id WHERE p.fiscal_year = 2026 ) SELECT timing, COUNT(*) AS payment_applications FROM timed GROUP BY timing ORDER BY payment_applications DESC; ``` **What it returns.** Of the 4,471 payment applications in fiscal 2026, 1,584 were on time, 2,447 were 1 to 30 days late and 440 were more than 30 days late. Most payments arrive a little after the due date, which is useful to know before you set a collections rule. The `WITH` block is a common table expression. It names the calculated `timing` column once, so the outer query can group by it. Without it, you would repeat the whole `CASE` expression inside the `GROUP BY`. MariaDB runs the same query with one change: `DATEDIFF` takes its dates in the other order, so the second condition reads `DATEDIFF(p.payment_date, i.due_date) <= 30`. ## IFERROR: COALESCE and NULLIF `IFERROR(B2/C2, 0)` hides a divide-by-zero. SQL Server does not hide it: a divide by zero stops the whole query with an error. Use `NULLIF` to turn a zero denominator into `NULL`. Dividing by `NULL` gives `NULL`, so there is no error. `COALESCE` then turns that `NULL` into the value you want to show. The next query gives each client's average invoice in fiscal 2026. Runs on `cpafirm`. ```sql SELECT c.client_name, COUNT(i.invoice_id) AS invoices, COALESCE(SUM(i.total_amount) / NULLIF(COUNT(i.invoice_id), 0), 0) AS average_invoice FROM clients AS c LEFT JOIN invoices AS i ON i.client_id = c.client_id AND i.fiscal_year = 2026 WHERE c.client_id IN (1, 2, 3, 4) GROUP BY c.client_name ORDER BY c.client_name; ``` **What it returns.** Kim, Jennifer averages 475.00 over 2 invoices. Martin, Maria & Edward averages 600.00. Smith, Angela & Anthony has no invoices, and the average shows 0 rather than an error. Wright, Jeffrey averages 1,580.00. **Use zero with care.** A zero average for a client with no invoices can look like a real amount. When a blank is more honest, keep the `NULL`. ## Text: LEFT, SUBSTRING, TRIM, CONCAT These functions do what the text formulas do. `LEFT(text, n)` takes the first characters. `SUBSTRING(text, start, length)` is `MID`. `TRIM` removes leading and trailing spaces. `CONCAT` joins pieces of text, the same as `&`. The query below pulls the prefix and year out of each invoice number and builds a label from the client number and name. Runs on `cpafirm`. ```sql SELECT TOP 5 i.invoice_number, LEFT(i.invoice_number, 3) AS prefix, SUBSTRING(i.invoice_number, 5, 4) AS invoice_year, TRIM(c.client_name) AS client_name, CONCAT(c.client_number, ' ', c.client_name) AS label FROM invoices AS i JOIN clients AS c ON c.client_id = i.client_id ORDER BY i.invoice_id DESC; ``` **What it returns.** Five of the newest invoices. Each has the prefix INV, the year 2026 as text, and a label such as C-10207 Morris, Maria & Joseph. The year in the number matches the issue date on all 21,282 invoices. A check like that can catch a mis-keyed invoice, and this one returns zero mismatches. ## Dates: EOMONTH, YEAR, DATEDIFF In Excel you use `EOMONTH` for month-end, `YEAR` and `MONTH` to split a date, and subtraction for days. The database has the same functions. `EOMONTH` returns the last day of the month for the date it is given. `DATEDIFF(day, start, end)` returns the number of days between two dates. This query groups the fiscal 2026 receipts by month-end, the same as a pivot grouped by a month column. Runs on `cpafirm`. ```sql SELECT EOMONTH(payment_date) AS month_end, SUM(amount) AS receipts FROM payments WHERE fiscal_year = 2026 GROUP BY EOMONTH(payment_date) ORDER BY month_end; ``` **What it returns.** Twelve rows, one for each month from July 2025 to June 2026. July receipts are 278,746.69 and the heaviest month is April 2026 at 537,834.09. The year total is 3,783,708.16. MariaDB writes the month-end as `LAST_DAY(payment_date)`. Its `DATEDIFF` takes the two dates in the other order and returns days without a unit: `DATEDIFF(payment_date, due_date)` is the same as the SQL Server `DATEDIFF(day, due_date, payment_date)`. ## Running totals: SUM OVER Your running total is `=SUM($B$2:B2)`, copied down. The window function `SUM(...) OVER (ORDER BY ...)` does the same with no helper column. The `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` part says "from the first row down to this one". Runs on `cpafirm`. ```sql WITH monthly AS ( SELECT EOMONTH(payment_date) AS month_end, SUM(amount) AS receipts FROM payments WHERE fiscal_year = 2026 GROUP BY EOMONTH(payment_date) ) SELECT month_end, receipts, SUM(receipts) OVER (ORDER BY month_end ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total FROM monthly ORDER BY month_end; ``` **What it returns.** Each month shows its receipts and the total so far. The last row is the year total, 3,783,708.16. The running total is calculated after the grouping, so it works on the monthly figures rather than on the individual payments. > Diagram: Columns for receipts in July, August and September 2025, with a running total line climbing from 278,746.69 to 489,675.96 to 697,950.67. ## RANK: RANK() OVER `RANK.EQ` gives each value its position in a list. In SQL, `RANK() OVER (ORDER BY ...)` does the same. The query below ranks clients by fiscal 2026 billing. Runs on `cpafirm`. ```sql WITH billed AS ( SELECT c.client_id, c.client_name, SUM(i.total_amount) AS billed_fy2026 FROM clients AS c JOIN invoices AS i ON i.client_id = c.client_id WHERE i.fiscal_year = 2026 GROUP BY c.client_id, c.client_name ) SELECT TOP 5 RANK() OVER (ORDER BY billed_fy2026 DESC) AS place, client_name, billed_fy2026 FROM billed ORDER BY place; ``` **What it returns.** Watson Fitness Corp. first at 65,775.00, then Sammamish Tire & Brake LLC at 61,267.50, The Juniper Shelf Corp. at 44,970.00, Larson Construction Inc. at 40,570.00 and Boise Chiropractic Inc. at 38,712.50. Ties need a choice. `RANK` gives tied values the same place and then skips the next place. `DENSE_RANK` does not skip. `ROW_NUMBER` gives every row a different number even when values tie. > Exercise: Which three clients have the most invoices in total? Show the client name and the number of invoices, most first. ## Subtotals: GROUP BY ROLLUP Excel's Data, Subtotal inserts a subtotal line for each group and a grand total at the bottom. `GROUP BY ROLLUP` does the same in one query. The `GROUPING` function marks the subtotal rows so you can label them. This is the revenue by service line and service, for fiscal 2026. Runs on `cpafirm`. ```sql SELECT CASE WHEN GROUPING(sl.line_name) = 1 THEN 'All service lines' ELSE sl.line_name END AS service_line, CASE WHEN GROUPING(sl.line_name) = 1 THEN '' WHEN GROUPING(s.service_name) = 1 THEN 'Subtotal' ELSE s.service_name END AS service, SUM(il.amount) AS revenue FROM invoice_lines AS il JOIN invoices AS i ON i.invoice_id = il.invoice_id JOIN engagements AS e ON e.engagement_id = il.engagement_id JOIN services AS s ON s.service_id = e.service_id JOIN service_lines AS sl ON sl.service_line_id = s.service_line_id WHERE i.fiscal_year = 2026 GROUP BY ROLLUP (sl.line_name, s.service_name) ORDER BY GROUPING(sl.line_name), sl.line_name, GROUPING(s.service_name), s.service_name; ``` **What it returns.** Each service line, its services and a subtotal line after them. The last row is the grand total, 3,919,677.50, which again matches the invoice total. The `ORDER BY` puts the grand total last and each subtotal after its own detail rows. MariaDB writes `GROUP BY sl.line_name, s.service_name WITH ROLLUP`. Its subtotal rows show `NULL` in the grouped columns, so you label them in the select list. ## The spreadsheet with a million rows Excel can hold 1,048,576 rows, and past a certain size the file becomes the problem. It is slow to open, slow to recalculate, and hard to share, and every copy of it becomes a version. A database keeps the table in one place. You ask a question, and the server reads what the question needs and returns one answer. This query counts every journal line in the general ledger and shows the date range. It returns one row, and no file opens. Runs on `cpafirm`. ```sql SELECT COUNT(*) AS journal_lines, MIN(e.entry_date) AS first_entry, MAX(e.entry_date) AS last_entry FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id; ``` **What it returns.** 93,613 journal lines, from 2021-01-01 to 2026-10-24. The count is small enough to check in Excel, which is the point of the sample. The same query works unchanged when the table holds tens of millions of rows. For a worked example at that size, read [Big data: manufacturer orders](/analyze/big-orders). ## Try it Run these on `cpafirm`. Each one is a translation of a sheet task you already know. 1. **Filter and count.** How many clients have no invoices at all? Compare this with the 239 clients who had none in fiscal 2026. ```sql SELECT COUNT(*) AS clients_never_billed FROM clients AS c WHERE NOT EXISTS (SELECT 1 FROM invoices AS i WHERE i.client_id = c.client_id); ``` 2. **Group and count.** How many invoices were issued in each fiscal year? Sort the years so the newest is first. ```sql SELECT fiscal_year, COUNT(*) AS invoices FROM invoices GROUP BY fiscal_year ORDER BY fiscal_year DESC; ``` 3. **Pivot.** Change the pivot query above so it shows the four quarters of fiscal 2026 for each service line, and then add a column that shows each line's share of the year total. Use `NULLIF` on the year total to protect the division. For the query building blocks behind all of these, read [SQL essentials](/analyze/sql-essentials). For the close queries that use the same tables, read [SQL for accountants](/analyze/sql-for-accountants). --- # SQL essentials: read data, join it, summarize it This guide teaches the SQL you use every day, with the LearnSQL sample databases. Run every example in the [SQL editor](/editor) or the [Playground](/playground). The examples use `classicmodels`, a model-car wholesaler: `customers`, `orders`, `orderdetails`, `products`, `productlines`, `employees`, `offices` and `payments`. SQL Server is the primary engine here, and MariaDB is the second. They share most of SQL but not all of it. A table at the end lists the differences. Every query in this guide is a read-only `SELECT`. ## The shape of a query ```sql SELECT column_list FROM table WHERE row_filter GROUP BY grouping_columns HAVING group_filter ORDER BY sort_columns ``` You write it in that order. The database **runs** it in a different order: `FROM`, `WHERE`, `GROUP BY`, `HAVING`, `SELECT`, `ORDER BY`. That is why you cannot use a `SELECT` alias in `WHERE`, but you can in `ORDER BY`. > Diagram: The six clauses of a query in the order you write them, and the order the database runs them: FROM, WHERE, GROUP BY, HAVING, SELECT, then ORDER BY. ## Read rows ```sql -- SQL Server SELECT TOP 10 customerName, country, creditLimit FROM classicmodels.customers ORDER BY creditLimit DESC; ``` ```sql -- MariaDB SELECT customerName, country, creditLimit FROM customers ORDER BY creditLimit DESC LIMIT 10; ``` - `SELECT *` is fine for a look, bad for a report. Name the columns you need. - A SQL Server table lives in a **schema**. In the sample databases the schema is named after the database, so write `classicmodels.customers`. - `ORDER BY` is the only thing that guarantees order. Without it a result set has **no** order, even if it looks sorted. - `DISTINCT` removes duplicate rows: `SELECT DISTINCT country FROM classicmodels.customers`. > Exercise: List the five customers with the highest credit limits. Show the customer name, the country and the credit limit, highest limit first. ## Filter with WHERE ```sql SELECT customerName, country, creditLimit FROM classicmodels.customers WHERE country = 'USA' AND creditLimit >= 100000 AND state IN ('CA', 'NY') AND customerName LIKE 'A%'; ``` | Test | Example | |---|---| | Equal, not equal | `=`, `<>` (or `!=`) | | Range | `BETWEEN 1000 AND 5000` (includes both ends) | | List | `IN ('CA', 'NY')` | | Pattern | `LIKE 'A%'` (starts with A), `LIKE '%shop%'`, `_` is one character | | Combine | `AND`, `OR`, `NOT`. Use parentheses when you mix `AND` and `OR`. | ### NULL is not a value > **Watch out.** `NULL` means unknown. Comparing anything to `NULL` with `=` gives unknown, not true, so the row is dropped. ```sql SELECT customerName FROM classicmodels.customers WHERE state IS NULL; -- right SELECT customerName FROM classicmodels.customers WHERE state = NULL; -- returns nothing ``` Handle it on purpose: `state IS NOT NULL`, or give a default with `COALESCE(state, 'n/a')` (works in both engines). > **Tip.** Aggregates such as `SUM` and `AVG` ignore `NULL`; `COUNT(*)` counts rows and `COUNT(column)` counts non-null values. ## Calculate and label with CASE ```sql SELECT customerName, creditLimit, CASE WHEN creditLimit >= 100000 THEN 'High' WHEN creditLimit >= 50000 THEN 'Medium' ELSE 'Low' END AS creditBand FROM classicmodels.customers; ``` `CASE` checks its `WHEN`s from the top and stops at the first match. Without `ELSE` an unmatched row gets `NULL`. > Exercise: Count the customers in each credit band. High is a limit of 100000 or more, Medium is 50000 up to 100000, and Low is under 50000. Show the band and the number of customers. ## Join tables A join puts rows from two tables side by side when a key matches. Use the **foreign keys**: the [Schema explorer](/relationships) shows them as lines. > Diagram: Two small tables, Vendors and Invoices, are joined on VendorID. Only invoices with a matching vendor appear in the result, with the vendor name added. ```sql SELECT o.orderNumber, o.orderDate, c.customerName FROM classicmodels.orders AS o JOIN classicmodels.customers AS c ON c.customerNumber = o.customerNumber; ``` | Join | Keeps | |---|---| | `INNER JOIN` (or `JOIN`) | Only rows that match on both sides | | `LEFT JOIN` | Every row on the left, with `NULL` where nothing matches | | `RIGHT JOIN` | The mirror image. Most people swap the tables and use `LEFT`. | | `FULL OUTER JOIN` | Everything from both sides (SQL Server; MariaDB has no `FULL` join) | | `CROSS JOIN` | Every pairing. Rarely what you want. | > Diagram: Customers and Orders joined two ways. An inner join keeps only customers with orders. A left join keeps every customer and shows NULL for Pine Group, which has no orders. **Find the missing ones** (an anti-join): customers with no orders. ```sql SELECT c.customerName FROM classicmodels.customers AS c LEFT JOIN classicmodels.orders AS o ON o.customerNumber = c.customerNumber WHERE o.orderNumber IS NULL; ``` Three habits prevent most join bugs: 1. **Join on the declared key**, and use the full key when the key has two columns (`orderdetails` has `orderNumber` and `productCode`). 2. **Count before and after.** If a join makes the row count grow, one side is not unique on that key. 3. **Put filters on the right table in the `ON`, not the `WHERE`, for a `LEFT JOIN`.** A `WHERE` on the right table turns it into an inner join. ## Summarize with GROUP BY ```sql SELECT p.productLine, COUNT(*) AS lines, SUM(d.quantityOrdered) AS units, SUM(d.quantityOrdered * d.priceEach) AS sales FROM classicmodels.orderdetails AS d JOIN classicmodels.products AS p ON p.productCode = d.productCode GROUP BY p.productLine ORDER BY sales DESC; ``` > Diagram: Six general ledger lines grouped by account become three rows, one per account, with a line count and a total for each. - Every column in `SELECT` must be in `GROUP BY` or inside an aggregate (`SUM`, `COUNT`, `AVG`, `MIN`, `MAX`). - `WHERE` filters **rows before** grouping. `HAVING` filters **groups after** it: `HAVING SUM(...) > 100000`. - `COUNT(DISTINCT customerNumber)` counts unique customers. ## Dates | Goal | SQL Server | MariaDB | |---|---|---| | Today | `CAST(GETDATE() AS date)` | `CURDATE()` | | Year of a date | `YEAR(d)` | `YEAR(d)` | | First of the month | `DATEFROMPARTS(YEAR(d), MONTH(d), 1)` | `DATE_FORMAT(d, '%Y-%m-01')` | | Add days | `DATEADD(day, 30, d)` | `DATE_ADD(d, INTERVAL 30 DAY)` | | Days between | `DATEDIFF(day, a, b)` | `DATEDIFF(b, a)` | | Format as text | `FORMAT(d, 'yyyy-MM')` | `DATE_FORMAT(d, '%Y-%m')` | **Group sales by month** (SQL Server): ```sql SELECT DATEFROMPARTS(YEAR(o.orderDate), MONTH(o.orderDate), 1) AS month, SUM(d.quantityOrdered * d.priceEach) AS sales FROM classicmodels.orders AS o JOIN classicmodels.orderdetails AS d ON d.orderNumber = o.orderNumber GROUP BY DATEFROMPARTS(YEAR(o.orderDate), MONTH(o.orderDate), 1) ORDER BY month; ``` The sample dates are shifted every day so the latest order is today. Do not hard-code a date; ask the data for its own maximum: `SELECT MAX(orderDate) FROM classicmodels.orders`. > Exercise: For each order year, show the total sales (quantity times price, rounded to cents) and the number of orders. Oldest year first. ## Text ```sql UPPER(x) LOWER(x) LTRIM(x) RTRIM(x) LEN(x) -- SQL Server LENGTH(x) -- MariaDB SUBSTRING(x, 1, 3) REPLACE(x, 'a', 'b') CONCAT(a, ' ', b) ``` `CONCAT` ignores `NULL`s; `a + b` in SQL Server does not (the result is `NULL`). ## Subqueries and CTEs A **subquery** is a query inside a query. ```sql -- customers who spend more than the average payment SELECT customerNumber, SUM(amount) AS paid FROM classicmodels.payments GROUP BY customerNumber HAVING SUM(amount) > (SELECT AVG(amount) FROM classicmodels.payments); ``` A **CTE** (`WITH`) names a step so a long query reads top to bottom: ```sql WITH line_sales AS ( SELECT o.customerNumber, d.quantityOrdered * d.priceEach AS sales FROM classicmodels.orders AS o JOIN classicmodels.orderdetails AS d ON d.orderNumber = o.orderNumber ) SELECT TOP 5 customerNumber, SUM(sales) AS total FROM line_sales GROUP BY customerNumber ORDER BY total DESC; ``` Use `EXISTS` to test for related rows: `WHERE EXISTS (SELECT 1 FROM classicmodels.orders o WHERE o.customerNumber = c.customerNumber)`. It handles `NULL`s better than `NOT IN`, which returns nothing if the list contains a `NULL`. ## Combine result sets `UNION ALL` stacks results with the same columns and keeps duplicates. `UNION` removes duplicates (and is slower). `INTERSECT` and `EXCEPT` exist in SQL Server (MariaDB has `INTERSECT` and `EXCEPT` from 10.3). ## SQL Server and MariaDB at a glance | | SQL Server | MariaDB | |---|---|---| | Limit rows | `TOP n` or `OFFSET 0 ROWS FETCH NEXT n ROWS ONLY` | `LIMIT n` | | Quote a name | `[Order Details]` | `` `Order Details` `` | | Null default | `ISNULL(x, 0)`, `COALESCE` | `IFNULL(x, 0)`, `COALESCE` | | String join | `+` or `CONCAT` | `CONCAT` | | Current time | `GETDATE()`, `SYSDATETIME()` | `NOW()` | | Integer divide | `5 / 2` is `2` | `5 / 2` is `2.5000`, use `DIV` | | Case | Depends on collation, usually case-insensitive | Depends on collation, usually case-insensitive | ## Common mistakes | Mistake | Fix | |---|---| | Totals too big after a join | A join multiplied rows. Check key uniqueness, or aggregate each side first, then join. | | `WHERE x = NULL` | Use `IS NULL`. | | `HAVING` used to filter rows | Use `WHERE` for rows, `HAVING` for aggregates. | | Using `SELECT` alias in `WHERE` | Repeat the expression, or wrap in a CTE. | | `Invalid column name` | Check the schema prefix and the spelling, or an alias used before it exists. | | `LEFT JOIN` returns the same as `INNER` | A `WHERE` on the right table removed the unmatched rows. Move it to `ON`. | ## Where next [SQL deep end](/analyze/sql-deep-end) covers window functions, recursive queries, indexes and execution plans, and patterns such as deduplication and gaps. Then take what you built to a report in [DAX fundamentals](/analyze/fundamentals) or [Power Query M](/analyze/power-query-m-basics). --- # SQL deep end: windows, recursion, plans and patterns This guide is for when you know `SELECT`, `JOIN` and `GROUP BY` and want the techniques that separate quick queries from reliable ones. It assumes [SQL essentials](/analyze/sql-essentials). Examples use `classicmodels` on SQL Server; the notes mark where MariaDB differs. ## Window functions A **window function** calculates across a set of rows **without collapsing them**, so you keep the detail and add a number next to it. ```sql SELECT orderNumber, customerNumber, orderDate, ROW_NUMBER() OVER (PARTITION BY customerNumber ORDER BY orderDate) AS nth_order FROM classicmodels.orders; ``` `OVER (...)` defines the window: `PARTITION BY` splits rows into groups (like `GROUP BY`, but rows stay), and `ORDER BY` orders within each group. ### The four ranking functions | Function | Ties | Example values for 10, 10, 8 | |---|---|---| | `ROW_NUMBER()` | Always unique | 1, 2, 3 | | `RANK()` | Same rank, then a gap | 1, 1, 3 | | `DENSE_RANK()` | Same rank, no gap | 1, 1, 2 | | `NTILE(4)` | Splits rows into 4 buckets | quartiles | **Top 1 per group** (the best order per customer): rank inside a CTE, filter outside. A window function cannot be used in `WHERE`. ```sql WITH ranked AS ( SELECT o.customerNumber, o.orderNumber, SUM(d.quantityOrdered * d.priceEach) AS order_total, ROW_NUMBER() OVER (PARTITION BY o.customerNumber ORDER BY SUM(d.quantityOrdered * d.priceEach) DESC) AS rn FROM classicmodels.orders o JOIN classicmodels.orderdetails d ON d.orderNumber = o.orderNumber GROUP BY o.customerNumber, o.orderNumber ) SELECT customerNumber, orderNumber, order_total FROM ranked WHERE rn = 1; ``` ### Running totals and moving averages ```sql SELECT paymentDate, amount, SUM(amount) OVER (ORDER BY paymentDate, checkNumber ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, AVG(amount) OVER (ORDER BY paymentDate, checkNumber ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS moving_avg_7 FROM classicmodels.payments; ``` Always write the **frame** (`ROWS BETWEEN ...`) yourself. The default frame with an `ORDER BY` is `RANGE`, which treats rows with equal sort values as one group and can surprise you. Add a unique tie-breaker (`checkNumber`) to the order. ### Compare to the previous or next row ```sql SELECT customerNumber, paymentDate, amount, LAG(amount) OVER (PARTITION BY customerNumber ORDER BY paymentDate) AS prev_amount, amount - LAG(amount, 1, 0) OVER (PARTITION BY customerNumber ORDER BY paymentDate) AS change FROM classicmodels.payments; ``` `LAG(x, n, default)` looks `n` rows back; `LEAD` looks forward. `FIRST_VALUE` and `LAST_VALUE` return the first or last value of the window. For `LAST_VALUE` you must widen the frame to `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`, or it returns the current row. **Percent of total** is a window over everything: `amount * 1.0 / SUM(amount) OVER ()`. MariaDB supports window functions from version 10.2 with the same syntax. ## Recursive CTEs: trees and sequences `employees.reportsTo` points at another employee. A **recursive CTE** walks that tree. ```sql WITH org AS ( SELECT employeeNumber, lastName, reportsTo, 0 AS depth, CAST(lastName AS varchar(400)) AS path FROM classicmodels.employees WHERE reportsTo IS NULL UNION ALL SELECT e.employeeNumber, e.lastName, e.reportsTo, o.depth + 1, CAST(o.path + ' > ' + e.lastName AS varchar(400)) FROM classicmodels.employees e JOIN org o ON e.reportsTo = o.employeeNumber ) SELECT depth, path FROM org ORDER BY path; ``` The first `SELECT` is the **anchor** (the top of the tree). The second joins back to `org` itself to find the next level, and repeats until nothing new appears. SQL Server stops at 100 levels unless you add `OPTION (MAXRECURSION 0)`. On MariaDB write `WITH RECURSIVE org AS (...)` and use `CONCAT` for the path. A recursive CTE also generates a calendar: start from one date and add a day until you reach the end date. ## Reshape: pivot with conditional aggregation Turning rows into columns is the same everywhere with `CASE` inside an aggregate, and easy to read: ```sql SELECT p.productLine, SUM(CASE WHEN YEAR(o.orderDate) = 2025 THEN d.quantityOrdered * d.priceEach ELSE 0 END) AS sales_2025, SUM(CASE WHEN YEAR(o.orderDate) = 2026 THEN d.quantityOrdered * d.priceEach ELSE 0 END) AS sales_2026 FROM classicmodels.orders o JOIN classicmodels.orderdetails d ON d.orderNumber = o.orderNumber JOIN classicmodels.products p ON p.productCode = d.productCode GROUP BY p.productLine; ``` SQL Server also has `PIVOT` and `UNPIVOT`, and `CROSS APPLY (VALUES ...)` to unpivot columns into rows. ## Totals and subtotals: ROLLUP and GROUPING SETS ```sql -- SQL Server SELECT p.productLine, YEAR(o.orderDate) AS yr, SUM(d.quantityOrdered * d.priceEach) AS sales FROM classicmodels.orders o JOIN classicmodels.orderdetails d ON d.orderNumber = o.orderNumber JOIN classicmodels.products p ON p.productCode = d.productCode GROUP BY ROLLUP (p.productLine, YEAR(o.orderDate)); ``` `ROLLUP(a, b)` returns detail rows, a subtotal for each `a`, and a grand total. `GROUPING SETS` lets you list the exact combinations. Use `GROUPING(col)` to tell a subtotal's `NULL` from a real `NULL`. MariaDB writes it `GROUP BY a, b WITH ROLLUP`. ## Patterns worth knowing ### Deduplicate: keep one row per key ```sql WITH d AS ( SELECT *, ROW_NUMBER() OVER (PARTITION BY customerName ORDER BY customerNumber) AS rn FROM classicmodels.customers ) SELECT * FROM d WHERE rn = 1; ``` Changing the `ORDER BY` decides **which** duplicate survives (newest, biggest, first). ### Latest row per group `ROW_NUMBER() ... ORDER BY date DESC` then `rn = 1`, as above. The alternative, `WHERE date = (SELECT MAX(date) ...)` with a correlated subquery, returns ties as extra rows. ### Gaps and islands: find runs To group consecutive numbers or dates, subtract a row number: values in the same run share the same difference. ```sql WITH n AS ( SELECT DISTINCT CAST(orderDate AS date) AS d FROM classicmodels.orders ), g AS ( SELECT d, DATEADD(day, -ROW_NUMBER() OVER (ORDER BY d), d) AS grp FROM n ) SELECT MIN(d) AS run_start, MAX(d) AS run_end, COUNT(*) AS days FROM g GROUP BY grp ORDER BY run_start; ``` ### Top N per group without a window `SELECT TOP (3) ... CROSS APPLY` works on SQL Server: for each parent row, run a small ordered subquery. It reads well when the parent list is small. ### Always-correct "is there a match" Prefer `EXISTS` over `IN` for large or nullable lists, and `NOT EXISTS` over `NOT IN`. ## Performance: why a query is slow Most slow queries are slow for one of four reasons. 1. **Too many rows are read.** The database scanned a whole table because nothing helped it find the few rows you asked for. 2. **A function hides the column.** `WHERE YEAR(orderDate) = 2026` cannot use an index on `orderDate`. Write a range instead: `WHERE orderDate >= '2026-01-01' AND orderDate < '2027-01-01'`. A condition that can use an index is called **sargable**. 3. **A join multiplies rows** before you aggregate. Aggregate the many side first (in a CTE), then join. 4. **Sorting or grouping huge sets** that did not need to be that large. Filter and project early. ### Indexes in one minute An index is a sorted copy of one or more columns with pointers to the rows, like a book's index. It speeds up lookups, joins and sorts on those columns and slows writes slightly. A **composite** index on `(customerNumber, orderDate)` helps filters on `customerNumber` and on `customerNumber` plus `orderDate`, not on `orderDate` alone. The sample databases are read-only here, so you cannot create indexes, but you can read which ones exist and recognize what a plan does with them. ### Read an execution plan In SQL Server Management Studio or Azure Data Studio choose **Include Actual Execution Plan**. In MariaDB use `EXPLAIN`. Look for: | In the plan | Means | Often fixed by | |---|---|---| | Table or index **scan** on a big table | Whole table read | A filter that can use an index, a better index | | **Key lookup** repeated many times | Index found rows, then went back for more columns | Include the columns in the index, or select fewer | | **Hash match** or **sort** with a big memory grant | Large intermediate sets | Filter earlier, aggregate before joining | | Estimated rows far from actual rows | Stale statistics or a hard-to-estimate predicate | Update statistics, simplify the predicate | Use `SET STATISTICS IO, TIME ON` on SQL Server to see logical reads and time per query. > **Tip.** Fewer **logical reads** is the most reliable sign of a better query. ## Transactions and isolation (just enough) Each statement is atomic. A **transaction** groups statements so they all happen or none do. Readers can see unfinished changes at low isolation levels. On SQL Server, `READ COMMITTED SNAPSHOT` avoids readers waiting on writers. For reporting on a live system, read from a replica or a snapshot rather than adding `NOLOCK`, which can return duplicate or missing rows. ## Habits of reliable SQL - **Alias every table** and qualify every column. - **Name the key** in joins; never join "on whatever looks right". - **Check counts** at each step: rows in, rows out, distinct keys. - **Format for reading**: one clause per line, one join per line. - **Put business rules in one place** (a view or a CTE), then reuse it. - **Test on a small slice first**, then widen. ## Where next Take a query like these to Power BI: the same grouping becomes a measure in [DAX fundamentals](/analyze/fundamentals), and the same joins become relationships in [Relationships and the data model](/analyze/relationships). To shape data before it loads, see [Power Query M](/analyze/power-query-m-basics). --- # Power Query editor: clean data without code Most finance data arrives messy. A bank export has a title block above the header. A trial balance has group names that appear once and subtotals in the middle. A monthly report runs the months across the top. You can fix all of this by hand, but you will redo it every month. Power Query records each fix as a step. Next month you refresh, and every step runs again. This guide is for people who use Excel or Power BI and have never opened the editor. It does not teach M, the language behind it. You can use the editor for a long time without writing a line of M. Each click below shows the M it writes, so you can read it when you need to. The first guide in this track is this one. The accounting recipes are in [Power Query for accounting data](/analyze/power-query-accounting-data-prep), and the language itself is in [M basics](/analyze/power-query-m-basics). ## Where the editor lives **In Excel:** on the **Data** tab, choose **Get Data**, pick a source such as **From File > From Workbook** or **From Table/Range**, then **Transform Data**. The editor opens on that data. **In Power BI Desktop:** on the **Home** tab, choose **Transform data**. Either way, the editor opens on a copy of the data. Your original file is not changed until you load the result back. > Diagram: A simplified mockup of the Power Query editor window: a ribbon on top, a formula bar, a queries pane on the left, a preview grid in the middle and an Applied Steps pane on the right. ## The window, piece by piece - **Ribbon.** The tabs (Home, Transform, Add Column, View) hold every click in this guide. Most clicks are on Transform. - **Formula bar.** Shows the M for the step you have selected. You can edit it here, but you do not have to. - **Queries pane.** The list of queries on the left. Each one is a separate table. A bank export, a trial balance and a sales report are three queries, and they can feed each other. - **Preview grid.** The middle. It shows the result of the selected step, a few hundred rows at a time. - **Applied Steps.** The list on the right. Each click adds one step, in order. Select a step to see the table at that point. Delete or rename a step to change the chain. Applied Steps is the most important pane. Read it from top to bottom to see exactly how the data was cleaned. ## Three files you will meet The examples use three files. They are made-up examples in the shape of real exports, not files from the sample databases. Each has a problem that the editor can fix. **A bank export (CSV).** Four lines of account information sit above the header. Dates are text. | Row | Content | |---|---| | 1 to 4 | Account name, account number, period, blank line | | 5 | Date, Description, Debit, Credit, Balance | | 6 onward | 03/02/2026, Coffee shop, 4.75, (blank), 1,240.10 | **A trial balance (Excel).** The group name is typed once, in the first row of each group. Subtotal rows say "Total Cash". A blank row separates the sections. | Group | Account No | Account | Debit | Credit | |---|---|---|---|---| | Cash | 1000 | Operating | 272,404.68 | | | | 1010 | Savings | 900,000.00 | | | Total Cash | | | 1,172,404.68 | | | Revenue | 4000 | Gifts | | 900,000.00 | **A monthly sales report (Excel).** One row per product line, one column per month, and a total column at the end. | Product line | Jan | Feb | Mar | Total | |---|---|---|---|---| | Consulting | 12,400 | 13,100 | 11,900 | 37,400 | | Training | 4,200 | 3,800 | 5,100 | 13,100 | ## The 14 clicks Each click below has four parts: what it does, when to use it, the M it writes, and what to watch for. Some clicks are paired, because you use them together. ### 1. Promote headers **What it does.** Turns the first row into column names. **When.** The export has a header row that Excel or the file reader did not recognize, or the first row holds the names. ```powerquery Table.PromoteHeaders(Source, [PromoteAllScalars = true]) ``` **Watch for.** Promote after you remove the title block. If the title lines are still in place, the editor promotes the first title line as the header. ### 2. Change type, and set the locale **What it does.** Tells the editor what each column holds: date, number or text. **When.** Always. A date stored as text cannot be sorted or filtered by month, and a number stored as text will not add up. Set the type on every column you calculate with. ```powerquery Table.TransformColumnTypes(Source, {{"Date", type date}, {"Amount", type number}}) ``` > **Watch out.** A date of 03/04/2026 is March 4 in the US and April 3 in the UK. A decimal comma in a German file (1.234,56) reads as the wrong number unless the editor knows the locale. Use **Data Type > Using Locale** on the Transform tab and pick the file's locale. The M adds a culture argument: ```powerquery Table.TransformColumnTypes(Source, {{"Date", type date}}, "en-GB") ``` > **Watch out.** The editor guesses the type from the first rows. If a later row breaks the guess, you get an error in that cell. Check the column for errors after you set the type. ### 3. Remove top rows and blank rows **What it does.** Drops the title block, and drops the empty rows between sections. **When.** Every report export has some of each. ```powerquery Table.Skip(Source, 4) ``` Blank rows are removed with a filter. This is the M the editor writes for **Remove Blank Rows**: ```powerquery Table.SelectRows(Source, each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))) ``` **Watch for.** Remove the top rows before you promote headers. Count the title lines by looking at the preview, not from memory. ### 4. Split column **What it does.** Splits one column into two or more, on a separator. **When.** A bank description reads "Vendor name - Invoice 1042". You want the vendor and the memo in separate columns. ```powerquery Table.SplitColumn(Source, "Description", Splitter.SplitTextByDelimiter(" - ", QuoteStyle.Csv), {"Vendor", "Memo"}) ``` **Watch for.** Split on a separator the text uses every time. If one row has two hyphens, the split puts the second part in a third column. Check the number of new columns. ### 5. Trim and clean text **What it does.** Removes leading and trailing spaces, and non-printing characters that come from copying out of other systems. **When.** Vendor names look the same in the preview but do not match when you join or group. The usual cause is a trailing space. ```powerquery Table.TransformColumns(Source, {{"Vendor", each Text.Trim(Text.Clean(_)), type text}}) ``` **Watch for.** Trim fixes spaces, not spelling. "Larson Flooring" and "Larson Flooring Inc" stay different until you rename one of them. ### 6. Replace values **What it does.** Swaps one value for another, in one column or all of them. **When.** The export uses a dash for zero, or "N/A" for a blank. ```powerquery Table.ReplaceValue(Source, "-", null, Replacer.ReplaceValue, {"Amount"}) ``` **Watch for.** Replacing a value in an amount column with null makes the column blank, not zero. If you want zero, replace with 0 and set the type afterward. ### 7. Fill down **What it does.** Copies a value down into the blank cells below it, until the next value appears. **When.** The trial balance has the group name only in the first row of each group. Without fill down, the other rows have no group, and a group total will leave them out. > Diagram: A report-style export where the account group name appears only on the first row of each group. After Fill Down, every row carries its group name. ```powerquery Table.FillDown(Source, {"Group"}) ``` **Watch for.** Remove the subtotal rows before you fill down. A "Total Cash" row is copied into every blank row below it, so the group label becomes wrong. Remove the subtotals, then fill, then check the group totals against the report. ### 8. Unpivot columns **What it does.** Turns columns into rows. The month columns become one month column and one amount column. **When.** The monthly sales report has one column per month. A pivot table, a chart and a DAX measure all want one row per month. Unpivot first. > Diagram: A wide table with one column per month becomes a long table with one row per account and month. The wide table has three accounts and three months. ```powerquery Table.UnpivotOtherColumns(Source, {"Product line"}, "Month", "Amount") ``` The Total column comes through too. Remove it before you unpivot. If you leave it in, the total becomes a fourth month, and summing the Amount column counts every sale twice. **Watch for.** After unpivot, the Amount column is text. Set its type with click 2. ### 9. Merge queries **What it does.** Joins two tables on a shared key, like a lookup. It brings a column from one table into the other. **When.** The bank export has account numbers, and you want the account names from the chart of accounts. The merge is the lookup. ```powerquery Table.NestedJoin(Ledger, {"Account No"}, Accounts, {"Account No"}, "Accounts", JoinKind.LeftOuter) ``` Merge creates a column of tables. Expand it to pull out the column you want: ```powerquery Table.ExpandTableColumn(Merged, "Accounts", {"Account Name"}, {"Account Name"}) ``` **Watch for.** Use a left join to keep every ledger row even when no account matches. Then filter for nulls to find the missing accounts. An inner join drops those rows without warning. The join key must have the same type on both sides. A number in one query and text in the other will not match. ### 10. Append queries **What it does.** Stacks tables that have the same columns, one under another. **When.** You have one export per month, and you want one table for the year. > Diagram: Two operations. Append stacks three monthly exports into one table with the same columns. Merge joins a ledger table and a chart of accounts on the account number to add the account name. ```powerquery Table.Combine({January, February, March}) ``` **Watch for.** Append matches columns by name. A column called "Amt" in one export and "Amount" in another makes two columns, with blanks in each. Rename before you append. ### 11. Group by **What it does.** Summarizes rows, such as a total by account or a count by vendor. **When.** You need a total by account from the detail, for a check against the report. ```powerquery Table.Group(Source, {"Account No"}, {{"Total", each List.Sum([Amount]), type number}}) ``` **Watch for.** Group by the key only. If you add a column that is not in the key, the group splits into more rows than you expect. ### 12. Add conditional column **What it does.** Adds a column whose value depends on a test, such as "Review" when an amount is over a limit. **When.** You want to flag items for follow-up, such as entries over an approval limit. ```powerquery Table.AddColumn(Source, "Flag", each if [Amount] > 10000 then "Review" else "OK", type text) ``` **Watch for.** The limit is written into the step. When the limit changes, change the step, or use a parameter so it lives in one place. The [audit analytics guide](/analyze/audit-analytics-sql) uses the same kind of threshold test in SQL. ### 13. Combine files from a folder **What it does.** Reads every file in a folder and stacks them into one table. **When.** The bank sends one CSV per month to a folder. You want the whole year in one query, and next month's file should join on refresh. ```powerquery let Files = Folder.Files(FolderPath), CsvOnly = Table.SelectRows(Files, each [Extension] = ".csv"), Parsed = Table.AddColumn(CsvOnly, "Data", each Table.PromoteHeaders(Csv.Document([Content], [Delimiter = ",", Encoding = 65001]), [PromoteAllScalars = true])), Expanded = Table.ExpandTableColumn(Parsed, "Data", {"Date", "Account", "Amount"}, {"Date", "Account", "Amount"}), Typed = Table.TransformColumnTypes(Expanded, {{"Date", type date}, {"Amount", type number}}) in Typed ``` `FolderPath` is a parameter that holds the folder path. Set it once in **Manage Parameters**, not in each query. **Watch for.** Each file must have the same columns. The `Expanded` step names the columns, so a file with a renamed column causes an error on refresh. That error is the right result: it tells you the export changed. ### 14. Refresh **What it does.** Runs every step again on the current files. **When.** Each month, after you save the new file in the folder or the source updates. In Excel, choose **Data > Refresh All**. In Power BI Desktop, choose **Home > Refresh**. The editor's **Refresh Preview** button reruns the steps without loading the result. **Watch for.** A refresh reruns every step, so a step that depends on a fixed row count will break. Filter by content, such as a date or an account, not by position. ## Keep the steps readable Rename each step as you go. **Filtered Rows** tells you nothing a month later. **Removed Title Rows** tells you what happened. Right-click a step in Applied Steps and choose **Rename**. Do not delete a step in the middle to fix a problem. Change it, or add a step after it. Deleting a middle step can break every step below it, and the error appears in a place far from the cause. ## Where to go next - The recipes for real accounting files, including a bank statement and a trial balance with merged cells, are in [Power Query for accounting data](/analyze/power-query-accounting-data-prep). - For the language behind the editor, read [M basics](/analyze/power-query-m-basics), then [M tables](/analyze/power-query-m-tables) for the table functions shown above. - For patterns that scale to large files, read [M patterns](/analyze/power-query-m-patterns). - To use the cleaned data in a report, go to [Power BI for finance teams](/analyze/power-bi-for-finance). --- # M basics: queries, values, lists and records Every click you make in Power Query writes a line of **M**, the formula language behind Power Query in Excel, Power BI and Fabric. You can get a long way without reading it. You get much further once you can. This guide teaches M with the LearnSQL sample databases, so you can try the examples on real data. You do not need to install anything extra. In Excel choose **Data > Get Data > Launch Power Query Editor**. In Power BI Desktop choose **Transform data**. Then open **View > Advanced Editor** to see the M for the query you are on. To connect to the sample data, see [Connect Excel and Power BI](/connect). The server is `learnsql.cpetoday.com,1433`. In the examples the schema is named after the database (`classicmodels`); check the Navigator for the one your database uses. The examples use `classicmodels` and, for messy-data practice, `pqlab` (see the [Power Query playground](/analyze/power-query-lab)). This is part 1 of 4. Part 2 covers [tables](/analyze/power-query-m-tables). ## What a query looks like ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "classicmodels"), Orders = Source{[Schema = "classicmodels", Item = "orders"]}[Data], Shipped = Table.SelectRows(Orders, each [status] = "Shipped"), Result = Table.SelectColumns(Shipped, {"orderNumber", "orderDate", "customerNumber"}) in Result ``` Read it like this: - `let ... in` is the whole query. Between `let` and `in` you write **steps**, each one a name, an equals sign and an expression. After `in` you say which step is the answer. - Each step usually takes the **previous step** as its first argument. That chain is what the **Applied Steps** pane shows. - Steps are separated by commas. The last step before `in` has **no** comma. A stray or missing comma is the most common M error. - M is **case sensitive**. `Table.SelectRows` works, `table.selectrows` does not. Column names are case sensitive too: `[Status]` is not `[status]`. - Step names that contain spaces are written `#"Changed Type"`. This is why the editor writes `#"Removed Columns"` for you. If you name your own steps without spaces, you never need the quotes. - M is **lazy**. A step only runs if something needs it. A step that nothing uses is never evaluated, and that is also why a step can be slow without you noticing until you use its result. A query does not have to be a table. These are all valid queries: ```powerquery let Answer = 6 * 7 in Answer ``` ```powerquery "Hello, " & "world" ``` The second form has no `let`: a single expression is a complete query. > Diagram: A Power Query query runs in three stages: a source, then a list of applied steps, each one a line of M, then a load to the data model or a worksheet. ## Values and types | Type | Example | Notes | |---|---|---| | number | `42`, `3.14`, `1e3` | One type for integers and decimals. Use `Int64.Type` or `Currency.Type` only when you type a column. | | text | `"hello"` | Double quotes only. Double a quote to include one: `"say ""hi"""`. | | logical | `true`, `false` | Lower case. | | null | `null` | An unknown value. Most operations on `null` give `null`. | | date | `#date(2026, 3, 31)` | Year, month, day. | | time | `#time(14, 30, 0)` | | | datetime | `#datetime(2026, 3, 31, 14, 30, 0)` | | | datetimezone | `#datetimezone(2026, 3, 31, 14, 30, 0, -8, 0)` | Adds an offset from UTC. | | duration | `#duration(1, 2, 0, 0)` | Days, hours, minutes, seconds. | | list | `{1, 2, 3}` | Ordered values. | | record | `[Name = "Ann", Age = 41]` | A single row: field names and values. | | table | `#table({"a", "b"}, {{1, 2}, {3, 4}})` | Rows and columns. | | function | `(x) => x + 1` | Functions are values too. | Some rules worth knowing early: - Text and numbers do **not** mix. `"Total: " & 5` is an error. Write `"Total: " & Text.From(5)`. - Comparisons with `null` follow SQL's habit: `null = null` is `true` in M (unlike SQL), but `null > 1` is `null`. Test with `= null` or `<> null`, or give a default with `??` (below). - `??` returns the right side when the left side is `null`: `[discount] ?? 0`. - `and`, `or`, `not` are words, not symbols. `<>` means not equal. `&` joins text, lists and records. - Comments: `// to end of line` and `/* block */`. ### Setting a column's type Type a column with a **type ascription** inside `Table.TransformColumnTypes`: ```powerquery Typed = Table.TransformColumnTypes(Orders, {{"orderDate", type date}, {"customerNumber", Int64.Type}}) ``` `type text`, `type number`, `type date`, `type datetime`, `type logical`, `Int64.Type` and `Currency.Type` cover most needs. Set types **once, near the end**, after the cleanup steps that might otherwise error on bad values. Type conversion honors the culture of your PC, so a date like `03/04/2026` can read two ways. Use `Table.TransformColumnTypes(..., "en-US")` to be explicit. ## Lists A list is an ordered collection in braces. ```powerquery {1, 2, 3} {1..10} // 1 through 10 {"a".."e"} // "a", "b", "c", "d", "e" List.Numbers(0, 5, 10) // 0, 10, 20, 30, 40 (start, count, step) ``` Indexes start at **0**. `MyList{0}` is the first item. A position that does not exist is an error, unless you add `?`: `MyList{99}?` returns `null`. The list functions you will use most: | Goal | Function | |---|---| | Change each item | `List.Transform({1,2,3}, each _ * 10)` | | Keep matching items | `List.Select({1,2,3,4}, each _ > 2)` | | Total, average, min, max | `List.Sum`, `List.Average`, `List.Min`, `List.Max` | | Count | `List.Count`, `List.NonNullCount` | | Distinct, sorted | `List.Distinct`, `List.Sort` | | Join text | `Text.Combine({"a","b","c"}, ", ")` | | Combine two lists | `{1,2} & {3,4}` | | Roll a running value | `List.Accumulate` (see [functions and patterns](/analyze/power-query-m-patterns)) | A **column of a table is a list**. `Orders[orderNumber]` gives you all the order numbers as a list, so `List.Sum(Orderdetails[quantityOrdered])` totals a column in one line. ## Records A record is one row. ```powerquery let Person = [Name = "Ann", City = "Reno", Age = 41] in Person[City] // "Reno" ``` `Person[City]` reads a field. `Person[Country]?` returns `null` if the field is missing, instead of an error. Records appear everywhere in Power Query because **every row of a table is a record**. That is why `each [status] = "Shipped"` works: inside `each`, `[status]` means "the `status` field of the current row". ## `each` and `_` `each` is shorthand for a one-argument function whose argument is called `_`. ```powerquery each [qty] * [price] // same as: (_) => _[qty] * _[price] each _ * 10 // multiply the current list item each Text.Upper([name]) ``` Read `each` as "for each row (or item), do this". When you need to refer to an outer row from inside a nested `each`, name the argument yourself: ```powerquery (row) => List.Sum(List.Transform({1..3}, (n) => n * row[qty])) ``` --- Next: [M tables: shaping rows, columns, joins and groups](/analyze/power-query-m-tables). Series: [basics](/analyze/power-query-m-basics), [tables](/analyze/power-query-m-tables), [functions and patterns](/analyze/power-query-m-patterns), [performance](/analyze/power-query-m-performance). --- # M tables: shaping rows, columns, joins and groups Part 2 of 4. Examples use `classicmodels`; see [M basics](/analyze/power-query-m-basics) for how to connect and read a query. ## Tables Most of your M work is table functions. Here is the set that covers daily use. Each takes a table and returns a new table. ### Choose rows and columns ```powerquery Table.SelectRows(T, each [amount] > 1000 and [country] = "USA") Table.SelectColumns(T, {"orderNumber", "orderDate"}) Table.RemoveColumns(T, {"comments"}) Table.RenameColumns(T, {{"orderNumber", "Order #"}}) Table.ReorderColumns(T, {"orderNumber", "status"}) Table.Distinct(T, {"customerNumber"}) Table.Sort(T, {{"orderDate", Order.Descending}}) Table.FirstN(T, 10) Table.Skip(T, 2) Table.RemoveFirstN(T, 2) ``` ### Add and change columns ```powerquery Table.AddColumn(T, "Total", each [quantityOrdered] * [priceEach], type number) Table.TransformColumns(T, {{"status", Text.Upper, type text}}) Table.ReplaceValue(T, null, 0, Replacer.ReplaceValue, {"discount"}) Table.FillDown(T, {"region"}) ``` `Table.AddColumn` takes a name, a function for each row and an optional type. **Always give the type** as the fourth argument. Without it the new column is `any` and loads as text. ### Combine tables ```powerquery // Stack: same columns, more rows Table.Combine({Orders2025, Orders2026}) // Join (a "merge" in the UI): then expand the columns you want Merged = Table.NestedJoin(Orders, {"customerNumber"}, Customers, {"customerNumber"}, "Customer", JoinKind.LeftOuter), Expanded = Table.ExpandTableColumn(Merged, "Customer", {"customerName", "country"}, {"customerName", "country"}) ``` The join kinds are `JoinKind.Inner`, `LeftOuter`, `RightOuter`, `FullOuter`, `LeftAnti` and `RightAnti`. They match the SQL joins in the [Schema explorer](/relationships) Join Lab. Merge on **columns with the same data type**, or no rows will match. ### Group (the `GROUP BY` of Power Query) ```powerquery Table.Group( Lines, {"productCode"}, { {"Units", each List.Sum([quantityOrdered]), type number}, {"Lines", each Table.RowCount(_), Int64.Type} } ) ``` Inside a group, `_` is the sub-table for that group, so `Table.RowCount(_)` counts its rows. Use `Table.Group` instead of looping. ### Pivot and unpivot ```powerquery // Columns become rows: keep the ID columns, unpivot the rest Table.UnpivotOtherColumns(Wide, {"region"}, "Month", "Sales") // Rows become columns Table.Pivot(Long, List.Distinct(Long[Month]), "Month", "Sales", List.Sum) ``` Unpivoting is how you turn a spreadsheet-style report (months across the top) into a proper table. Practice it on `lab_piv_sales_wide` in `pqlab`. > Diagram: A wide table with a column for each month becomes a long table with one row for each region and month. Made-up numbers, for illustration. ### Headers and layout ```powerquery Table.PromoteHeaders(T, [PromoteAllScalars = true]) Table.DemoteHeaders(T) Table.Transpose(T) ``` ## Text, dates and numbers ```powerquery Text.Trim([name]) Text.Clean([name]) Text.Upper([x]) Text.Lower([x]) Text.Proper([x]) Text.Start([x], 3) Text.End([x], 4) Text.Middle([x], 2, 5) Text.Contains([x], "ship") Text.StartsWith([x], "A") Text.Replace([x], "-", "") Text.Length([x]) Text.Split([x], ",") Text.BeforeDelimiter([x], "@") Text.PadStart(Text.From([n]), 6, "0") // 42 -> "000042" ``` ```powerquery Date.From([stamp]) // datetime to date Date.Year([d]) Date.Month([d]) Date.Day([d]) Date.MonthName([d]) Date.DayOfWeekName([d]) Date.StartOfMonth([d]) Date.EndOfMonth([d]) Date.StartOfWeek([d]) Date.AddDays([d], 30) Date.AddMonths([d], -1) Date.ToText([d], "yyyy-MM") // text with a format Duration.Days([shipped] - [ordered]) // days between two dates Date.From(DateTime.LocalNow()) // today ``` > **Watch out.** Subtracting two dates gives a **duration**, not a number. Wrap it: `Duration.Days(...)`. ```powerquery Number.Round([x], 2) Number.RoundUp([x]) Number.RoundDown([x]) Number.From("12.5") Number.FromText("12.5") Text.From(12.5) Number.Mod(7, 3) Number.IntegerDivide(7, 3) ``` --- Previous: [M basics: queries, values, lists and records](/analyze/power-query-m-basics). Next: [M functions and patterns](/analyze/power-query-m-patterns). Series: [basics](/analyze/power-query-m-basics), [tables](/analyze/power-query-m-tables), [functions and patterns](/analyze/power-query-m-patterns), [performance](/analyze/power-query-m-performance). --- # M functions and patterns Part 3 of 4: your own functions, control flow, error handling, running totals, date tables and a complete worked example. ## Your own functions A function is `(inputs) => result`. ```powerquery let Tax = (amount as number, optional rate as nullable number) as number => amount * (rate ?? 0.08), Example = Tax(100) // 8 in Example ``` Types after `as` are optional but catch mistakes early. `optional` makes an argument skippable. **Turn a query into a function**, then call it on every row. This is how you read many files or call many pages: ```powerquery let OrdersFor = (cust as number) as table => Table.SelectRows(Orders, each [customerNumber] = cust), Customers = Table.SelectColumns(CustomersTable, {"customerNumber"}), WithOrders = Table.AddColumn(Customers, "Orders", each OrdersFor([customerNumber])) in WithOrders ``` > **Watch out.** Use this sparingly against a database. It calls the function once per row, which can turn one fast query into thousands of slow ones (see query folding below). Prefer a merge or `Table.Group` when you can. A function that calls itself needs `@` in front of its own name: `Factorial = (n) => if n <= 1 then 1 else n * @Factorial(n - 1)`. ## Control flow and patterns ### if, then, else ```powerquery Table.AddColumn(T, "Size", each if [amount] >= 5000 then "Large" else if [amount] >= 1000 then "Medium" else "Small", type text) ``` `else` is required. M has no `elseif`; chain `else if`. ### Handling errors `try ... otherwise` gives a fallback when an expression errors: ```powerquery Table.AddColumn(T, "Qty", each try Number.FromText([qty_text]) otherwise null, type nullable number) ``` `try` on its own returns a record `[HasError, Value]` or `[HasError, Error]`, which is how you can find out *why* something failed. To drop bad rows afterwards use `Table.RemoveRowsWithErrors(T, {"Qty"})`, and `Table.ReplaceErrorValues(T, {{"Qty", 0}})` to substitute a value. ### Running total with `List.Accumulate` ```powerquery let Sorted = Table.Sort(Source, {{"orderDate", Order.Ascending}}), Amounts = Sorted[amount], Steps = List.Accumulate(Amounts, {0}, (state, next) => state & {List.Last(state) + next}), Running = List.Skip(Steps, 1), Result = Table.FromColumns(Table.ToColumns(Sorted) & {Running}, Table.ColumnNames(Sorted) & {"RunningTotal"}) in Result ``` `List.Accumulate(list, seed, (state, item) => newState)` walks the list and carries a value forward. It is slow on very large lists. For big tables compute running totals in SQL or DAX instead. ### A date table ```powerquery let Start = #date(2024, 1, 1), End = #date(2026, 12, 31), Days = List.Dates(Start, Duration.Days(End - Start) + 1, #duration(1, 0, 0, 0)), AsTable = Table.FromList(Days, Splitter.SplitByNothing(), {"Date"}, null, ExtraValues.Error), Typed = Table.TransformColumnTypes(AsTable, {{"Date", type date}}), Year = Table.AddColumn(Typed, "Year", each Date.Year([Date]), Int64.Type), Month = Table.AddColumn(Year, "Month", each Date.Month([Date]), Int64.Type), MonthName = Table.AddColumn(Month, "Month name", each Date.MonthName([Date]), type text) in MonthName ``` A date table you build once is the basis of every [time intelligence](/analyze/time-intelligence) measure. ### Keep the latest row per customer ```powerquery let Sorted = Table.Buffer(Table.Sort(Payments, {{"paymentDate", Order.Descending}})), Latest = Table.Distinct(Sorted, {"customerNumber"}) in Latest ``` `Table.Buffer` pins the sorted table in memory. Without it `Table.Distinct` may keep a different row than the one you sorted first. Buffer only small or medium tables. ### Parameters A **parameter** (Home > Manage Parameters) is a named value you can reuse. In M a parameter is just a query holding a value with metadata, so you use it by name: `Table.SelectRows(Orders, each [orderDate] >= StartDate)`. Parameters let you change the date range once instead of editing every query. ## A complete worked example Sales by product line and year from `classicmodels`: three tables, a merge, a calculation, a group and a type. ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "classicmodels"), Orders = Source{[Schema = "classicmodels", Item = "orders"]}[Data], Lines = Source{[Schema = "classicmodels", Item = "orderdetails"]}[Data], Products = Source{[Schema = "classicmodels", Item = "products"]}[Data], // keep only what we need, as early as possible O = Table.SelectColumns(Orders, {"orderNumber", "orderDate", "status"}), L = Table.SelectColumns(Lines, {"orderNumber", "productCode", "quantityOrdered", "priceEach"}), P = Table.SelectColumns(Products, {"productCode", "productLine"}), // join the three LO = Table.NestedJoin(L, {"orderNumber"}, O, {"orderNumber"}, "Order", JoinKind.Inner), LOx = Table.ExpandTableColumn(LO, "Order", {"orderDate", "status"}), LP = Table.NestedJoin(LOx, {"productCode"}, P, {"productCode"}, "Product", JoinKind.Inner), LPx = Table.ExpandTableColumn(LP, "Product", {"productLine"}), // calculate and group Valid = Table.SelectRows(LPx, each [status] <> "Cancelled"), WithYear = Table.AddColumn(Valid, "Year", each Date.Year([orderDate]), Int64.Type), WithSales = Table.AddColumn(WithYear, "Sales", each [quantityOrdered] * [priceEach], Currency.Type), Grouped = Table.Group(WithSales, {"productLine", "Year"}, {{"Sales", each List.Sum([Sales]), Currency.Type}}), Sorted = Table.Sort(Grouped, {{"productLine", Order.Ascending}, {"Year", Order.Ascending}}) in Sorted ``` Notice the pattern: **select columns, join, filter, add, group, sort.** Nearly every report query has this shape. --- Previous: [M tables: shaping rows, columns, joins and groups](/analyze/power-query-m-tables). Next: [M performance, errors and when to use M](/analyze/power-query-m-performance). Series: [basics](/analyze/power-query-m-basics), [tables](/analyze/power-query-m-tables), [functions and patterns](/analyze/power-query-m-patterns), [performance](/analyze/power-query-m-performance). --- # M performance, errors and when to use M Part 4 of 4: query folding, the errors you will meet, how M compares with SQL and DAX, and where to practice. ## Query folding: why order matters When your source is a database, Power Query tries to **fold** your steps into one SQL query that runs on the server. Filters, column selection, simple group by, sort and joins usually fold. Then the database does the heavy work and sends back only the answer. Steps that cannot be written in SQL stop the folding. From that step on, Power Query pulls every row to your PC and works there. Common culprits: - a custom function or `each` that uses a function SQL has no match for; - `Table.Buffer`; - adding an index column, or `List.Accumulate`; - merging two queries that come from different sources. > Diagram: Steps that fold run as one SQL query on the server and return a small result. A step that does not fold, such as adding an index column, makes every row come to your PC. How to see it: 1. Right-click a step in Applied Steps. 2. If **View Native Query** is available, the step folds. If it is greyed out, folding stopped at or before that step. Rules that keep queries fast: 1. **Filter rows and remove columns first.** Both fold, and they shrink everything after. 2. **Put the steps that cannot fold last.** 3. **Avoid per-row functions on database tables** (see [functions and patterns](/analyze/power-query-m-patterns)). 4. For very large tables, write the join or aggregate in SQL and let Power Query load the result. `Value.NativeQuery` lets you send your own SQL: `Value.NativeQuery(Source, "SELECT TOP 100 * FROM orders")`. The sample databases are read-only, so use it for `SELECT`s. ## When something goes wrong | Message | What it means | Fix | |---|---|---| | `Expression.Error: The column 'X' of the table wasn't found.` | A step names a column that no longer exists or has different capitalization. | Check the spelling and case. Look at the step before. If the source column was renamed, change the name here. | | `DataFormat.Error: We couldn't convert to Number.` | A cell holds text such as `N/A` or `1,234 USD`. | Clean first with `Text.Replace`, then convert, or use `try ... otherwise null`. | | `Expression.Error: We cannot convert the value null to type Text.` | A text function got `null`. | Provide a default: `Text.Trim([name] ?? "")`. | | `Token Comma expected` / `Token Literal expected` | Missing or extra comma, bracket or quote. | Check the line above the one highlighted. | | `Expression.Error: The name 'X' wasn't recognized.` | Spelling, case, or the step is defined *below* where you use it. | Fix the name. The order of steps does not matter to M, but names must exist. | | `Formula.Firewall: Query references other queries or steps, so it may not directly access a data source.` | Two sources with different privacy levels are being mixed. | In **Data source settings**, set both to the same privacy level, or split the work into two queries. | | A merge returns no matches | Different types (text vs number) or stray spaces. | Match the types and `Text.Trim` the key. | ## M, DAX and SQL: which to use | Job | Best tool | Why | |---|---|---| | Clean and reshape data (split, unpivot, fix types, combine files) | **M** | Runs at refresh time; results are stored. | | Join and aggregate big tables | **SQL** (or M that folds) | The database is the fastest place to do it. | | Measures that react to slicers (totals, ratios, YTD) | **DAX** | Calculated when someone filters the report. See [DAX fundamentals](/analyze/fundamentals). | | A date table | **M** or DAX | Both work; see [M functions and patterns](/analyze/power-query-m-patterns). | | A calculation on one row | M column or DAX column | Prefer M or SQL; DAX columns increase model size. | A good rule: **shape the data in M or SQL, then calculate in DAX.** ## Practice with the sample data - The [Power Query playground](/analyze/power-query-lab) holds 86 tables built for practice, one per skill, each deliberately messy. - `lab_hdr_title_rows` is the first to try: remove top rows, promote headers. - `lab_piv_sales_wide` practices unpivoting. - `core_orders`, `core_customers` and `core_products` are a clean retail set for merges and groups. - After you can write a step by hand, build the same query in DAX in [Relationships and the data model](/analyze/relationships). Three habits make the biggest difference: **read the M the editor writes**, **filter early**, and **name your steps** so the Applied Steps list tells the story. --- Previous: [M functions and patterns](/analyze/power-query-m-patterns). Series: [basics](/analyze/power-query-m-basics), [tables](/analyze/power-query-m-tables), [functions and patterns](/analyze/power-query-m-patterns), [performance](/analyze/power-query-m-performance). --- # Data modeling for finance A data model is the set of tables you load into Power BI or Power Pivot and the relationships between them. A good model makes most measures one line long. A bad model gives totals that look right and are not. This guide covers the finance decisions behind a model: what one row means, how budgets and dates fit, how a chart of accounts becomes a statement, and the mistakes that double a total. The DAX side of these ideas is in [DAX for financial reporting](/analyze/dax-financial-reporting), and the shape of each sample database is in [Data structures and Power BI limits](/analyze/data-structures). Each query below runs on the sample database named in its block. The numbers were checked against that data before they were published. ## Facts and dimensions in the general ledger A **fact** table holds the measurements: one row per event, with the amounts. A **dimension** table holds the descriptions you filter and group by: the account, the fund, the date, the cost center. Together they form a star: a fact in the middle, dimensions around it, and one relationship per dimension. In the general ledger the fact is the journal line. Each line holds a date, an account, the fund and cost center it was charged to, and an amount in a debit or a credit column. The dimensions are the chart of accounts, the funds, the cost centers and the calendar. > Diagram: A star schema: a General Ledger lines fact table in the middle, joined one-to-many from four dimension tables: Date, Account, Department and Vendor. The `nonprofit` sample is already in this shape. Its journal line table carries `fund_id` and `cost_center_id` as well as `gl_account_id`, so the fund and the account both come from a dimension. The next query uses that. It totals FY2026 expense by net asset class, which is the restriction group of the fund: with or without donor restrictions. Runs on `nonprofit`. ```sql SELECT f.net_asset_class, SUM(l.debit - l.credit) AS expense_net FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id JOIN funds AS f ON f.fund_id = l.fund_id WHERE e.fiscal_year = 2026 AND a.account_type = 'Expense' GROUP BY f.net_asset_class ORDER BY expense_net DESC; ``` **What it returns.** Expenses without donor restrictions are 5,085,990.02 and expenses with donor restrictions are 2,228,961.07. The measure is one formula, debit minus credit, on the fact. The grouping comes from the dimension. That split is the design: the fact stays small and the descriptions stay in one place. ## Grain: decide what one row means The **grain** is what one row of a table represents. Write it in one sentence before you build anything: "One row is one journal line," or "One row is one budget amount for one fiscal year and account." If you cannot write that sentence, the table is not ready. Tables with different grains do not belong in one table, and comparing two grains needs care. These are the grains in the sample databases you will meet in this guide: | Table | One row is | Database | |---|---|---| | `gl_journal_lines` | one debit or credit on one account, fund and cost center, from one entry | `nonprofit`, `cpafirm` | | `budgets` | one budget amount for one fiscal year, account, fund and cost center | `nonprofit` | | `fact_budget` | one budget amount for one fiscal month, category and region | `retailwh` | | `fact_sales` | one line on a customer order | `retailwh` | | `fact_inventory_snapshot` | one product at one store at one month-end | `retailwh` | > Diagram: Three levels of grain. One journal line holds the most detail. Summing by month gives one row per month and account. Summing by fiscal year gives the grain of an annual budget. The rule is simple. Bring both to the same grain first, then compare them. The next section shows what goes wrong without that step. > **Watch out.** A fact at a coarse grain, such as a yearly budget, cannot be joined to a fine one, such as a journal line, without multiplying the coarse amount. ## Budget against actual: align the grain first The question: **which expense accounts are over budget?** The budget is one row per account per fiscal year. The journal has thousands of lines for the same account in the same year. A direct join makes one budget row match every line, so the budget is counted once per line. The query below is the mistake. It joins the budget straight to the journal lines and sums the budget. Runs on `nonprofit`. ```sql SELECT SUM(b.budget_amount) AS expense_budget_after_join FROM budgets AS b JOIN gl_accounts AS a ON a.gl_account_id = b.gl_account_id JOIN gl_journal_lines AS l ON l.gl_account_id = b.gl_account_id JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id AND e.fiscal_year = b.fiscal_year WHERE b.fiscal_year = 2026 AND a.account_type = 'Expense'; ``` **What it returns.** 3,722,082,400. The budget for all expense accounts in FY2026 is 7,334,200. The join multiplied every budget amount by the number of journal lines behind it. No error appears, and a report built on this number would look large and plausible, which is what makes the mistake dangerous. The fix is to aggregate each side to the common grain, which is fiscal year and account, and then join the two results. Runs on `nonprofit`. ```sql WITH budget AS ( SELECT gl_account_id, SUM(budget_amount) AS budget FROM budgets WHERE fiscal_year = 2026 GROUP BY gl_account_id ), actual AS ( SELECT l.gl_account_id, SUM(l.debit - l.credit) AS actual FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id WHERE e.fiscal_year = 2026 GROUP BY l.gl_account_id ) SELECT a.account_number, a.account_name, b.budget, ISNULL(x.actual, 0) AS actual, ISNULL(x.actual, 0) - b.budget AS over_under FROM budget AS b JOIN gl_accounts AS a ON a.gl_account_id = b.gl_account_id LEFT JOIN actual AS x ON x.gl_account_id = b.gl_account_id WHERE a.account_type = 'Expense' ORDER BY over_under DESC; ``` **What it returns.** Nineteen expense accounts. For an expense, a positive `over_under` means the account spent more than budget. The four over budget in FY2026 are: | Account | Budget | Actual | Over (+) | |---|---|---|---| | 6210 Food Purchases | 468,200.00 | 508,833.91 | +40,633.91 | | 6600 In-Kind Goods Distributed | 383,400.00 | 417,400.00 | +34,000.00 | | 6900 Pledge Write-offs | 28,900.00 | 30,345.00 | +1,445.00 | | 6520 Technology & Software | 179,200.00 | 180,015.14 | +815.14 | Every other expense account came in under budget. Food purchases is the one to ask about first, because it is the largest overrun. Across all expense accounts, actual spending is 7,314,951.09 against a budget of 7,334,200, so the year as a whole came in under plan, and the overruns sit in a few accounts. > Diagram: Top, a wrong join: one budget row is joined to every journal line of its account, so the budget is counted once per line. Bottom, the right way: sum the budget and the actual to account and year, then compare the two single rows. Food Purchases is 40,633.91 over budget. The same discipline applies when the grains are monthly. `retailwh` has a budget by fiscal month, category and region, and actual sales at the order line. Align both to month, category and region, then add up by category. Runs on `retailwh`. ```sql WITH act AS ( SELECT d.fiscal_year, d.fiscal_period, sub.category_key, s.region_key, SUM(f.net_amount_usd) AS actual FROM fact_sales AS f JOIN dim_date AS d ON d.date_key = f.order_date_key JOIN dim_product AS p ON p.product_key = f.product_key JOIN dim_subcategory AS sub ON sub.subcategory_key = p.subcategory_key JOIN dim_store AS s ON s.store_key = f.store_key WHERE d.fiscal_year = 2026 GROUP BY d.fiscal_year, d.fiscal_period, sub.category_key, s.region_key ), bud AS ( SELECT fiscal_year, fiscal_period, category_key, region_key, SUM(budget_usd) AS budget FROM fact_budget WHERE fiscal_year = 2026 GROUP BY fiscal_year, fiscal_period, category_key, region_key ) SELECT c.category_name, SUM(b.budget) AS budget, SUM(a.actual) AS actual, SUM(a.actual) - SUM(b.budget) AS variance FROM bud AS b LEFT JOIN act AS a ON a.fiscal_year = b.fiscal_year AND a.fiscal_period = b.fiscal_period AND a.category_key = b.category_key AND a.region_key = b.region_key JOIN dim_category AS c ON c.category_key = b.category_key GROUP BY c.category_name ORDER BY variance DESC; ``` **What it returns.** Outdoor is 78,477.93 over budget (1,853,524.30 against 1,775,046.37), Apparel is 44,079.26 over, Beauty is 7,802.11 over, Home is 31,279.41 under and Electronics is 211,045.05 under. The total of the actual column, 9,120,656.88, equals the FY2026 total of `fact_sales`. That check matters. A budget cell with no sales stays in the list because of the `LEFT JOIN`. A sale with no budget cell would drop out of the answer, and the total would be short. ## Dates: one calendar, many roles A **date table** has one row for every day, with no gaps, and every fiscal attribute you need. Every fact joins to it. A gap in the calendar breaks time-intelligence measures such as year to date. This check counts the rows in the `cpafirm` fiscal calendar against the number of days in its range. Runs on `cpafirm`. ```sql SELECT COUNT(*) AS calendar_rows, DATEDIFF(day, MIN(calendar_date), MAX(calendar_date)) + 1 AS days_in_range FROM fiscal_calendar; ``` **What it returns.** 2,556 rows and 2,556 days. No day is missing and none is repeated. **Name the fiscal year before you publish.** The sample databases do not all start the year in January. In `cpafirm` and `nonprofit`, fiscal 2026 runs from July 2025 to June 2026, and it is named for the year it ends. In `retailwh` the year starts in April. Check it on one date. Runs on `retailwh`. ```sql SELECT calendar_date, fiscal_year, fiscal_period_name FROM dim_date WHERE calendar_date IN ('2025-03-31', '2025-04-01'); ``` **What it returns.** 31 March 2025 is FY2025-P12, the last period of fiscal 2025. 1 April 2025 is FY2026-P01. The first day of a fiscal year is a common place for a report to slip a month, so confirm it against a known date. **Role-playing dates.** One invoice has three dates: issue, due and payment. Each one can answer a different question, and the model needs a date table for each role, or one date table with several relationships that are switched on as needed. This query puts the issue date and the due date of the same invoice into fiscal periods. Runs on `cpafirm`. ```sql SELECT TOP 3 i.invoice_number, i.issue_date, i.due_date, fi.fiscal_period_name AS issued_in, fd.fiscal_period_name AS due_in FROM invoices AS i JOIN fiscal_calendar AS fi ON fi.calendar_date = i.issue_date JOIN fiscal_calendar AS fd ON fd.calendar_date = i.due_date WHERE fi.fiscal_period <> fd.fiscal_period ORDER BY i.issue_date DESC; ``` **What it returns.** INV-2026-03848, issued 22 October 2026 and due 21 November 2026, is issued in FY2027-P04 and due in FY2027-P05. Billings are counted in the month issued, and receivables are aged from the due date. > **Watch out.** A report that uses the wrong date for each question will disagree with the aging. > Diagram: One invoice with three dates. Its issue date is the active relationship to one shared date table and lands in FY2027 period 4. Its due date lands in period 5 through an inactive relationship. Its payment date is blank because the invoice is not yet paid. Power BI handles this with one active relationship and the other relationships inactive. A measure activates an inactive one with `USERELATIONSHIP`. You can read the payment side of the same idea in the next query, which measures how far after the due date each payment arrived. Runs on `cpafirm`. ```sql SELECT p.payment_method, COUNT(*) AS applications, ROUND(AVG(CAST(DATEDIFF(day, i.due_date, p.payment_date) AS float)), 1) AS avg_days_after_due, SUM(CASE WHEN p.payment_date > i.due_date THEN 1 ELSE 0 END) AS paid_late FROM payment_applications AS pa JOIN payments AS p ON p.payment_id = pa.payment_id JOIN invoices AS i ON i.invoice_id = pa.invoice_id WHERE p.fiscal_year = 2026 GROUP BY p.payment_method ORDER BY applications DESC; ``` **What it returns.** ACH averages 8.1 days after the due date across 2,000 applications, and 1,304 of them were late. Checks average 8.7 days, credit cards 8.3. Trust applications show payment dates about 30 days before the due date. That is a pattern to ask about, not a finding to report. The same role idea works for a process. In `retailwh`, one order has an order date, a ship date and a delivery date, each a separate key into the date table. The next query joins the calendar three times, once per role. Runs on `retailwh`. ```sql SELECT COUNT(*) AS delivered_orders, ROUND(AVG(CAST(DATEDIFF(day, o.calendar_date, s.calendar_date) AS float)), 1) AS days_order_to_ship, ROUND(AVG(CAST(DATEDIFF(day, s.calendar_date, d.calendar_date) AS float)), 1) AS days_ship_to_delivery FROM fact_order_pipeline AS f JOIN dim_date AS o ON o.date_key = f.order_date_key JOIN dim_date AS s ON s.date_key = f.shipped_date_key JOIN dim_date AS d ON d.date_key = f.delivered_date_key WHERE f.status = 'Delivered'; ``` **What it returns.** 50,044 delivered orders, on average 2.0 days from order to ship and 4.3 days from ship to delivery. Each alias is one role of the same date table. > Exercise: For fiscal 2026, show the number of invoices and the billed total (rounded to cents) for each fiscal period. Use the fiscal calendar on the invoice issue date. Period order. ## Chart of accounts: the hierarchy that becomes the statements A chart of accounts is a hierarchy. Accounts roll up to parents, and the parents become the lines of the financial statements. In `cpafirm`, `gl_accounts` has a `parent_account_id`. Revenue account 4100 Tax Preparation Revenue rolls up to 4000 Professional Fee Revenue, and 5010 Salaries & Wages rolls up to 5000 Personnel Costs. > Diagram: Chart of accounts tree. Professional Fee Revenue (4000) is a parent with no postings and seven revenue accounts beneath it that carry postings. Personnel Costs (5000) is a parent with no postings and two expense accounts beneath it. Two rules keep the hierarchy honest: 1. **Post to leaves only.** A parent is a heading. If you total a parent together with its children, the children are counted twice, once in themselves and once in the parent. The next query checks that no line posts to a parent. Runs on `cpafirm`. ```sql SELECT COUNT(*) AS postings_to_parent_accounts FROM gl_journal_lines AS l WHERE l.gl_account_id IN (SELECT parent_account_id FROM gl_accounts WHERE parent_account_id IS NOT NULL); ``` **What it returns.** 0. The parent accounts carry no postings, so a total built from the leaves cannot double count. 2. **Use one sign convention for each statement.** The ledger stores debits and credits. A statement shows revenue as a positive number, and expenses as positive too. Revenue is credit minus debit, because revenue is normally a credit balance. Expense is debit minus credit. Assets are debit minus credit, and liabilities and equity are credit minus debit. This query builds an income statement for FY2026. It maps each leaf to its parent name and applies the sign convention. Runs on `cpafirm`. ```sql SELECT COALESCE(p.account_name, a.account_name) AS statement_line, a.account_type, SUM(CASE WHEN a.account_type = 'Revenue' THEN l.credit - l.debit ELSE l.debit - l.credit END) AS amount_shown_positive FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id LEFT JOIN gl_accounts AS p ON p.gl_account_id = a.parent_account_id WHERE e.fiscal_year = 2026 AND a.account_type IN ('Revenue', 'Expense') GROUP BY COALESCE(p.account_name, a.account_name), a.account_type ORDER BY a.account_type DESC, amount_shown_positive DESC; ``` **What it returns.** Professional Fee Revenue of 3,890,467.73, Personnel Costs of 2,177,596.99 and Operating Expenses of 322,160.68. The revenue line is 29,209.77 lower than the invoice total for the year, 3,919,677.50. That gap is account 4900, Billing Adjustments and Discounts, which carries a debit balance of 29,209.77. Every statement total should be explainable by its accounts, and this one is. > Exercise: How many accounts are in each account type? Show the type and the count, in alphabetical order by type. ## Balances are not additive across time Some numbers are **semi-additive**. You can add them across products, stores or accounts, but never across dates. A cash balance is the clearest example. The balance on 31 March plus the balance on 30 April is not a number anyone needs. The balance at the end of the period is the number. The query below reads the operating cash balance (account 1000) at four month-ends. Runs on `cpafirm`. ```sql WITH month_ends AS ( SELECT CAST(d AS date) AS month_end FROM (VALUES ('2026-03-31'), ('2026-04-30'), ('2026-05-31'), ('2026-06-30')) AS v(d) ) SELECT m.month_end, (SELECT SUM(l.debit - l.credit) FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE a.account_number = 1000 AND e.entry_date <= m.month_end) AS cash_balance FROM month_ends AS m ORDER BY m.month_end; ``` **What it returns.** 3,144,909.35 at 31 March, 3,412,550.19 at 30 April, 3,649,159.30 at 31 May and 3,695,503.29 at 30 June. The change over the quarter is the last balance minus the first, 550,593.94. Adding the four balances gives a figure of about 13.9 million, which is meaningless. The same test on inventory shows the risk more clearly. The next query adds the on-hand units of six month-ends in `retailwh`. Runs on `retailwh`. ```sql SELECT SUM(i.on_hand_units) AS on_hand_added_across_six_month_ends FROM fact_inventory_snapshot AS i JOIN dim_date AS d ON d.date_key = i.month_end_date_key WHERE d.calendar_date BETWEEN '2025-10-01' AND '2026-03-31'; ``` **What it returns.** 156,731 units. Each month-end is a count of much of the same stock, so adding the six counts counts the same units six times. The on-hand count at 31 March 2026 is the balance to report, and it is 23,662 units. In Power BI, the measure for a balance is the last value in the period, such as `LASTNONBLANK` or `CLOSINGBALANCEMONTH`. For an average balance, use an average over the periods. Never use a plain `SUM` over a date column for a balance. ## Many-to-many: the bridge that multiplies totals A **many-to-many** relationship is one where a row on each side can match many rows on the other. A CPA engagement has a preparer, a manager and a partner, and each of them is on many engagements. The database stores this with a **bridge table**, one row per engagement and person. The bridge is necessary, and it is also the most common source of double-counted totals. > Diagram: One sale of product 2 meets the bridge table, which holds two rows for product 2, one for tag 1 and one for tag 3. The join therefore produces two rows for the single sale, and its amount is counted twice. In `cpafirm`, `engagement_staff` is the bridge. Every engagement has a manager, a partner and at least one preparer, so the table holds about 3.25 rows per engagement. The next query sums the fee for FY2026 two ways. Runs on `cpafirm`. ```sql SELECT (SELECT SUM(e.fee_amount) FROM engagements AS e WHERE e.fiscal_year = 2026) AS fees_correct, (SELECT SUM(e.fee_amount) FROM engagements AS e JOIN engagement_staff AS s ON s.engagement_id = e.engagement_id WHERE e.fiscal_year = 2026) AS fees_after_staff_join; ``` **What it returns.** The correct fee total is 1,849,175.00. After the join it is 6,089,615.00, about three times as large, because each fee is counted once for each person on the engagement. The same thing happens in `retailwh`, where `bridge_product_tag` holds 468 rows for 262 products. Each product has one or more tags, so a sale of a tagged product is counted once for each tag. Runs on `retailwh`. ```sql SELECT (SELECT SUM(net_amount_usd) FROM fact_sales) AS sales_correct, (SELECT SUM(f.net_amount_usd) FROM fact_sales AS f JOIN bridge_product_tag AS b ON b.product_key = f.product_key) AS sales_after_tag_join; ``` **What it returns.** 38,834,618.54 for the correct total and 56,569,237.09 after the tag join. The fix is to keep each measure on the fact table and use the bridge only for filtering, or for counts that should count each tag, such as a count of tags per product. In Power BI, a many-to-many relationship through a bridge needs the same care, because a visual that groups by tag will show each sale once per tag, which is correct for that visual and wrong for a total. ## Slowly changing dimensions: what the row said at the time Customers move, change tiers and change segments. A **slowly changing dimension** keeps a row for each version of a customer, with the dates each version was valid. `retailwh` does this in `dim_customer`: 12,000 customers, and 5,314 of them have more than one version. Customer C000011 has three versions. Its loyalty tier changed from Bronze to Silver in September 2026, and its city changed from Birmingham to London in December 2022. The query below lists those versions. Runs on `retailwh`. ```sql SELECT customer_key, loyalty_tier, city, valid_from, valid_to, is_current FROM dim_customer WHERE customer_id = 'C000011' ORDER BY valid_from; ``` **What it returns.** Three rows. The first two are not current, and the third, with `is_current = 1`, is the version in effect today. The sales join depends on what you want to know. Joining the fact to `customer_key` reports each sale against the version that was in effect when it happened. That is the right answer for "what did Bronze customers buy?" The next query uses the fact-to-version join, grouped by the version's attributes. Runs on `retailwh`. ```sql SELECT v.loyalty_tier, v.city, COUNT(*) AS sales FROM fact_sales AS f JOIN dim_customer AS v ON v.customer_key = f.customer_key WHERE v.customer_id = 'C000011' GROUP BY v.loyalty_tier, v.city ORDER BY v.loyalty_tier, v.city; ``` **What it returns.** Bronze in Birmingham: 16 sales. Bronze in London: 46 sales. Silver in London: 4 sales. Those 66 sales add up to the customer's total. To report every sale under the customer's **current** tier and city, join the same history to the current version instead. The next query does that. Runs on `retailwh`. ```sql SELECT cur.loyalty_tier, cur.city, COUNT(*) AS sales FROM fact_sales AS f JOIN dim_customer AS v ON v.customer_key = f.customer_key JOIN dim_customer AS cur ON cur.customer_id = v.customer_id AND cur.is_current = 1 WHERE v.customer_id = 'C000011' GROUP BY cur.loyalty_tier, cur.city; ``` **What it returns.** All 66 sales under Silver in London. Both answers are correct. They answer different questions, so label each report with which one it shows. A department reorganization works the same way: a department that moves to a new division should keep its history under the old division. ## Bidirectional filters and ambiguous paths Each relationship in a model has a direction, the way a slicer filters the table on its far side. A **bidirectional** relationship filters both ways. Used carelessly, it creates loops, and when two paths lead from one table to another, Power BI cannot tell which one a total should follow. The total then depends on which visual asked for it. The practical rule is to keep relationships single-direction from the dimension to the fact, which is the star shape in the first section. Turn on a bidirectional filter only when a specific measure needs it, put that choice in the measure, and test the total against a SQL query on the source. A relationship that exists for convenience is usually the one that gives the wrong number. ## Checks that catch wrong totals Build these into the model before anyone reads a report: 1. **Reconcile the subledger to the general ledger.** Invoices billed in FY2026 should equal the revenue posted from invoice entries. In `cpafirm` both are 3,919,677.50. Runs on `cpafirm`. ```sql SELECT (SELECT SUM(total_amount) FROM invoices WHERE fiscal_year = 2026) AS invoiced, (SELECT SUM(l.credit - l.debit) FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE e.source_type = 'Invoice' AND e.fiscal_year = 2026 AND a.account_type = 'Revenue') AS revenue_from_invoice_entries; ``` 2. **Count the rows.** A table built from a source should have the same number of rows as the source, or a known, explained difference. 3. **Check the grain.** Count the rows, then count the distinct values of the key that defines the grain. The two numbers must match. 4. **Look for orphans.** A child row with no parent is a broken link, and it drops out of any report that joins to the parent. > Exercise: Show the net sales in USD for each fiscal year, rounded to whole dollars, with the number of sale lines. Join the sales to the date table on the order date key. Earliest year first. ## Try it Run these on the databases named in each line. Change one thing at a time and check the result. 1. **Runs on `nonprofit`.** Run the budget against actual query again with `fiscal_year = 2025`. Which expense accounts were over budget last year? Is food purchases over budget again? 2. **Runs on `cpafirm`.** Add `'2026-09-30'` to the list of month-ends in the cash balance query. Does the balance on that date look like a continuation of the trend? 3. **Runs on `cpafirm`.** Count the days in fiscal 2026. The answer should match the number of days from 1 July 2025 to 30 June 2026. ```sql SELECT COUNT(*) AS days_in_fy2026 FROM fiscal_calendar WHERE fiscal_year = 2026; ``` For the model build itself, read [Power BI for finance teams](/analyze/power-bi-for-finance). For the measures that sit on top of a model like this, read [DAX for financial reporting](/analyze/dax-financial-reporting). --- # Relationships and the data model ## Star schema first Model with one fact table per business process in the middle (Sales, Orders, Payments) and dimension tables around it (Calendar, Product, Customer, Store). Relationships are one to many from dimension to fact, single direction. A star schema is smaller, faster and makes DAX simple; snowflaked dimensions (Product to Subcategory to Category) work but flatten them with RELATED columns or Power Query merges when you can. > Diagram: A Product table on the one side filters a Sales table on the many side. Selecting Helmet keeps only the Helmet sale. A filter on Sales does not reach Product. ## Relationship properties - **Cardinality**: one-to-many (the normal case), many-to-one, one-to-one, many-to-many (avoid unless a bridge table is not possible). - **Cross-filter direction**: Single (dimension filters fact) or Both. Both can create ambiguity and slow queries; prefer Single and use `CROSSFILTER` or a bridge table inside a measure when you need the other way. - **Active versus inactive**: only one active path between two tables. Extra paths (Order Date and Ship Date both pointing at Calendar) are inactive and used by `USERELATIONSHIP` in a measure. - **Referential integrity**: if the fact has keys with no dimension row the model adds a (Blank) member. Fix the data rather than relying on it. - Keys: use integer surrogate keys for relationships; hide them from the field list. ## RELATED and RELATEDTABLE - `RELATED(Product[Brand])` in a calculated column (or iterator) on the many side fetches one value from the one side through the active relationship. It needs row context on the many side. - `RELATEDTABLE(Sales)` on the one side returns that row's related fact rows, for use in `COUNTROWS(RELATEDTABLE(Sales))` or `SUMX(RELATEDTABLE(Sales), ...)`. - A measure rarely needs RELATED: filters already flow. RELATED is for calculated columns and iterators that do arithmetic across tables, like `SUMX(Sales, Sales[Quantity] * RELATED(Product[UnitCost]))`. ## LOOKUPVALUE `LOOKUPVALUE(Product[Brand], Product[ProductKey], Sales[ProductKey])` looks a value up without a relationship (and returns blank or an alternate when not found, an error if several rows differ). > **Tip.** Slower and clumsier than a relationship; use for tables you cannot relate. ## Role-playing dimensions One Calendar, several date columns in the fact (Order, Ship, Due). Keep the Order Date relationship active and create measures with `USERELATIONSHIP` for the others: ``` Sales by Ship Date = CALCULATE([Total Sales], USERELATIONSHIP(Sales[ShipDate], 'Calendar'[Date])) ``` Or load the Calendar twice (Ship Calendar), which makes both usable in one visual at the cost of a second table. ## Many-to-many Prefer a bridge (factless) table with one-to-many relationships to both sides (for example Customer to Account to Customer): filters then flow through it. > **Watch out.** Use a many-to-many relationship only when the grain is not recoverable, and understand that the numbers can double count in totals. A measure can use `TREATAS` to apply one side's values onto the other without any relationship. ## Hierarchies and flattening Create a hierarchy (Category > Subcategory > Product) in the model for drill down. For snowflake dimensions bring the parent attribute down: `Product[Category] = RELATED(Category[Category])` (needs the chain of relationships) or merge the tables in Power Query. ## Disconnected tables and slicers that drive measures A table with no relationships (for example a Measure Choice table, a What-if parameter, a Top N number) can feed a measure: `SELECTEDVALUE('Top N'[Top N Value], 10)` or a `SWITCH` that picks a measure. Field parameters (Modeling > New parameter > Fields) are the modern way to let users choose which measure or axis a visual shows. ## Security and relationships Row-level security filters on a dimension flow to the fact through relationships, so keep them single direction. A bidirectional relationship can leak filtering across roles; test with View as. ## Model design checklist Unique keys on the one side; no calculated columns that a measure can replace; hide keys and technical columns; one Calendar marked as the date table; display folders and descriptions for measures; consistent formatting; no unnecessary bidirectional or many-to-many relationships; remove columns you do not use (memory and refresh time). --- # Using the LearnSQL databases for DAX, Power Pivot and Power BI ## Connection details - SQL Server (best for Excel, Power Pivot and Power BI): server `learnsql.cpetoday.com,1433` (a comma before the port), SQL Server authentication (login supplied with the CPE Today course), Import mode. Accept the certificate prompt (public teaching server, self-signed). In Excel: Data > Get Data > From Database > From SQL Server Database. In Power BI Desktop: Get data > SQL Server. In M: `Sql.Database("learnsql.cpetoday.com,1433", "contoso")` then `Source{[Schema="contoso", Item="FactSales"]}[Data]`. - The schema name equals the database name on SQL Server (contoso.FactSales, northwind.Orders, classicmodels.orders), not dbo. (Exceptions: sage100, xtreme and the dbo tables use dbo, qbd uses QuickBooks, qbo uses QuickBooksOnline, and adventureworks puts the original schema in the table name, e.g. adventureworks.Production_Product.) - MariaDB/MySQL: server `learnsql.cpetoday.com`, port 3306, databases acme, northwind, world and the mirrored sample databases. Power BI Desktop uses the MySQL database connector (needs the MySQL driver); Excel uses From MySQL Database (needs the connector installed). SQL Server is the master copy; use it when you have the choice. - Read-only: you can only query. The Web API (`https://learnsql.cpetoday.com/api/mssql//?limit=100`) needs no login and can be loaded with Power Query From Web. - Always choose Import, not DirectQuery, for teaching: it is fast and works offline once loaded. Filter and select columns in Power Query before loading. - Repeated failed logins block an address for a while; copy the password from the course materials. ## Contoso (retail sales; the best DAX playground) Tables in `contoso`: `FactSales`, `DimDate`, `DimProduct`, `DimProductSubcategory`, `DimProductCategory`, `DimStores` (the store table), `DimChannel`, `DimGeography`, `DimPromotion`, `DimEntity`, `DimCurrency` and `DimCustomer`, plus `FactExchangeRate`, `FactBudget` and `FactInventory` (see "Contoso extensions" below). On SQL Server `FactSales` is a 104,940-row sample for speed (every row where `SalesKey % 10 = 0`) and `FactSalesFull` holds all 1,048,575 rows (January 2016 to December 2018); on MariaDB `FactSales` holds all 1,048,575 rows and there is no `FactSalesFull`. Every key is a declared primary key and every relationship a declared, trusted foreign key, so Power BI and the Excel Data Model detect the relationships on their own. All keys are whole numbers (`int`), and `DateKey` is a true `date`. Ready-made views save the joins: `vw_Product` (product with subcategory and category), `vw_Store` (store with geography, coordinates and a quality flag), `vw_Sales` (the fact with date, product, store and channel names), `vw_SalesFull` (the same over the full table, SQL Server only) and `vw_SalesByMonthCategory` (monthly totals by category). Columns that matter: - `FactSales`: SalesKey, DateKey (date), channelKey (lower-case c), StoreKey, ProductKey, PromotionKey, CurrencyKey, UnitCost, UnitPrice, SalesQuantity, ReturnQuantity, ReturnAmount, DiscountQuantity, DiscountAmount, TotalCost, SalesAmount. - `DimDate`: Datekey (date), CalendarYear, CalendarHalfYear, CalendarQuarter, CalendarMonth (a yyyymm number such as 201603, so it sorts correctly), CalendarMonthLabel (the month name as text), CalendarWeek, CalendarDayOfWeek, FiscalYear, FiscalQuarter, FiscalMonth, IsWorkDay, IsHoliday, HolidayName, and label columns. It covers 2014-2020, so filter it to the data's 2016-2018 (or build your own Calendar in M). - `DimStores`: StoreKey and GeographyKey (both int), StoreName, StoreType, Status, OpenDate, CloseDate, EmployeeCount, SellingAreaSize, plus `Latitude`, `Longitude` and `GeoQuality` (the coordinates come from the store's original location data; where they were swapped or wrong they were repaired and flagged in GeoQuality). `DimGeography`: GeographyKey (int), ContinentName, RegionCountryName, StateProvinceName, CityName, Latitude, Longitude. `DimChannel`: ChannelKey, ChannelName (Store, Online, Catalog, Reseller). Model recipe (star schema): - Sales (from FactSales or FactSalesFull): keep SalesKey, Date, channelKey (rename ChannelKey), StoreKey, ProductKey, UnitCost, UnitPrice, SalesQuantity, ReturnQuantity, ReturnAmount, DiscountAmount, TotalCost, SalesAmount. Hide the key columns. - Calendar (from DimDate or built in M): Date, Year, Quarter, Month Number, Month (sorted by Month Number), Year-Month (sorted by a YearMonthKey integer). Mark as date table. - Product -> Subcategory -> Category (a snowflake): either flatten with merges or relate the chain (Sales[ProductKey] to Product, Product[ProductSubcategoryKey] to Subcategory, Subcategory[ProductCategoryKey] to Category; the category table's key is named `ProductCategoryID`, rename it to ProductCategoryKey if you like; the relationship is declared in the database). - Store (DimStores) related to Geography through GeographyKey; Channel (DimChannel). Control totals on the FULL fact table (`FactSalesFull`): Total Sales $3,826,462,615.76, Total Cost $1,629,711,680.67, Gross Margin % 57.4%; by year 2016 $1,440,347,077.25, 2017 $1,210,810,889.77, 2018 $1,175,304,648.74. The 10% sample gives roughly a tenth of each: amounts differ, ratios (margin %, share by channel) stay close. Always check which fact table a project loaded before comparing to answers, and recompute control totals with SQL on the table you loaded. Good first measures: Total Sales, Total Cost, Gross Margin, Gross Margin %, Units Sold, Average Unit Price, Online share of sales, Sales by channel, Returns %, Sales PY, YoY %. ## Contoso extensions: customers, budget, currencies and inventory Four additions make Contoso a fuller model. None of them changes a sales amount, so the control totals above still hold. - **`DimCustomer`** (30,000 customers): CustomerKey, FirstName, LastName, Gender, BirthDate, MaritalStatus, YearlyIncome, Occupation, Education, NumberChildrenAtHome, CustomerSegment (Value, Core, Premium), EmailPromotion, SignupDate, AcquisitionChannelKey (to DimChannel) and GeographyKey (the customer's home city, to DimGeography). `FactSales` (and `FactSalesFull`) now carry `CustomerKey`. Every Online and Catalog sale has a customer, about a quarter of Store sales do (a loyalty card), and Reseller sales have none, so the key is blank on purpose for the rest. No customer buys before their SignupDate. Use it for distinct customer counts, new versus returning customers, cohorts, RFM and age bands (build the age band yourself from BirthDate). - **`FactBudget`**: one row per month (the first of the month), store and product category, with BudgetSalesQuantity, BudgetSalesAmount and BudgetTotalCost. Budgets run from 90% to 110% of what was sold, so some cells beat plan and some miss. It is at a coarser grain than the sales, so relate both to the shared Date, Store and Category dimensions instead of to each other (create a month-start column on your calendar to relate on). On SQL Server compare it with `FactSalesFull`; the 10% `FactSales` sample is about a tenth of the budget. - **Currencies**: `DimCurrency` now lists 18 currencies. Sales amounts stay in US dollars (CurrencyKey 1); each store has a local `CurrencyKey` (euro-zone stores in EUR, and so on; stores in countries without a modelled currency use USD). `FactExchangeRate` holds a daily AverageRate and EndOfDayRate for 2016-2018, expressed as local currency per one US dollar, so local amount = US dollars x rate. Use it to practice converting in DAX, switching the reporting currency and a two-column relationship (date and currency). There are no rows for 29 February, because `DimDate` has none. - **`FactInventory`**: a month-end snapshot of OnHandQuantity, OnOrderQuantity, SafetyStockQuantity and InventoryValue for each physical store and product subcategory. A balance must never be added across months; use `LASTNONBLANK`, `CLOSINGBALANCEMONTH` or an average, and divide by sales for stock turn and days of supply. - **Views**: `vw_Customer`, `vw_CustomerSummary` (first and last purchase, sales per customer), `vw_SalesLocalCurrency`, `vw_BudgetVsActual` (budget next to actual with the variance) and `vw_InventoryMonthly`. `vw_Sales` and `vw_SalesFull` now include CustomerKey. Measures to try: Customers = DISTINCTCOUNT(Sales[CustomerKey]); New Customers (customers whose first purchase is in the period); Sales per Customer; Budget Variance % = DIVIDE([Sales] - [Budget], [Budget]); Sales (local currency) = SUMX(Sales, Sales[SalesAmount] * LOOKUPVALUE(FactExchangeRate[AverageRate], FactExchangeRate[DateKey], Sales[DateKey], FactExchangeRate[CurrencyKey], RELATED(Stores[CurrencyKey]))); Stock on Hand = CALCULATE(SUM(FactInventory[OnHandQuantity]), LASTDATE(Calendar[Date])). ## classicmodels (orders for a model-car wholesaler, 2003-2005) Tables: customers, employees, offices, orderdetails, orders, payments, productlines, products. Model: orders (header) and orderdetails (lines) with a LineTotal = quantityOrdered * priceEach; products -> productlines; customers; payments as a second fact (customerNumber, paymentDate, amount). Measures: Revenue = SUMX(orderdetails, quantityOrdered * priceEach), Orders = DISTINCTCOUNT(orders[orderNumber]), Payments, Credit utilisation. Two facts (orders and payments) both relate to customers and a shared Calendar; payments lag orders, a good time-intelligence and "AR" exercise. A good Excel Power Pivot first project: small and clean. Orders span 2003-2005, so build the Calendar for those dates. ## northwind (small trading company, text keys) Tables: Customers, Employees, Orders, `Order Details`, Products, Shippers, Suppliers, Invoices, Purchase Orders and Details, Inventory Transactions. Relationships are on text (Orders.Customer holds the customer's company name; Order Details.Product holds the product name), and there are only about 48 orders, so it is thin for DAX; good for Power Query, merges and SUMIFS in Excel. Primary and foreign keys have been declared on Orders, Order Details, Customers, Shippers and Products (names must match exactly). ## dvdrental (DVD rentals) Tables: payment, rental, customer, film, category, film_category, inventory, store, staff, address, city, country, actor, film_actor. A large-ish payments fact (about 16,000 rows) with rental dates; build Payments -> Rental -> Inventory -> Film -> Category. Measures: Revenue, Rentals, Average rental duration (`DATEDIFF`), Late returns, Revenue per film, Top categories, Customers with repeat rentals. Dates are 2005-2006: the Calendar must cover them. ## world (countries and cities) Tables: country, city, countrylanguage. No time dimension: use it to learn relationships (country Code to city CountryCode), hierarchies (Continent > Region > Country), Top N, and map visuals. Measures: Population = SUM(country[Population]), GDP per capita proxy = DIVIDE(SUM(country[GNP]) * 1000000, SUM(country[Population])), Life expectancy = AVERAGE. ## adventureworks, xtreme, qbd, qbo, sage100, acme - `adventureworks` (SQL Server) is the OLTP version of AdventureWorks (82 tables named like `Production_Product`, `Person_Person`, `HumanResources_Employee`, `Sales_SalesOrderHeader`, `Sales_SalesOrderDetail`): a normalized transactional model, so build a star schema from SalesOrderHeader/Detail, Product, ProductSubcategory, ProductCategory, Customer and SalesTerritory before using DAX. - `xtreme` (dbo): Customer, Orders, `Orders Detail`, Product, `Product Type`, Supplier, Purchases, Employee, `Employee Addresses`, Credit, Financials, `Xtreme Info`. A compact order-entry model that is good for a first star schema. - `qbd` (schema `QuickBooks`, 84 tables) and `qbo` (schema `QuickBooksOnline`, 47 tables): QuickBooks Desktop-style and QuickBooks Online-style accounting data (Customers, Invoices and InvoiceLineItems, Bills, Payments, Accounts, JournalEntries ...). Excellent for AR/AP aging, P&L by class and customer profitability exercises. Use the schema name in the M navigation: `Source{[Schema="QuickBooks", Item="Invoices"]}[Data]`. - `sage100` (dbo, hundreds of tables with Sage 100 names such as AR_Customer, AP_InvoiceHeader): a real accounting-system shape; use only the handful of tables you need. - `acme` exists on MariaDB only for now; connect with the MySQL connector. Use the describe_database and describe_table tools to see the exact tables and columns before modeling. ## Model checklist for these databases 1. Pick the fact table(s) and the grain (one row per order line, per payment). 2. Choose dimensions. 3. Cast keys to Whole number. 4. Build or reuse a Calendar covering the data's years, sorted month names, marked as date table. 5. Relationships one-to-many, single direction. 6. Base measures, then everything else. 7. Check totals against a SQL query on the same database (SELECT SUM(...)) before trusting the report. --- # Data structures and Power BI limits Almost every database you will meet is one of a handful of shapes. Knowing the shape tells you how to join it, how to model it in Power BI or Excel, and which numbers are safe to add up. This page lists the common relational structures, which sample database teaches each, and what Power BI and Excel cannot do with them. ## The structures, and where to practice each | Structure | What it looks like | Practice with | |---|---|---| | **Normalized (transactional) database** | Many small tables, each fact stored once, joined by keys. Built for entering data, not for reporting. | `northwind`, `classicmodels`, `adventureworks`, `world` | | **Header and line** | An order or invoice row plus one row per line. Totals live on the header; quantities and amounts live on the lines. | `northwind` (Orders and Order Details), `cpafirm` (invoices and invoice lines), `bigbear` (permits and inspections) | | **Star schema** | One fact table in the middle, flat dimensions around it. The best shape for Power BI. | `contoso`, [`retailwh`](/analyze/retail-warehouse) | | **Snowflake schema** | A star whose dimensions are split into sub-tables (product, subcategory, category). | `retailwh` (product and store) | | **Slowly changing dimension (type 2)** | A new dimension row, with valid-from and valid-to dates, each time an attribute changes. | `retailwh` (`dim_customer`, `dim_product_price`) | | **Role-playing dimension** | One date table used for order date, ship date and more. | `retailwh`, `northwind` | | **Degenerate dimension** | An identifier kept on the fact table, such as an order number. | `retailwh` (`order_id`) | | **Bridge table (many-to-many)** | Two tables linked through a third that holds the pairs. | `retailwh` (product tags), `nonprofit` (donor interests) | | **Parent-child hierarchy** | A table that points at itself: employee and manager, account and parent account, wholesaler and retailer. | `cpafirm`, `nonprofit`, `bigbear` (employees), the charts of accounts, `mfgorders` (`dim_customer`) | | **Bill of materials** | A recursive parts-in-parts structure several levels deep. | [`mfgorders`](/analyze/big-orders) (`bom`) | | **Periodic snapshot** | A balance recorded at regular intervals, such as month-end inventory. Semi-additive. | `retailwh` (`fact_inventory_snapshot`) | | **Accumulating snapshot** | One row per process instance with a date for each stage; dates fill in over time. | `retailwh` (`fact_order_pipeline`) | | **Factless fact** | Keys only, recording that something happened or was possible. | `retailwh` (`fact_promo_coverage`) | | **Facts at different grain** | Budgets by month and category, actuals by transaction. | `retailwh` (`fact_budget`), `bigbear` (`budgets`) | | **Fund accounting and the general ledger** | Balanced journal entries by fund, department and account. | `nonprofit`, `bigbear`, `cpafirm`, `qbo`, `sage100` | | **Multi-currency** | A local amount, a daily exchange rate and the converted amount. | `retailwh` (`fx_rates`) | | **Time series and event logs** | Very many timestamped rows, often with quality flags. | `smarthome`, `bigbear` (weather, dispatch calls) | | **Messy and semi-structured** | Wide tables, key-value pairs, JSON, text logs and inconsistent codes. | `pqlab` | | **Geography** | Addresses with latitude and longitude, and boundaries. | Every lab database; polygons in `retailwh` (`dim_region.boundary_geojson`) | | **A very large fact table** | Tens of millions of rows or more, kept small with columnstore compression. | `mfgorders` | Not yet covered: a graph of relationships between people or companies, and a stock-movement ledger where the balance is a running total. If you need either, say so. ## How to model each in Power BI and Excel - **Star schema:** load each table as is, relate each fact key to its dimension (one-to-many, single direction), hide the keys, and write measures on the fact table. - **Snowflake:** either keep the chain of relationships (it works) or merge the sub-tables into one flat dimension in Power Query. Flattening is faster and simpler for report authors. - **Slowly changing dimension:** relate the fact to the **surrogate key** (the version). To report on today's attributes instead of the attributes at the time, add a column with the current value to each version, or relate through the business key to the current row. - **Role-playing dates:** one active relationship and others inactive. Switch with `USERELATIONSHIP` in a measure, or load the date table once per role. - **Bridge table:** a many-to-many relationship, or a relationship from each side to the bridge with filter direction set towards the fact. Expect a product to be counted once per tag. - **Parent-child hierarchy:** Power BI does not follow a self-join. Add `PATH([id], [parent_id])` in a calculated column and flatten it with `PATHITEM` into level columns. - **Snapshots:** never add a balance across time. Use the last value in the period (`LASTNONBLANK`, or `CLOSINGBALANCEMONTH`) or an average. - **Facts at a different grain:** relate both facts to shared dimensions (category, month) rather than to each other, and compare in measures. - **Multi-currency:** store the local amount and the rate on the day, convert in the fact table or with a rate table related on date and currency. ## What Power BI and Excel cannot do **Relationships** - A relationship uses **one column** on each side. For a composite key, build a combined column (or an index) first. - Only **one active relationship** can exist between two tables; others are inactive until a measure activates them. - No **circular** relationship paths and no ambiguous paths. Bidirectional filters make ambiguity easy to create. - A relationship does not follow a **self-reference** by itself (see `PATH` above). - The Excel **Data Model** has only the one-to-many relationship type, so a many-to-many needs a bridge table. Power BI supports many-to-many relationships, with caveats. **Data types that do not come through cleanly** - **Spatial** columns (`geography`, `geometry`), `hierarchyid` and `sql_variant` are not loaded. Convert them in the query: `STAsText()` for spatial, `ToString()` for hierarchyid, and a `CAST` for variants. - **Decimal numbers** in Power BI are stored as 64-bit floating point (about 15 significant digits) or as fixed decimal (four decimal places). A SQL `decimal(38,10)` loses precision, and money with more than four decimal places is rounded in fixed decimal. - **Dates** before about 1900 are unreliable in DAX, and a date table must have no gaps for time intelligence to work. - **JSON, XML and delimited lists** arrive as text; parse them in Power Query. - **Images, documents and free text** are stored as binary or text but cannot be analyzed as such. **Size and speed** - An Excel **worksheet** holds 1,048,576 rows, but the Data Model can hold more, limited by memory (32-bit Excel is limited to 2 GB for the whole process). - A Power BI **Import** model in the Pro service is limited to 1 GB compressed; Premium and Fabric allow far larger models. Check your license for the current limits. - **DirectQuery** returns at most one million rows from any single query, is slower, and restricts some DAX and Power Query steps. Excel does not offer it. - **High-cardinality columns** (order numbers, timestamps with seconds, long text) compress poorly and dominate model size. Keep integer keys, split a timestamp into a date and a time, and drop columns no report uses. - **Pre-aggregate** a very large fact table, or use incremental refresh and Power BI aggregations, so that most visuals read thousands of rows instead of millions. See [Big data: manufacturer orders](/analyze/big-orders). **Features that differ** - Row-level security exists in Power BI, not in a shared Excel workbook. - Calculation groups are authored in Power BI Desktop (or external tools), not in the Excel Power Pivot window. - Excel PivotTables can query a Power BI semantic model through "Analyze in Excel", and then use its measures. ## Where to go next - A walk through every warehouse pattern, with queries: [Retail data warehouse (retailwh)](/analyze/retail-warehouse). - Very large tables and the bill of materials: [Big data: manufacturer orders (mfgorders)](/analyze/big-orders). - Model design: [Fundamentals](/analyze/fundamentals) and [Time intelligence](/analyze/time-intelligence). --- # Power BI Desktop how-to ## Get the LearnSQL data in Home > Get data > SQL Server. Server `learnsql.cpetoday.com,1433` (comma), Database `contoso` (or another), Data Connectivity mode Import. Sign in with the Database tab (SQL Server login that comes with the course), accept the certificate prompt. Select the tables in the Navigator and click Transform Data to filter and shape before loading. In the contoso database the fact table `contoso.FactSales` is a ~100,000-row sample; the whole 1,048,575 rows are in `contoso.FactSalesFull`. Keys in some dimensions are float: change them to Whole number in Power Query so relationships work. Open an existing generated project: extract the zip completely, open the `.pbip` file in Power BI Desktop, click Refresh, sign in when prompted. > Diagram: Import copies data into the model at refresh, so visuals are fast. DirectQuery sends a query to the database each time a visual changes, so the data is live but slower. ## Model view Open Model view (left rail) to see tables and relationships. Drag a key column onto another to create a relationship; double click a line to set cardinality and direction, or Manage relationships. Select a column and set Data category, Sort by column, Format, Summarization, Hidden. Group measures with Display folder (Properties pane). Create a hierarchy: right click a column > Create hierarchy, then drag more columns onto it. Mark the calendar: select the Calendar table > Table tools > Mark as date table. ## Report view basics Pick a visual in the Visualizations pane, drag fields to wells (Category/X-axis, Values/Y-axis, Legend, Small multiples, Tooltips). Sort with the visual's ... menu. Format on the paint-roller tab. Slicers: Dropdown or List or Between styles under Visual > Slicer settings. Sync slicers across pages: View > Sync slicers. Bookmarks, buttons and selection pane give interactivity. Edit interactions (Format > Edit interactions) controls which visuals a visual filters. Drillthrough: a page with fields in Drill through filters; right click a data point > Drill through. Report page tooltips: create a page, Page information > Allow use as tooltip, set the page size to Tooltip; a visual uses it via Format > General > Tooltips > Type: Report page. Page navigation buttons: Insert > Buttons > Navigator, or a button with the Action type Page navigation. ## Writing DAX in Desktop - **New measure**: Home or Table tools > New measure, type the formula in the bar. Where a measure lives (its home table) does not affect results. Format it from the Measure tools tab (currency, percent, decimals). - **New column** (calculated column) and **New table** (Table tools > New table) for DAX tables. - **Quick measure**: Home > Quick measure for templates such as year-over-year change; read the DAX it writes, since it is a good learning aid. - **DAX query view**: left rail icon. Write `EVALUATE` queries, run them against the open model, use `DEFINE MEASURE Sales[Test] = ...` to try measures before saving, then "Update model with changes". The Quick queries on a table or column write starter queries. Copilot can help draft DAX in this view. - **TMDL view**: shows the model as TMDL text, editable, with Apply. Use it for bulk changes and for scripts such as user-defined functions or object-level security. - **Visual calculations** (preview in some builds): calculations defined inside a visual using the visual's own rows, such as RUNNINGSUM, MOVINGAVERAGE, PREVIOUS, NEXT, FIRST, LAST. Right click a visual > New visual calculation. They are not in the semantic model. - **Performance analyzer**: View > Performance analyzer > Start recording > Refresh visuals; copy a query to DAX query view to tune it. ## Parameters and dynamic interactivity - **Numeric range parameter**: Modeling > New parameter > Numeric range: creates a table `Parameter` using GENERATESERIES and a measure `Parameter Value = SELECTEDVALUE(...)`; add the slicer. - **Field parameter**: Modeling > New parameter > Fields: choose measures or columns, it creates a calculated table with `NAMEOF` entries; use a slicer on it, and put the parameter field in a visual's well to switch measures/axes. - **Calculation group**: Model view > Calculation group (Home > Calculation group), name the group and the column, add calculation items with `SELECTEDMEASURE()` formulas (YTD, PY, YoY %). Calculation groups require "discourage implicit measures" to be turned on, which Desktop does when you create the first group. Format a percent item using the item's Format string expression: `"0.0%"`. - **Bookmarks**: View > Bookmarks, to capture a visual state; with the selection pane, toggle visuals for layout switching. - **Conditional formatting**: visual Format > select the element (cell elements for tables and matrices, columns for bars, data labels) > fx > Format style: Gradient, Rules, or Field value (a color measure). Icons for KPIs and arrows. - **Dynamic title/subtitle**: fx next to the Title text, choose a measure that returns text. ## Row-level security Modeling > Manage roles > Create, pick the table, enter a DAX filter, for example static `[Region] = "West"` or dynamic `[Email] = USERPRINCIPALNAME()` on a security table related to the model. Test with Modeling > View as. After publishing, assign members to the role in the service (Dataset > Security). Roles filter the table and everything related through single-direction relationships. Object-level security (hide columns or tables) needs TMDL or Tabular Editor: `columnPermission Employees = none`. ## Power Query tips for modeling - Remove columns you do not need, set types, and rename columns to business names before load. - Merge Queries to flatten snowflake dimensions; Append Queries to stack similar tables. - Query folding: right click a step > View native query; if greyed out the step is not being pushed to SQL Server. Keep filtering and column selection early. - Create a Calendar with List.Dates (see patterns topic) or reference an existing DimDate and filter its range. - Do not use a native SQL statement for the learning server: use the table navigation and Power Query steps so students learn M and folding. ## Publish and refresh Home > Publish to a Power BI workspace (needs a Power BI account). Schedule refresh in the service; import models on a cloud workspace refreshing from a server on the internet usually need credentials set in Dataset settings; an on-premises data gateway is for sources inside a private network. The LearnSQL server is reachable from the internet, so a gateway is not needed. Share with a workspace app or a link; set permissions on the workspace and dataset; for Premium/PPU features (incremental refresh, paginated reports) check licensing. ## PBIP projects and source control Save as Power BI project (.pbip) (File > Options > Preview features or the Save as > Power BI Project): the model becomes TMDL text and the report becomes PBIR JSON in folders, friendly to Git. Files named `definition.pbir`, `definition.pbism`, `*.SemanticModel`, `*.Report`. The generator in this MCP writes those folders (create_power_bi_project). ## Typical report-building order 1. Connect and shape (Power Query). 2. Model view: relationships, keys hidden, Calendar marked. 3. Base measures (Total Sales, Cost, Margin) in a folder. 4. Time intelligence and percent measures. 5. Visuals with a consistent grid (pos on a 1920 x 1080 canvas), cards on top, trends and breakdowns below, slicers on the left. 6. Tooltips, drillthrough, and bookmarks. 7. Performance check. 8. Security and publish. > Diagram: Six stages of a Power BI report, in order: get data, transform, model, measures, visuals and publish. --- # Power Pivot in Excel (the Excel Data Model) Power Pivot is the DAX engine inside Excel for Windows: it holds many tables with relationships in the Data Model, and PivotTables, PivotCharts and cube formulas read from it. It uses the same DAX as Power BI, with a smaller feature set. Availability: Microsoft 365 Apps for Windows and Excel 2016/2019/2021 Professional Plus editions; not in Excel for Mac or the web (Mac can use a model built on Windows but cannot create one). If the Power Pivot tab is missing: File > Options > Add-ins > Manage COM Add-ins > Go > tick Microsoft Power Pivot for Excel. ## Getting data into the Data Model - **Power Query (recommended)**: Data > Get Data > From Database > From SQL Server Database, server `learnsql.cpetoday.com,1433`, database `contoso`, Import mode, SQL Server login. In the Navigator tick several tables and choose Transform Data or Load To... > Only Create Connection and tick Add this data to the Data Model. Filter rows and remove columns in Power Query before loading: smaller models are faster. - **From an Excel table**: select the table, Insert > PivotTable, tick Add this data to the Data Model. Or Power Pivot > Add to Data Model. A table loaded this way is a Linked Table that refreshes when the sheet changes. - **From the Power Pivot window**: Power Pivot > Manage opens the window; Home > Get External Data > From Database > From SQL Server. - Prefer loading to the Data Model only (no worksheet copy) for large tables, and load dimension and fact tables separately rather than one wide joined table. - Excel for Windows 64-bit is strongly advised for big models; a workbook with a Data Model is limited by memory and file size (the file is limited to about 2 GB, and 32-bit Excel runs out of memory far sooner). > Diagram: Several tables load into the Excel Data Model, where relationships link them. A PivotTable and slicers read the model, and measures live in the model. The numbers are made up. ## Relationships and the model diagram Power Pivot > Manage > Home > Diagram View: drag the key column of the fact table onto the key column of the dimension. Or Data > Relationships (Data > Relationships > New). Excel's relationships are one-to-many with a single filter direction (many-to-one fact to dimension); bidirectional and many-to-many cardinalities are limited compared with Power BI. Create relationships on integer keys, one active relationship between two tables. ## Date table Power Pivot > Design > Date Table > Mark as Date Table, choose the date column, which must contain unique, contiguous dates. Build the calendar in Power Query (see the patterns topic) or use Date Table > New to have Excel create one. The DAX time intelligence functions then work. ## Writing measures (the right way in Excel) - Power Pivot > Measures > New Measure. Fill in Table name (where the measure lives), Measure name, Formula (you can type DAX with IntelliSense), Category and Format (Currency, Percentage, Number ...). Click Check formula, OK. Edit later through Measures > Manage Measures. - In the Power Pivot window you can also type a measure in the calculation area below a table: `Total Sales := SUM(Sales[SalesAmount])` (the `:=` operator, without a table name qualifier, only here). - Measures appear in the PivotTable Fields list with an fx icon. Always use explicit measures; dragging a numeric column into Values creates an implicit measure ("Sum of SalesAmount") that you cannot reuse in other DAX. - Calculated columns: in the Power Pivot window, click Add Column, type `=` and the formula (`=RELATED(Product[Brand])`). Rename the column. Use sparingly: they add to workbook size. - KPIs: right click a measure in the PivotTable field list or Power Pivot > KPIs > New KPI: base measure, target (a measure or absolute value), and status thresholds. - Hierarchies: in the Power Pivot diagram view, create a hierarchy (Category > Subcategory > Product) for drilling in PivotTables. ## Features Power BI has that Power Pivot does not (or hides) Verify against your Excel build, because Microsoft adds to Excel's Data Model over time; as a guide: - **No New Table (calculated tables)** command in Power Pivot. Workarounds: create the table in Power Query (the Calendar, Top N list), load a linked Excel table, or define a DAX query table (below). - **No calculation groups, field parameters, or user-defined functions** created in the Excel UI. Time intelligence variants are separate measures (YTD, PY, YoY %). - **No row-level security** in Excel's Data Model (security belongs to Power BI, Analysis Services or the file's sharing). - **No composite models, DirectQuery or incremental refresh**: it is import only. - **Visuals are PivotTables and PivotCharts** (plus slicers, timelines, conditional formatting, sparklines, and 3D Maps), not the Power BI visual library. - **Many-to-many and bidirectional relationships** are limited, so model as a star schema and use TREATAS or bridge tables in DAX. - Tools like **DAX Studio** connect to Excel's Data Model (open the workbook, then connect) to run DAX queries and measure performance, and Tabular Editor can edit the embedded model through its Excel connection with caution (back up first). ## PivotTables on the Data Model Insert > PivotTable > Use this workbook's Data Model. In the field list choose All, then drag dimension columns to Rows/Columns/Filters and measures to Values. Slicers (PivotTable Analyze > Insert Slicer) connect to every PivotTable that shares the Data Model (Report Connections). A timeline slicer needs a date column. Show Values As (% of Grand Total, Running Total, Difference From) is a PivotTable feature, not DAX; use DAX when you need the same result in any context. Format a measure once in the measure definition (Format), not in the PivotTable's Value Field Settings, so every PivotTable shows it identically. ## Cube formulas (CUBEVALUE, CUBEMEMBER) Convert a PivotTable to formulas (PivotTable Analyze > OLAP Tools > Convert to Formulas) to lay out a pixel-perfect report. Examples against the Data Model connection `ThisWorkbookDataModel`: ``` =CUBEVALUE("ThisWorkbookDataModel", "[Measures].[Total Sales]", "[Calendar].[Year].&[2017]", "[Channel].[Channel].&[Online]") =CUBEMEMBER("ThisWorkbookDataModel", "[Channel].[Channel].&[Online]") =CUBERANKEDMEMBER("ThisWorkbookDataModel", "[Product].[Product].[Product].Members", 1) ``` Member keys use the ampersand form `[Table].[Column].&[Value]`, measures are `[Measures].[Name]`. Referencing cells (slicer selections via CUBESET) makes the report interactive. ## Query the Data Model with DAX (a DAX query table) Data > Existing Connections > Tables tab > pick any table in the Data Model > Open > choose Table. Then Table Tools > Design > Properties (or Data > Queries & Connections > right-click the connection > Properties > Definition): set Command Type to DAX and Command Text to a query such as: ``` EVALUATE SUMMARIZECOLUMNS('Calendar'[Year], Product[Brand], "Sales", [Total Sales], "Margin %", [Gross Margin %]) ORDER BY 'Calendar'[Year], [Sales] DESC ``` The result lands in a worksheet table that refreshes with the model, a way to get a DAX-calculated table in Excel without New Table. ## Refresh and sharing Data > Refresh All refreshes Power Query queries and the Data Model. Save the connection credentials (Data > Queries & Connections > Properties > Definition). Sharing: send the workbook (the Data Model rides inside it) or publish to OneDrive/SharePoint; Excel on the web can open and refresh-view PivotTables built on a Data Model but cannot edit the model (check your tenant for current behavior). Large models: reduce columns and rows, avoid wide text columns, load only needed tables. ## Typical first lesson in Excel (Contoso) 1. Data > Get Data > From SQL Server, server `learnsql.cpetoday.com,1433`, database `contoso`, tables FactSales, DimDate, DimProduct, DimProductSubcategory, DimProductCategory, DimStores, DimChannel; Transform Data, keep the columns you need, Close & Load To > connection only, Add to Data Model. 2. Power Pivot > Manage > Diagram View: relate FactSales to the dimensions. 3. Mark the calendar table as date table. 4. Add measures `Total Sales := SUM(FactSales[SalesAmount])`, `Total Cost`, `Gross Margin`, `Gross Margin %`. 5. Insert > PivotTable from the Data Model: Category on rows, Year on columns, Total Sales in values. 6. Add `Sales YTD = TOTALYTD([Total Sales], DimDate[Datekey])` (use your date table's column) and `Sales PY`. Name the Power Query steps (rename Source, Navigation) for clarity and keep keys as whole numbers. --- # DAX fundamentals Data Analysis Expressions (DAX) is the formula language of Power BI, Power Pivot in Excel and Analysis Services tabular models. It is a functional language: every formula is an expression that returns a value or a table. Most DAX confusion comes down to one idea, evaluation context, so learn that first. ## What DAX is for: measures, calculated columns, calculated tables - **Measure**: a formula evaluated at query time, in the filter context of wherever it is placed (a card, a matrix cell, a chart bar). It stores no data, so it is the right default for anything you aggregate: `Total Sales = SUM(Sales[SalesAmount])`. - **Calculated column**: evaluated row by row when the model refreshes and stored in the table, using row context. Use it when the result is a property of the row you will filter, group or slice by (for example `Sales[Margin] = Sales[SalesAmount] - Sales[TotalCost]` or a Customer age band). It costs memory, so do not use a calculated column for something a measure can do. - **Calculated table**: a table produced by a DAX table expression, computed at refresh (`Calendar = CALENDAR(DATE(2016,1,1), DATE(2018,12,31))`). Available in Power BI Desktop and Analysis Services; Power Pivot in Excel has no New Table command (see the Power Pivot topic for workarounds). - Rule of thumb: aggregate with a measure; classify or filter by with a column; build lookup/helper tables as tables. ## Row context and filter context - **Filter context** is the set of filters currently applied to the model: slicers, rows and columns of a matrix, visual and page filters, and filters added by CALCULATE. Aggregators such as SUM, COUNTROWS and DISTINCTCOUNT only see the rows that survive the filter context. This is why one measure shows a different number in every cell. - **Row context** means "the current row of one table". Calculated columns have it automatically; iterator functions (SUMX, AVERAGEX, FILTER, ADDCOLUMNS ...) create it while they loop. Inside row context you can refer to a column of the current row, as in `Sales[Quantity] * Sales[UnitPrice]`. - Row context does NOT filter other tables, and by itself it does not filter the same table's aggregations. To turn a row context into a filter, wrap the expression in CALCULATE (this conversion is called context transition). That is why a measure inside an iterator, such as `SUMX(Customer, [Total Sales])`, returns each customer's sales. - Filters flow along relationships from the one side to the many side (dimension to fact). A slicer on Product therefore filters Sales, but a filter on Sales does not filter Product unless the relationship is bidirectional. > Diagram: Row context: a calculated column reads one row at a time. Filter context: a measure sums only the rows the filters allow, here the rows where Channel equals Online. ## Implicit versus explicit measures Dragging a numeric column into a visual makes an implicit measure (a Sum of that column). Prefer explicit measures: they are reusable in other measures, can be formatted once, and show up in the DAX you can read. Hide the raw numeric columns and set `discourageImplicitMeasures` on the model if you want to force it. ## Syntax essentials - Column reference: `Table[Column]`. Quote table names that contain spaces or special characters: `'Order Details'[Quantity]`. Measure reference: `[Measure Name]` with no table. Always qualify columns with their table and never qualify measures, so you can tell them apart at a glance. - Operators: `+ - * /`, `^`, comparison `= <> < > <= >=`, text concatenation `&`, logical `&&` (and) `||` (or) `NOT()`, `IN {"A","B"}` and `IN VALUES(Table[Col])`. - Text is in double quotes, a double quote inside text is doubled `""`. Dates are `DATE(2018,12,31)` or the typed literal `DATE`; do not compare dates with text. Boolean values are TRUE() and FALSE(). - Comments: `// line` , `-- line`, `/* block */`. Indentation and line breaks are free; the DAX formatter (daxformatter.com) or Format Query in DAX query view makes them readable. - `BLANK()` is not zero and not an empty string. A BLANK in a measure makes that row disappear from many visuals. `COALESCE(x, 0)` turns blank into 0 when you want a number to show. ## Variables: VAR and RETURN ``` Margin % = VAR Sales = [Total Sales] VAR Cost = [Total Cost] RETURN DIVIDE(Sales - Cost, Sales) ``` Variables are evaluated once, in the context where they are defined (not where they are used), can hold a scalar or a table, and make formulas faster and easier to read. A variable cannot be changed after it is set, and it is evaluated before CALCULATE changes the context, a frequent source of surprise (and the reason the prior-year pattern below works). ## DIVIDE, BLANK and error handling Use `DIVIDE(numerator, denominator, [alternate])` instead of `/`: it returns BLANK (or the alternate) when the denominator is zero or blank. > **Tip.** Use `IFERROR` sparingly because it hides real mistakes and is slower; prefer testing the condition (`IF(ISBLANK(x), ...)`, `DIVIDE`). `ISBLANK`, `ISEMPTY`, `HASONEVALUE`, `SELECTEDVALUE(Column, [alternate])` are the main tests for what the context holds. ## Data types and formatting Whole number, Decimal (floating point), Fixed decimal (currency, 4 decimals), Text, Date/time, True/False. Floating-point math can leave tiny remainders; wrap comparisons in ROUND when equality matters. Format with the Format string of the measure (Measure tools or the model's formatString), not with the FORMAT function, because FORMAT returns text and breaks sorting and charts. Use FORMAT only for labels such as dynamic titles. ## A first set of measures for a sales model ``` Total Sales = SUM(Sales[SalesAmount]) Total Cost = SUM(Sales[TotalCost]) Gross Margin = [Total Sales] - [Total Cost] Gross Margin % = DIVIDE([Gross Margin], [Total Sales]) Units Sold = SUM(Sales[SalesQuantity]) Order Lines = COUNTROWS(Sales) Customers = DISTINCTCOUNT(Sales[CustomerKey]) Average Price = DIVIDE([Total Sales], [Units Sold]) ``` Build every later measure on top of these so a business rule lives in one place. --- # CALCULATE and filter context CALCULATE is the most important function in DAX: `CALCULATE(, , , ...)` evaluates the expression in a filter context that you modify. Almost every non-trivial measure is CALCULATE in some form. ## How CALCULATE works 1. The incoming filter context (slicers, rows, columns) is copied. 2. Any row context is converted to filters (context transition). 3. Each filter argument is applied: it either replaces, removes or adds to the filters on the columns it names. 4. The expression is evaluated in the new context. Filter arguments are evaluated in the ORIGINAL context and then applied together as an AND, never one after another. > Diagram: CALCULATE takes the slicer's Store filter, replaces it with a Channel equal to Online filter, and evaluates Total Sales for Online only. ## Boolean filter arguments `CALCULATE([Total Sales], Channel[Channel] = "Online")` is shorthand for filtering ALL values of the column that meet the condition. A Boolean filter REPLACES any existing filter on that same column, so on a table of channels every row shows the Online figure. Boolean filters can name only one column and cannot call a measure. Use IN for a list: `Channel[Channel] IN {"Store", "Online"}`. Combine columns by listing several arguments (they are ANDed) or by using a table filter. ## Table filter arguments and FILTER When a condition needs a measure or two columns together, pass a table: `CALCULATE([Total Sales], FILTER(ALL(Product[Brand]), [Units Sold] > 1000))`. FILTER iterates a table row by row, so give it the smallest table that works (a single column through ALL or VALUES, not the fact table). `CALCULATE(COUNTROWS(Sales), Sales[SalesAmount] > 10000)` is better than `COUNTROWS(FILTER(Sales, Sales[SalesAmount] > 10000))` because the column filter is compiled efficiently. ## Removing filters: ALL, REMOVEFILTERS, ALLEXCEPT, ALLSELECTED - `REMOVEFILTERS(Table | Column ...)`: only valid as a CALCULATE filter modifier; clears the filters. Preferred for readability. - `ALL(Table | Column ...)`: same effect inside CALCULATE, and also works as a table function that returns every row or every distinct value, ignoring filters. - `ALLEXCEPT(Table, Col1, Col2)`: clears every filter on the table except the columns listed (for "percent of the group" measures). - `ALLSELECTED(Table | Column)`: restores the filters that came from OUTSIDE the current visual (slicers, page and report filters) and ignores the visual's own rows and columns. This is the "percent of the selection" denominator. - `ALLNOBLANKROW` leaves out the blank row that a broken relationship can add. ``` % of Total = DIVIDE([Total Sales], CALCULATE([Total Sales], REMOVEFILTERS())) -- ignores everything % of Selected = DIVIDE([Total Sales], CALCULATE([Total Sales], ALLSELECTED(Category))) % of Category = DIVIDE([Total Sales], CALCULATE([Total Sales], REMOVEFILTERS(Subcategory))) -- keeps Category filter ``` `REMOVEFILTERS()` with no arguments removes every filter in the model, but keep in mind that it also removes the filters a user chose in slicers. ## KEEPFILTERS `CALCULATE([Total Sales], KEEPFILTERS(Category[Category] = "Computers"))` intersects the new filter with the existing one instead of replacing it, so categories other than Computers show blank. Use it when the incoming selection must still count, and inside SUMMARIZECOLUMNS and iterator tables so a filter does not widen the rows. ## VALUES, SELECTEDVALUE, HASONEVALUE, ISFILTERED - `VALUES(Column)` returns the distinct values currently visible in the column (a one-column table, plus a blank row if a relationship has unmatched rows). `DISTINCT` is the same without that blank row. - `SELECTEDVALUE(Column, [alternate])`: the single visible value, otherwise the alternate (blank by default). The usual tool for slicer-driven logic. - `HASONEVALUE(Column)`, `ISFILTERED(Column)`, `ISCROSSFILTERED`, `ISINSCOPE(Column)`: test what the context holds. ISINSCOPE is the right way to detect which level of a matrix hierarchy a cell is on, to fix subtotals. ## USERELATIONSHIP, CROSSFILTER, TREATAS - `CALCULATE([Total Sales], USERELATIONSHIP(Sales[ShipDate], Calendar[Date]))` makes an inactive relationship active for that calculation only (role-playing dates: order date versus ship date). - `CROSSFILTER(Sales[ProductKey], Product[ProductKey], BOTH)` changes the filter direction for that calculation without making the model bidirectional. - `TREATAS(VALUES(Budget[Month]), Calendar[Date])` applies the values of one column as filters on another with no relationship; it is the way to use a disconnected table or to join on a different grain. ## Evaluation order and common mistakes - A measure reference like `[Total Sales]` is secretly `CALCULATE(SUM(...))`, so it always converts row context to filters, which is why measures work inside iterators and calculated columns behave differently. - Replacing versus keeping: a Boolean filter replaces, KEEPFILTERS intersects, ALL removes. When a result "ignores my slicer", you removed it; when it ignores the row, you replaced the row's filter. - Do not put the SAME column in two Boolean filters expecting OR; two arguments are ANDed (no rows). Use IN or a single `||` expression on one column: `Product[Color] = "Red" || Product[Color] = "Blue"` is allowed when it names one column only. - A filter on a dimension column does not reach other dimensions unless the fact table connects them. To count products that were ever sold, filter through the fact table. - Calculation groups and CALCULATE interact through SELECTEDMEASURE; if a measure ignores a calculation item, check precedence. > **Watch out.** Context transition on a table with duplicate rows can double count, because each duplicate row's filter matches both. Make a unique key or avoid CALCULATE in the iterator. --- # Time intelligence in DAX Time intelligence compares and accumulates over dates: year to date, prior year, year over year, rolling 12 months, fiscal periods. It needs a proper date table. ## The date table requirements - One row for every day, with no gaps, covering every date in your fact tables (and ideally whole years). The Date column must be Date or DateTime type with no time portion and no duplicates. - Relate it (one to many) to the date column of each fact table. In Power BI, Mark as date table (Table tools) with that column; in Power Pivot use Design > Mark as Date Table. If you mark it, the built-in time intelligence functions get a guarantee of a contiguous date column. - Never use the auto date/time hidden tables for serious work: turn the option off (Options > Data load) and build one shared Calendar. - Month names need a sort column (Month Number) so January sorts before April: set Sort by column on Month. Do the same for Year-Month text (sort by a YearMonthKey integer) and weekday names. - Build it in Power Query (M) or in DAX. DAX version: ``` Calendar = VAR MinDate = DATE(YEAR(MIN(Sales[OrderDate])), 1, 1) VAR MaxDate = DATE(YEAR(MAX(Sales[OrderDate])), 12, 31) RETURN ADDCOLUMNS( CALENDAR(MinDate, MaxDate), "Year", YEAR([Date]), "Quarter", "Q" & QUARTER([Date]), "Month Number", MONTH([Date]), "Month", FORMAT([Date], "MMMM"), "Year-Month", FORMAT([Date], "yyyy-MM"), "Weekday Number", WEEKDAY([Date], 2), "Weekday", FORMAT([Date], "dddd"), "Fiscal Year", "FY" & (YEAR([Date]) + IF(MONTH([Date]) >= 7, 1, 0)) ) ``` `CALENDARAUTO()` builds a calendar from all date columns in the model; prefer an explicit range so stray dates do not widen it. > Diagram: A month strip for 2024 and 2025 with June 2025 selected. Year to date covers January to June 2025, prior year to date covers January to June 2024, and rolling 12 months covers July 2024 to June 2025. ## Year, quarter and month to date ``` Sales YTD = TOTALYTD([Total Sales], 'Calendar'[Date]) Sales YTD (fiscal June) = TOTALYTD([Total Sales], 'Calendar'[Date], "6/30") Sales QTD = TOTALQTD([Total Sales], 'Calendar'[Date]) Sales MTD = TOTALMTD([Total Sales], 'Calendar'[Date]) -- Same thing written with CALCULATE, which also works for anything TOTALYTD cannot do: Sales YTD = CALCULATE([Total Sales], DATESYTD('Calendar'[Date])) ``` The optional last argument of TOTALYTD / DATESYTD is the year end date as text, for fiscal years ("6/30", "3/31"). The functions use the Calendar date column, never the fact table's date column. ## Prior period and growth ``` Sales PY = CALCULATE([Total Sales], SAMEPERIODLASTYEAR('Calendar'[Date])) Sales PY (alt) = CALCULATE([Total Sales], DATEADD('Calendar'[Date], -1, YEAR)) YoY $ = [Total Sales] - [Sales PY] YoY % = DIVIDE([Total Sales] - [Sales PY], [Sales PY]) Sales PM = CALCULATE([Total Sales], DATEADD('Calendar'[Date], -1, MONTH)) Sales PQ = CALCULATE([Total Sales], DATEADD('Calendar'[Date], -1, QUARTER)) ``` `DATEADD` shifts the dates in the current context by an interval; `PARALLELPERIOD` returns the whole period (not just the visible days). Both need contiguous dates in the context, so a matrix that puts Quarter on rows and Year on columns can fail with "contiguous dates"; put Year then Quarter on rows. When the current period is incomplete (a partial current year), `SAMEPERIODLASTYEAR` compares the same days, which is what you usually want. ## Rolling periods ``` Sales Rolling 12M = CALCULATE([Total Sales], DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -12, MONTH)) Sales Rolling 3M Avg = AVERAGEX(DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -3, MONTH), [Total Sales]) -- average of daily values: usually not what you want Monthly Avg Rolling 3M = VAR Months = DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -3, MONTH) RETURN DIVIDE(CALCULATE([Total Sales], Months), 3) ``` `DATESBETWEEN(Calendar[Date], start, end)` is the general form; `DATESINPERIOD(dates, startDate, number, interval)`; intervals are DAY, MONTH, QUARTER, YEAR. Use LASTDATE/MAX of the date column as the anchor so the window ends at the last visible day. ## Cumulative (running) total ``` Running Total = CALCULATE([Total Sales], FILTER(ALL('Calendar'[Date]), 'Calendar'[Date] <= MAX('Calendar'[Date]))) -- Over visible dates only (respects a year slicer): Running Total (selected) = CALCULATE([Total Sales], FILTER(ALLSELECTED('Calendar'[Date]), 'Calendar'[Date] <= MAX('Calendar'[Date]))) ``` Use a unique date or month key in the axis. A running total that restarts each year: add `'Calendar'[Year] = MAX('Calendar'[Year])` to the filter or use DATESYTD. ## Period boundaries and balances (semi-additive measures) - `FIRSTDATE`, `LASTDATE`, `STARTOFMONTH`, `ENDOFMONTH`, `STARTOFYEAR`, `ENDOFYEAR`, `OPENINGBALANCEMONTH`, `CLOSINGBALANCEMONTH` and the quarter/year variants. - Inventory, bank balances, headcount do not add up over time. Use the last value: `Closing Stock = CALCULATE(SUM(Stock[Units]), LASTDATE('Calendar'[Date]))` or, if the last day may have no row, `CALCULATE(SUM(Stock[Units]), LASTNONBLANK('Calendar'[Date], CALCULATE(COUNTROWS(Stock))))`. ## Fiscal calendars Add Fiscal Year, Fiscal Quarter and Fiscal Month Number columns to the Calendar (the examples above), sort them properly, and give TOTALYTD the fiscal year end. Do not hard code a start month in several measures; put it in the Calendar columns. ## Same period comparisons that need care - **Week based (52/53 week)**: add a Week Number and a Week Year column in the Calendar and compare with `CALCULATE([Total Sales], ALL('Calendar'), VALUES('Calendar'[Week]), 'Calendar'[Year] = MAX('Calendar'[Year]) - 1)` style filters. DATEADD by 365 days drifts. - **Working days**: add an IsWorkday flag (and holidays) to the Calendar, then `CALCULATE(COUNTROWS('Calendar'), 'Calendar'[IsWorkday] = TRUE())`. - **Month over month when a month has no data**: DATEADD returns blank for a missing prior period; show 0 with `COALESCE` only if appropriate. - **To date, but only up to the last date that has data**: `VAR LastSale = CALCULATE(MAX(Sales[OrderDate]), ALL(Sales))`, then filter the Calendar to dates <= LastSale so the current year is not compared with a full prior year. ## Pitfalls - The date column you pass must be the Calendar's, not the fact table's. - Time intelligence functions return a table of dates, so they belong in CALCULATE filters (or as the iterator table of an X function). - If you pass Year and Month-name text columns instead of a date, the functions do nothing useful. A measure that is blank in every row usually means the relationship to the Calendar is missing or the Calendar has no matching dates. - Calculation groups are the clean way to give every measure YTD/PY/YoY variants (see the patterns topic). --- # Iterators and table functions ## Aggregators versus iterators `SUM(Sales[SalesAmount])` aggregates a column. Its generalization, `SUMX(Table, expression)`, evaluates an expression for every row of a table and then sums the results. Use an X function whenever the value to add up needs row-level arithmetic: `SUMX(Sales, Sales[SalesQuantity] * Sales[UnitPrice])`. The family: SUMX, AVERAGEX, MINX, MAXX, COUNTX, COUNTAX, PRODUCTX, MEDIANX, PERCENTILEX.INC/EXC, STDEVX.P/S, RANKX, CONCATENATEX, GEOMEANX. > Diagram: SUMX visits each row of a sales table, multiplies quantity by price on that row, and adds the results to get 820. SUM of quantity alone gives 20 units. ## Average of ratios versus ratio of sums `AVERAGEX(Sales, DIVIDE(Sales[SalesAmount], Sales[SalesQuantity]))` is the simple average of line prices. `DIVIDE([Total Sales], [Units Sold])` is the weighted (quantity-weighted) average. They differ, and for money the weighted one is nearly always what is meant. Same for margin %: divide total margin by total sales instead of averaging row margins. ## Iterating over a dimension and context transition `SUMX(VALUES(Customer[Customer]), [Total Sales])` computes the measure once per customer (the measure reference triggers context transition) and adds the results. The grain of the iteration decides the answer: iterate Product to total after a per-product rule, iterate Month for monthly averages. ``` Average Sales per Customer = AVERAGEX(VALUES(Customer[CustomerKey]), [Total Sales]) Average Daily Sales = AVERAGEX(VALUES('Calendar'[Date]), [Total Sales]) Peak Month Sales = MAXX(VALUES('Calendar'[Year-Month]), [Total Sales]) Capped Commission = SUMX(VALUES(Employee[Employee]), MIN([Sales Commission], 5000)) ``` Totals that do not match the visible rows are explained by the iteration grain: the total row of a matrix is evaluated with no row filter, so a measure with an IF on a per-row condition needs an iterator over the row level to make totals add up (the "subtotal fix" in the patterns topic). ## RANKX and TOPN ``` Product Rank = RANKX(ALLSELECTED(Product[Product]), [Total Sales], , DESC, Dense) Top 10 Products Sales = CALCULATE([Total Sales], TOPN(10, ALLSELECTED(Product[Product]), [Total Sales], DESC)) Others = [Total Sales] - [Top 10 Products Sales] ``` RANKX(table, expression, [value], [order], [ties]): the table must be the set you are ranking against; use ALL or ALLSELECTED on the column in the visual so the rank is computed across the right set. In a visual that shows product rows, RANKX(ALLSELECTED(Product[Product]), ...) ranks each row among the products selected. Add ties handling (Dense or Skip). TOPN(n, table, orderBy, order, ...) returns the n rows; ties can return more than n. ## Table functions that build tables - `FILTER(table, condition)`: row by row filter. `ALL(table)`, `ALL(column)`, `VALUES`, `DISTINCT`, `ALLSELECTED`, `ALLEXCEPT`. - `ADDCOLUMNS(table, "name", expression, ...)`: adds computed columns (the new column is referred to as [name]). `SELECTCOLUMNS(table, "name", expr, ...)`: projects a new set of columns. - `SUMMARIZE(table, groupCol, ...)`: groups; avoid adding aggregate expressions in SUMMARIZE, use `ADDCOLUMNS(SUMMARIZE(...), "Total", [Total Sales])` instead. `SUMMARIZECOLUMNS(groupCols, filters, "name", expr, ...)`: the engine behind most visuals; it removes rows where all measures are blank. - `GROUPBY`: groups with iterators over CURRENTGROUP(); rarely better than SUMMARIZE. - `CROSSJOIN(t1, t2)`, `GENERATE(t1, t2)` / `GENERATEALL` (apply a table expression for every row), `NATURALINNERJOIN`, `NATURALLEFTOUTERJOIN`, `UNION`, `INTERSECT`, `EXCEPT`, `GENERATESERIES(start, end, step)`, `DATATABLE`, `ROW("name", value)`, `{...}` table constructors such as `{("Sales", 1), ("Cost", 2)}`. - `TOPN`, `SAMPLE`, `ADDMISSINGITEMS`, `ROLLUP`/`ROLLUPGROUP`/`ISSUBTOTAL` (with SUMMARIZE), `CALCULATETABLE(table, filters)`. ``` Customer Sales Table = ADDCOLUMNS( SUMMARIZE(Sales, Customer[Customer]), "Sales", [Total Sales], "Orders", [Order Lines] ) Customers Above Average = VAR Avg = AVERAGEX(ALLSELECTED(Customer[Customer]), [Total Sales]) RETURN COUNTROWS(FILTER(ALLSELECTED(Customer[Customer]), [Total Sales] > Avg)) ``` ## Counting - `COUNTROWS(table)`: rows. `COUNT(column)` counts non-blank numbers; `COUNTA` non-blank anything; `DISTINCTCOUNT(column)` and `DISTINCTCOUNTNOBLANK`. `COUNTBLANK`. `APPROXIMATEDISTINCTCOUNT` for DirectQuery speed. - Count of customers who bought: `DISTINCTCOUNT(Sales[CustomerKey])`. Count of customers in a dimension regardless of sales: `COUNTROWS(Customer)`. They differ whenever the dimension has rows without sales. - Count of items meeting a measure condition: `COUNTROWS(FILTER(VALUES(Product[Product]), [Total Sales] > 100000))`. ## Text, logic and information functions you will use constantly `IF`, `SWITCH(TRUE(), cond1, result1, cond2, result2, default)`, `AND/OR/NOT`, `COALESCE`, `ISBLANK`, `ISNUMBER`, `ISTEXT`, `ISERROR`, `IFERROR`, `CONCATENATEX(table, expr, ", ")`, `FORMAT`, `LEFT/RIGHT/MID/LEN/FIND/SEARCH/SUBSTITUTE/TRIM/UPPER/LOWER`, `VALUE`, `FIXED`, `USERNAME`, `USERPRINCIPALNAME`, `LOOKUPVALUE`, `CONTAINSROW`, `CONTAINS`, `NAMEOF`, `SELECTEDMEASURE`, `SELECTEDMEASUREFORMATSTRING`. ## Iterator performance - The cost of an iterator is rows iterated times the cost of the inner expression. Iterate a dimension column (hundreds or thousands of values), not the fact table (millions), and avoid measures inside iterators over huge tables. - `SUMX(Sales, Sales[Quantity] * Sales[UnitPrice])` over a fact table is fine: it is a simple row calculation the engine runs in one pass. Putting a measure or CALCULATE inside forces a context transition per row. - Prefer `COUNTROWS(FILTER(...))` over nested iterators, and `CALCULATE(..., column filter)` over `FILTER(table, ...)` where possible. --- # DAX pattern cookbook Recipes use a sales star schema: Sales (SalesAmount, TotalCost, SalesQuantity, OrderDate/DateKey), Calendar, Product, Category, Subcategory, Store, Channel, Geography, Customer. Adapt the names. ## Percent of total and share of parent ``` % of Grand Total = DIVIDE([Total Sales], CALCULATE([Total Sales], REMOVEFILTERS())) % of Selected = DIVIDE([Total Sales], CALCULATE([Total Sales], ALLSELECTED())) % of Column Total (Channel on columns) = DIVIDE([Total Sales], CALCULATE([Total Sales], REMOVEFILTERS(Product))) % of Parent = DIVIDE([Total Sales], CALCULATE([Total Sales], REMOVEFILTERS(Subcategory))) % of Group (any hierarchy level) = VAR Level = IF(ISINSCOPE(Product[Product]), 1, IF(ISINSCOPE(Subcategory[Subcategory]), 2, 3)) RETURN SWITCH(Level, 1, DIVIDE([Total Sales], CALCULATE([Total Sales], ALLSELECTED(Product[Product]))), 2, DIVIDE([Total Sales], CALCULATE([Total Sales], ALLSELECTED(Subcategory[Subcategory]))), DIVIDE([Total Sales], CALCULATE([Total Sales], ALLSELECTED()))) ``` ALLSELECTED() with no arguments returns all selected rows in the whole model. Keep the denominator's filter removal at the same level as the numerator's grouping. ## Year to date, prior year, growth ``` Sales YTD = TOTALYTD([Total Sales], 'Calendar'[Date]) Sales PY = CALCULATE([Total Sales], SAMEPERIODLASTYEAR('Calendar'[Date])) YoY % = DIVIDE([Total Sales] - [Sales PY], [Sales PY]) YTD vs PY YTD % = VAR cur = [Sales YTD] VAR py = CALCULATE([Sales YTD], SAMEPERIODLASTYEAR('Calendar'[Date])) RETURN DIVIDE(cur - py, py) ``` See the time intelligence topic for fiscal years, rolling 12 months, and partial years. ## Rolling average and moving window ``` Sales Rolling 12M = CALCULATE([Total Sales], DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -12, MONTH)) Avg Monthly Sales Last 12M = DIVIDE(CALCULATE([Total Sales], DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -12, MONTH)), 12) Sales Last 30 Days = CALCULATE([Total Sales], DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -30, DAY)) ``` ## Ranking and Top N with an "Others" row ``` Product Rank = IF(HASONEVALUE(Product[Product]), RANKX(ALLSELECTED(Product[Product]), [Total Sales], , DESC, Dense)) Top N Sales = VAR N = SELECTEDVALUE('Top N'[Top N Value], 10) RETURN CALCULATE([Total Sales], TOPN(N, ALLSELECTED(Product[Product]), [Total Sales], DESC)) Is Top N = IF([Product Rank] <= SELECTEDVALUE('Top N'[Top N Value], 10), 1, 0) ``` Create `Top N = GENERATESERIES(3, 20, 1)` as a disconnected What-if table. Filter a visual by the Is Top N measure (= 1). For an "Others" bar build a small table `{"Top N", "Others"}` and SWITCH on it. ## Pareto / ABC classification (cumulative percent of sales) ``` Product Sales Rank = RANKX(ALL(Product[Product]), [Total Sales], , DESC, Dense) Cumulative Sales % = VAR ThisRank = [Product Sales Rank] VAR Running = SUMX(FILTER(ALL(Product[Product]), [Product Sales Rank] <= ThisRank), [Total Sales]) RETURN DIVIDE(Running, CALCULATE([Total Sales], ALL(Product[Product]))) ABC Class = SWITCH(TRUE(), [Cumulative Sales %] <= 0.8, "A", [Cumulative Sales %] <= 0.95, "B", "C") ``` Use ALL on the product column in both the rank and the denominator so the class does not change with slicers (use ALLSELECTED to make it respond). Iterating a unique product list keeps it fast enough for a few thousand products. ## Distinct counts, new versus returning customers ``` Customers = DISTINCTCOUNT(Sales[CustomerKey]) First Purchase Date = CALCULATE(MIN(Sales[OrderDate]), ALL('Calendar')) -- per customer in a customer visual New Customers = COUNTROWS(FILTER(VALUES(Sales[CustomerKey]), CALCULATE(MIN(Sales[OrderDate]), ALL('Calendar')) >= MIN('Calendar'[Date]) && CALCULATE(MIN(Sales[OrderDate]), ALL('Calendar')) <= MAX('Calendar'[Date]))) Returning Customers = [Customers] - [New Customers] Active Customers Last 90 Days = CALCULATE(DISTINCTCOUNT(Sales[CustomerKey]), DATESINPERIOD('Calendar'[Date], MAX('Calendar'[Date]), -90, DAY)) ``` ALL('Calendar') removes the date filter so MIN is the customer's first ever purchase. > **Tip.** For performance with millions of customers, precompute First Purchase Date as a calculated column on Customer. ## Customers who bought A but not B, basket analysis ``` Customers Who Bought Computers = CALCULATE(DISTINCTCOUNT(Sales[CustomerKey]), Category[Category] = "Computers") Bought Computers and Cameras = COUNTROWS(INTERSECT( CALCULATETABLE(VALUES(Sales[CustomerKey]), Category[Category] = "Computers"), CALCULATETABLE(VALUES(Sales[CustomerKey]), Category[Category] = "Cameras and camcorders"))) Bought Computers Not Cameras = COUNTROWS(EXCEPT( CALCULATETABLE(VALUES(Sales[CustomerKey]), Category[Category] = "Computers"), CALCULATETABLE(VALUES(Sales[CustomerKey]), Category[Category] = "Cameras and camcorders"))) ``` INTERSECT, EXCEPT and UNION compare whole tables, which is the clean way to express "both", "but not" and "either". ## Budget versus actual and variance Put budget in its own fact at a coarser grain (month, department) related to the same Calendar and a shared dimension; do not force it into the sales table. Measures: ``` Actual = [Total Sales] Budget = SUM(Budget[Amount]) Variance = [Actual] - [Budget] Variance % = DIVIDE([Variance], [Budget]) Budget at Month Grain Only = IF(ISINSCOPE('Calendar'[Date]), BLANK(), [Budget]) -- hide a monthly budget at day level ``` When budget is monthly and actual is daily, hide budget at the day level, or allocate it with a day count: `DIVIDE([Budget], DAY(EOMONTH(MAX('Calendar'[Date]), 0)))`. ## Semi-additive balances (inventory, bank balance, headcount) ``` Closing Balance = CALCULATE(SUM(Balances[Amount]), LASTDATE('Calendar'[Date])) Closing Balance (last date with data) = CALCULATE(SUM(Balances[Amount]), LASTNONBLANK('Calendar'[Date], CALCULATE(COUNTROWS(Balances)))) Average Balance = AVERAGEX(VALUES('Calendar'[Date]), CALCULATE(SUM(Balances[Amount]))) ``` ## Fix wrong totals and subtotals > **Watch out.** A per-row calculation shows right in the rows but a wrong total? The total row evaluates the measure once on the whole set. Iterate the row level: ``` Commission = SUMX(VALUES(Employee[Employee]), IF([Total Sales] > 50000, [Total Sales] * 0.05, [Total Sales] * 0.03)) ``` To show a value only at the lowest level or hide subtotals: `IF(ISINSCOPE(Product[Product]), [Total Sales])`. To make a matrix show rows with no sales, add a measure like `[Total Sales] + 0` or use "Show items with no data" on the field. ## Field parameters and dynamic measure choice - Field parameters (Modeling > New parameter > Fields): add Total Sales, Total Cost and Gross Margin; Desktop creates a calculated table of `NAMEOF` entries. Put the slicer on it, and put the parameter column in a visual's value well (or axis) and the visual switches measures (or columns) with the selection. Field parameters need Power BI Desktop; Excel Power Pivot does not have them. - The older, portable way (works in Power Pivot too): a disconnected `Measure Choice` table (Enter Data) and `Selected Measure = SWITCH(SELECTEDVALUE('Measure Choice'[Measure]), "Sales", [Total Sales], "Cost", [Total Cost], [Total Sales])`. - Dynamic axis: a field parameter on columns (Category, Subcategory, Brand), or a calculated column per option plus SWITCH in a measure that chooses what to show. ## Calculation groups for time intelligence (YTD, prior year, YoY %) A calculation group applies one formula to every measure through `SELECTEDMEASURE()`, so you define YTD, PY and YoY % once instead of once per measure. Create it in Power BI Desktop (Model view > Calculation group; Desktop turns on "discourage implicit measures"), or with Tabular Editor. Items: ``` Current = SELECTEDMEASURE() YTD = CALCULATE(SELECTEDMEASURE(), DATESYTD('Calendar'[Date])) PY = CALCULATE(SELECTEDMEASURE(), SAMEPERIODLASTYEAR('Calendar'[Date])) YoY = SELECTEDMEASURE() - CALCULATE(SELECTEDMEASURE(), SAMEPERIODLASTYEAR('Calendar'[Date])) YoY % = VAR PY = CALCULATE(SELECTEDMEASURE(), SAMEPERIODLASTYEAR('Calendar'[Date])) RETURN DIVIDE(SELECTEDMEASURE() - PY, PY) -- Format string expression: "0.0%" ``` Put the calculation group column on a matrix's columns or in a slicer. Notes: calculation items do not change the data type, so give a percent item a format string expression (`"0.0%"`); precedence decides which group wins when two apply; measures defined with a calculation item inside CALCULATE interact, so test totals; in Power Pivot for Excel, calculation groups cannot be created in the UI, so use separate measures instead (Sales YTD, Sales PY, YoY %). ## Conditional formatting and dynamic labels ``` Margin Color = IF([Gross Margin %] >= 0.55, "#1B7F3B", IF([Gross Margin %] >= 0.45, "#B7791F", "#C0392B")) -- Format > Conditional formatting > Field value Sales Trend Arrow = IF([YoY %] >= 0, "▲", "▼") Dynamic Title = "Sales for " & SELECTEDVALUE('Calendar'[Year], "all years") & " in " & SELECTEDVALUE(Channel[Channel], "all channels") Top Product Label = "Best seller: " & CALCULATE(FIRSTNONBLANK(Product[Product], 1), TOPN(1, ALLSELECTED(Product[Product]), [Total Sales])) ``` Bind a color measure through Conditional formatting > Format style: Field value. Bind a dynamic title with the fx button next to Title. A measure that returns text can feed a card. ## Percentiles, median, histograms, outliers ``` Median Sale = MEDIANX(Sales, Sales[SalesAmount]) P90 Sale = PERCENTILEX.INC(Sales, Sales[SalesAmount], 0.9) StdDev = STDEVX.P(Sales, Sales[SalesAmount]) Sales Bin = SELECTEDVALUE(Bins[Bin]) -- Bins = GENERATESERIES(0, 5000, 500) Orders in Bin = VAR lo = SELECTEDVALUE(Bins[Bin]) RETURN CALCULATE(COUNTROWS(Sales), Sales[SalesAmount] >= lo, Sales[SalesAmount] < lo + 500) ``` ## Cohorts and retention Add `Customer[First Order Month]` (calculated column, MIN of order date in the customer's sales, truncated to month) and a Cohort Age measure: `DATEDIFF(MIN(Customer[First Order Month]), MAX('Calendar'[Date]), MONTH)`. Show a matrix with first order month on rows, cohort age on columns, and `DISTINCTCOUNT(Sales[CustomerKey])` as values, divided by the cohort size for a retention percentage. ## Price, volume and mix variance (outline) Compute Sales PY, Units PY and Price = Sales / Units for each period. Volume effect = (Units - Units PY) * Price PY. Price effect = (Price - Price PY) * Units. Mix needs a product-level iteration: SUMX over Product of the same formulas then subtract. Compute the effects at product level and sum, so they reconcile to the total change. ## Calendar table in Power Query (M) ``` let Start = #date(2016, 1, 1), End = #date(2018, 12, 31), Dates = List.Dates(Start, Duration.Days(End - Start) + 1, #duration(1, 0, 0, 0)), T = Table.FromList(Dates, Splitter.SplitByNothing(), {"Date"}, null, ExtraValues.Error), Typed = Table.TransformColumnTypes(T, {{"Date", type date}}), Year = Table.AddColumn(Typed, "Year", each Date.Year([Date]), Int64.Type), Month = Table.AddColumn(Year, "Month", each Date.ToText([Date], "MMMM", "en-US"), type text), MonthNo = Table.AddColumn(Month, "Month Number", each Date.Month([Date]), Int64.Type) in MonthNo ``` ## What-if parameter Modeling > New parameter > Numeric range creates a table (GENERATESERIES) and a measure `Parameter Value = SELECTEDVALUE('Discount'[Discount], 0)`. Use it inside measures: `Sales After Discount = [Total Sales] * (1 - [Discount Value])`. ## Show the selected slicer value, last refresh, and counts of selections ``` Selected Years = CONCATENATEX(VALUES('Calendar'[Year]), 'Calendar'[Year], ", ", 'Calendar'[Year], ASC) Last Refresh = "Updated " & FORMAT(MAX(RefreshLog[Timestamp]), "mmm d, yyyy h:mm AM/PM") -- RefreshLog table: a Power Query step with DateTime.LocalNow() Selected Count = COUNTROWS(VALUES(Product[Brand])) & " of " & COUNTROWS(ALL(Product[Brand])) & " brands" ``` --- # Power BI and Excel analysis recipes (how to ask for, and build, common reports) ## Step plan for "analyze X in Power BI" 1. State the question (what decision, which measure, which breakdown, which period). 2. Pick the database and tables (describe_database, describe_table), verify keys and join paths, and compute the answer in SQL first (run_sql) so you know the right number. 3. Model: fact grain, dimensions, a Calendar, relationships, base measures. 4. Build measures, then visuals; validate each visual against the SQL answer. 5. Format, add tooltips and drill, test slicers, then publish or share. ## Sales analysis report (the standard four-page layout) - Page 1 Overview: cards (Total Sales, Gross Margin %, Units, Customers), monthly line with prior-year line, bar of sales by category, map or bar by geography, year/channel slicers. - Page 2 Products: matrix of Category > Subcategory with Sales, Margin %, % of total; Top 10 table; scatter of Price vs Units sized by Margin. - Page 3 Customers or stores: Top N, ABC classification, new vs returning. - Page 4 Detail: drillthrough table with order lines. Measures to cover: Total Sales, Total Cost, Gross Margin, Gross Margin %, Units, Average Price, Sales PY, YoY %, Sales YTD, Rolling 12M, % of Total, Rank, Top N. ## Finance reports - **Actual versus budget**: Actual, Budget (separate fact), Variance, Variance %, Forecast, with conditional formatting (red/green) and a waterfall of variance by category. - **AR / AP aging**: add an Age column (days outstanding as of a report date), a bucket column (0-30, 31-60, 61-90, 90+) with sort order, and measures for open balance by bucket: `Open Balance = SUMX(Invoices, Invoices[Amount] - Invoices[Paid])`; a stacked bar by bucket. - **P&L**: Account table with a Report Order and type (Revenue, COGS, Expense); measures as amounts with sign handling (`IF(Accounts[Type] = "Revenue", -[Credit - Debit], [Credit - Debit])`); matrix with accounts on rows, months on columns; subtotals with ISINSCOPE. - **Cash flow and balances**: semi-additive measures with LASTDATE and a running total. - **Customer profitability**: sales, cost to serve, margin by customer, Pareto. ## Operations and HR Headcount (semi-additive: closing count per month), turnover % = leavers / average headcount, utilization = billable hours / available hours, backlog, cycle time with DATEDIFF, SLA % within target. ## Excel-first versions - Power Pivot: load tables to the Data Model, measures through Power Pivot > Measures, PivotTable + slicers + timeline; use CUBE formulas for fixed-layout reports. - Without Power Pivot: Power Query to shape, PivotTable on a table, SUMIFS/XLOOKUP formulas, dynamic arrays (FILTER, SORT, UNIQUE, LET) for what DAX would do (see the Excel practice workbooks on the site). - Choose Power BI when you need relationships across many tables, time intelligence on big data, interactive visuals, sharing to the web, or row-level security. Choose Excel when the audience lives in spreadsheets and the data is modest. ## Prompts that work well with this assistant - "Create a Power BI project for contoso with a time intelligence page: YTD, prior year, YoY %, rolling 12 months." - "Write DAX for new vs returning customers by month in the classicmodels database." - "How do I make a measure selector in Power BI and in Power Pivot?" - "Why does my percent of total show 100% on every row?" (check ALL vs ALLSELECTED, relationship, grain) - "Generate a sample report on dvdrental with revenue by category, top customers and a rental duration histogram." --- # DAX performance, style and troubleshooting ## Writing fast DAX - Fix the model before the formulas: star schema, integer keys, remove unused columns, reduce cardinality of text columns, split datetime into date and time, disable auto date/time. Most slowness is model shape. - Prefer a column filter inside CALCULATE to FILTER over a whole table; `FILTER(ALL(Product[Brand]), ...)` over `FILTER(Product, ...)`; iterate a dimension column over the fact table. - Use variables so an expression is evaluated once. Avoid repeating a measure many times. - Avoid IFERROR/IF inside iterators over millions of rows, nested iterators, and CALCULATE in a calculated column over a big table. - Use `DIVIDE`, `SELECTEDVALUE`, `ISINSCOPE`, `HASONEVALUE`; avoid `COUNTROWS(DISTINCT(...))` (use DISTINCTCOUNT), `VALUES` inside a Boolean filter where `IN` works. - Many distinct cards on one page each run their own queries; reduce visuals per page, use fewer high-cardinality fields in tables, and apply filters instead of showing thousands of rows. - DirectQuery: keep measures simple, avoid time intelligence over huge tables, and use aggregations or composite models. ## Measuring performance - **Performance analyzer** (View > Performance analyzer in Power BI Desktop): record, refresh visuals, see DAX query, visual display and other time. Copy a visual's query into DAX query view or DAX Studio. - **DAX Studio** (free external tool, also connects to the Excel Data Model): run queries, Server Timings (storage engine versus formula engine), Query Plan, and VertiPaq Analyzer for model size per column. - **DAX query view** (Power BI Desktop): `EVALUATE` queries run against the open model; write measures with `DEFINE MEASURE` to test them, then update the model. - **Best Practice Analyzer / Tabular Editor**: rule checks on the model; Tabular Editor also edits measures and calculation groups outside Desktop. Rule: the formula engine is single-threaded and the storage engine scans columns fast. Most speedups are about moving work to the storage engine (column filters, simple aggregations) and reducing materialization (fewer rows in iterators). ## Style conventions that prevent bugs - Qualify columns `Table[Column]`; never qualify measures `[Measure]`. - Put measures in a dedicated measure table or in the fact table, with display folders, descriptions and formats. - Name measures in plain business words ("Sales PY"), and build up from base measures. - Use `VAR` names that describe the value (`VAR TotalSalesPY = ...`), always `RETURN` on its own line, one function argument per line for long formulas, and format with the DAX formatter. - Do not hide logic in visual-level filters that nobody sees; put the rule in a measure with a name. ## Troubleshooting common errors - **"A single value for column 'X' in table 'Y' cannot be determined"**: you used a column where one value was expected, in a measure or calculated table, with several rows in context. Aggregate it (`MIN`, `MAX`, `SELECTEDVALUE`) or iterate with row context. - **"A table of multiple values was supplied where a single value was expected"**: a comparison like `Sales[ProductKey] = VALUES(Product[ProductKey])` with many values. Use `IN` or `SELECTEDVALUE`. - **"A circular dependency was detected"**: a calculated column or table depends on itself through relationships or other calculated objects, often after adding RELATED across a relationship that itself depends on the column. Break the chain, move logic to a measure, or use a different column. - **"The expression refers to multiple columns. Multiple columns cannot be converted to a scalar value"**: a column reference in a measure outside any iterator. Wrap in an aggregate. - **"Cannot find table / column 'X'"**: renamed or misspelled objects; table names with spaces need quotes `'Order Details'[Quantity]`. - **Measure shows the same number on every row**: the table you group by is not related to the fact, the relationship is inactive or in the wrong direction, or you removed filters with ALL. Check Model view. - **Totals do not equal the sum of the rows**: non-additive measure (distinct count, ratio), IF inside the measure evaluated once at the total level, or filtered rows hidden by blanks. Iterate the row level or accept non-additivity. - **Blank or wrong time intelligence**: Calendar not related, not contiguous, date column not marked, or you passed the fact table's date column. Also `DATEADD` fails with Quarter on rows and Year on columns. - **Blank row (Blank) in a visual**: fact keys with no dimension row. Fix data or filter it. - **"Can't determine relationships between the fields"**: two columns from unrelated tables in one visual (also a field parameter that lists only its parameter column). - **Visual too slow or "Resources exceeded"**: a high-cardinality column in a table or matrix, or a measure that materializes a huge table. Add filters, Top N, or simplify. - **Numbers differ from SQL**: filter context from a slicer or page filter, an inactive relationship, a Calendar that excludes dates, or different join logic (inner versus left). Rebuild the SQL with the same joins and filters. - **Refresh error: cyclic reference in Power Query**: sometimes appears on the first refresh of a project opened from a zip; refresh again, and extract the zip fully before opening the .pbip. - **".pbip fails to open: required artifact missing"**: the .pbip was opened from inside the zip. Extract everything first. ## Debugging technique Reproduce the number in a table visual with the pieces beside it: show the grouping column, the unfiltered measure, the filtered measure, and the count of rows (`COUNTROWS(Sales)`). In DAX query view, `EVALUATE SUMMARIZECOLUMNS(...)` with the same filters shows what the measure sees. Replace measures with their base parts one by one until the number changes. Use `CONCATENATEX(VALUES(...))` measures to display what a context holds. --- # DAX function reference (quick) Signatures use [optional] arguments. "Scalar" means returns one value, "table" means returns a table. ## Aggregation - `SUM(Column)`, `AVERAGE(Column)`, `MIN(Column)`, `MAX(Column)`: aggregate a column (MIN/MAX also take two scalars: `MAX(a, b)`). `COUNT(Column)` non-blank numbers, `COUNTA(Column)` non-blank values, `COUNTBLANK(Column)`, `COUNTROWS(Table)`, `DISTINCTCOUNT(Column)`, `DISTINCTCOUNTNOBLANK(Column)`, `APPROXIMATEDISTINCTCOUNT(Column)`. - Iterator versions take a table and an expression evaluated per row: `SUMX`, `AVERAGEX`, `MINX`, `MAXX`, `COUNTX`, `COUNTAX`, `PRODUCTX`, `MEDIANX`, `GEOMEANX`, `STDEVX.S`, `STDEVX.P`, `VARX.S`, `VARX.P`, `PERCENTILEX.INC`, `PERCENTILEX.EXC`, `CONCATENATEX(Table, Expr, [Delimiter], [OrderBy], [Order])`, `RANKX(Table, Expr, [Value], [Order], [Ties])`. - Column statistics: `MEDIAN`, `PERCENTILE.INC`, `PERCENTILE.EXC`, `STDEV.S`, `STDEV.P`, `VAR.S`, `VAR.P`, `GEOMEAN`. ## Filter and context - `CALCULATE(Expr, [Filter1], ...)`: evaluate Expr in a modified filter context. `CALCULATETABLE(Table, [Filter1], ...)`: same, returns a table. - `FILTER(Table, Condition)`: rows where Condition is true. `ALL([Table | Column ...])`: all rows/values, ignoring filters. `ALLEXCEPT(Table, Column1, ...)`. `ALLSELECTED([Table | Column])`. `ALLNOBLANKROW`. `REMOVEFILTERS([Table | Column])` (CALCULATE modifier). `KEEPFILTERS(Filter)`. `VALUES(Column | Table)`. `DISTINCT(Column | Table)`. `SELECTEDVALUE(Column, [Alternate])`. `HASONEVALUE(Column)`, `HASONEFILTER(Column)`. `ISFILTERED(Column | Table)`, `ISCROSSFILTERED(Column)`, `ISINSCOPE(Column)`, `ISEMPTY(Table)`. - `USERELATIONSHIP(Col1, Col2)`, `CROSSFILTER(Col1, Col2, Direction)` (NONE, ONEWAY, BOTH), `TREATAS(Table, Col1, ...)`, `RELATED(Column)`, `RELATEDTABLE(Table)`, `LOOKUPVALUE(Result, SearchCol, SearchValue, ..., [Alternate])`. - `EARLIER(Column, [N])` and `EARLIEST`: older row-context navigation; prefer variables. - `SELECTEDMEASURE()`, `SELECTEDMEASURENAME()`, `SELECTEDMEASUREFORMATSTRING()`, `ISSELECTEDMEASURE(m1, ...)`: calculation groups. ## Table construction and shaping `ADDCOLUMNS(Table, "Name", Expr, ...)`, `SELECTCOLUMNS(Table, "Name", Expr, ...)`, `SUMMARIZE(Table, GroupCol1, ..., ["Name", Expr, ...])`, `SUMMARIZECOLUMNS(GroupCol1, ..., [FilterTable ...], "Name", Expr, ...)`, `GROUPBY`, `TOPN(N, Table, OrderBy, [Order], ...)`, `SAMPLE`, `ROW("Name", Expr, ...)`, `DATATABLE("Col", Type, {{...}})`, `GENERATESERIES(Start, End, [Step])`, `CALENDAR(Start, End)`, `CALENDARAUTO([FiscalYearEndMonth])`, `CROSSJOIN`, `GENERATE`, `GENERATEALL`, `NATURALINNERJOIN`, `NATURALLEFTOUTERJOIN`, `UNION`, `INTERSECT`, `EXCEPT`, `ADDMISSINGITEMS`, `TREATAS`, `ROLLUP`, `ROLLUPGROUP`, `ISSUBTOTAL`, `CURRENTGROUP()`, `CONTAINS(Table, Col, Value, ...)`, `CONTAINSROW(Table | {list}, Value)`, `CONTAINSSTRING(Text, Within)`, `IN`. ## Logical and information `IF(Test, True, [False])`, `IF.EAGER`, `SWITCH(Expr, Value1, Result1, ..., [Else])` (use `SWITCH(TRUE(), ...)` for conditions), `AND(a, b)`, `OR(a, b)`, `NOT(a)`, `TRUE()`, `FALSE()`, `COALESCE(a, b, ...)`, `IFERROR(Expr, Alt)`, `ISBLANK`, `ISERROR`, `ISLOGICAL`, `ISNUMBER`, `ISTEXT`, `ISNONTEXT`, `ISEVEN`, `ISODD`, `BLANK()`, `ERROR("message")`, `NAMEOF(Column | Measure)`, `USERNAME()`, `USERPRINCIPALNAME()`, `USEROBJECTID()`, `CUSTOMDATA()`, `HASONEVALUE`, `PATH`, `PATHITEM`, `PATHLENGTH`, `PATHCONTAINS`, `PATHITEMREVERSE` (parent-child hierarchies). ## Math and statistics `DIVIDE(Num, Den, [Alt])`, `ABS`, `ROUND(n, digits)`, `ROUNDUP`, `ROUNDDOWN`, `MROUND`, `CEILING`, `FLOOR`, `INT`, `TRUNC`, `MOD`, `QUOTIENT`, `POWER`, `SQRT`, `EXP`, `LN`, `LOG`, `LOG10`, `SIGN`, `FACT`, `RAND`, `RANDBETWEEN`, `PI`, `CURRENCY`, `CONVERT`, trig functions, `XIRR(Table, Values, Dates, [Guess])`, `XNPV`, `IRR`, `NPV`, `PMT`, `FV`, `PV`, `RATE`, `NPER`, `SLN`, `DB`, `DDB`. ## Text `CONCATENATE`, `&`, `CONCATENATEX`, `LEFT`, `RIGHT`, `MID`, `LEN`, `FIND` (case sensitive, errors), `SEARCH` (case insensitive), `SUBSTITUTE`, `REPLACE`, `TRIM`, `UPPER`, `LOWER`, `PROPER`, `REPT`, `EXACT`, `VALUE`, `FORMAT(Value, "format")`, `FIXED`, `UNICHAR`, `UNICODE`, `TOCSV`, `TOJSON`. FORMAT strings: dates `"yyyy-MM-dd"`, `"mmm yyyy"`, `"dddd"`; numbers `"#,0.00"`, `"0.0%"`, `"$#,0"`. ## Date and time `DATE(y, m, d)`, `TIME`, `YEAR`, `MONTH`, `DAY`, `HOUR`, `MINUTE`, `SECOND`, `QUARTER`, `WEEKDAY(Date, [Type])` (2 = Monday is 1), `WEEKNUM(Date, [Type])`, `EOMONTH(Date, Months)`, `EDATE`, `TODAY()`, `NOW()`, `UTCTODAY`, `UTCNOW`, `DATEDIFF(Start, End, Interval)` (SECOND ... YEAR), `DATEVALUE`, `TIMEVALUE`, `YEARFRAC`. ## Time intelligence (need a date column of a proper date table) - Period to date: `DATESYTD(Dates, [YearEnd])`, `DATESQTD`, `DATESMTD`, `TOTALYTD(Expr, Dates, [Filter], [YearEnd])`, `TOTALQTD`, `TOTALMTD`. - Shifts: `SAMEPERIODLASTYEAR(Dates)`, `DATEADD(Dates, N, Interval)`, `PARALLELPERIOD(Dates, N, Interval)`, `PREVIOUSDAY/MONTH/QUARTER/YEAR`, `NEXTDAY/MONTH/QUARTER/YEAR`. - Windows: `DATESBETWEEN(Dates, Start, End)`, `DATESINPERIOD(Dates, Start, N, Interval)`. - Boundaries: `FIRSTDATE`, `LASTDATE`, `FIRSTNONBLANK(Col, Expr)`, `LASTNONBLANK(Col, Expr)`, `FIRSTNONBLANKVALUE`, `LASTNONBLANKVALUE`, `STARTOFMONTH/QUARTER/YEAR`, `ENDOFMONTH/QUARTER/YEAR`, `OPENINGBALANCEMONTH/QUARTER/YEAR`, `CLOSINGBALANCEMONTH/QUARTER/YEAR`. ## Window and visual-level functions (newer) `INDEX`, `OFFSET`, `WINDOW`, `RANK`, `ROWNUMBER`, `PARTITIONBY`, `ORDERBY`, `MATCHBY`: functions that navigate an ordered table, great for running totals and previous-row comparisons without time intelligence. Visual calculations add `RUNNINGSUM`, `MOVINGAVERAGE`, `PREVIOUS`, `NEXT`, `FIRST`, `LAST`, `COLLAPSE`, `EXPAND`, `RANGE` which refer to the visual's own axes. ``` Running Total (window) = SUMX(WINDOW(1, ABS, 0, REL, ALLSELECTED('Calendar'[Year-Month]), ORDERBY('Calendar'[Year-Month])), [Total Sales]) Previous Month Sales = CALCULATE([Total Sales], OFFSET(-1, ALLSELECTED('Calendar'[Year-Month]), ORDERBY('Calendar'[Year-Month]))) ``` Requirements vary by build; check Desktop's version and the documentation if a function is not recognized. ## Relationships and security `RELATED`, `RELATEDTABLE`, `USERELATIONSHIP`, `CROSSFILTER`, `TREATAS`, `USERPRINCIPALNAME()`, `USERNAME()`, `CUSTOMDATA()`. ## Query language (DAX query view, DAX Studio) ``` DEFINE VAR StartDate = DATE(2017, 1, 1) MEASURE Sales[Test Sales] = SUM(Sales[SalesAmount]) EVALUATE SUMMARIZECOLUMNS('Calendar'[Year], Category[Category], "Sales", [Test Sales]) ORDER BY 'Calendar'[Year], [Sales] DESC ``` `EVALUATE` returns a table; `DEFINE` creates query-scope VARs, MEASUREs, COLUMNs and TABLEs; `ORDER BY` sorts, `START AT` pages. INFO functions (`INFO.TABLES()`, `INFO.COLUMNS()`, `INFO.MEASURES()`, `INFO.RELATIONSHIPS()`) list model metadata in DAX query view. ## Operators precedence summary `^`, then unary minus, `* /`, `+ -`, `&`, comparisons, `NOT`, `&&`, `||`. Use parentheses when mixing. `IN` accepts a table or a list `{1, 2}`; comparing a column to a list needs a column on the left. --- # DAX deep end: how the engine evaluates your measure This guide is for when your measures work but you do not yet know **why** they work, or why one is fast and another is slow. It assumes [DAX fundamentals](/analyze/fundamentals) and [CALCULATE and filter context](/analyze/calculate). Read it slowly. The ideas build on each other. ## The two engines A DAX query is answered by two cooperating engines. | Engine | Job | Character | |---|---|---| | **Formula engine (FE)** | Parses DAX, builds the query plan, does everything that is not a simple scan: iterating row by row, branching with `IF` and `SWITCH`, complex calculations | Single-threaded. Flexible and slow. | | **Storage engine (SE)** | Scans compressed column data (the in-memory VertiPaq store) and returns aggregated results for simple requests | Multi-threaded and very fast. | Most DAX tuning is the art of **pushing work from the formula engine to the storage engine**. A simple `SUM(Sales[Amount])` filtered by a few columns is one fast storage-engine scan. A measure that loops over a million rows and calls `IF` on each is the formula engine doing it one row at a time. ## VertiPaq in one page VertiPaq stores each column separately and compresses it. What matters for you: - **Cardinality is the cost.** A column with few distinct values (a status, a year) compresses to almost nothing. A column with millions of distinct values (an ID, a timestamp, a free-text comment) is large and slow to scan. - **Columns, not rows.** A query that touches three columns reads three columns, however many exist. - **Dictionary encoding**: each distinct value is stored once and rows hold a small number pointing at it. This is why `DISTINCTCOUNT` on a low-cardinality column is quick. - **Relationships are lookups** between columns' dictionaries, so key columns with many distinct values cost memory on both sides. - **Calculated columns are stored data**, computed at refresh, taking memory. **Measures are computed at query time** and take none. Prefer measures, and push row-by-row calculations back to the source or [Power Query](/analyze/power-query-m-basics). - Split a datetime into a **date** and a **time** column. A full timestamp has enormous cardinality; the two halves have very little. ## Evaluation context, precisely Two kinds of context exist, and they behave differently. **Filter context** is a set of filters on columns. Every cell of a visual has one: the row's category, the column's year and any slicers combine into a filter context, and the measure is evaluated **within** it. `CALCULATE` changes it. **Row context** means "the current row of one table" and exists inside calculated columns and iterators (`SUMX`, `FILTER`, `ADDCOLUMNS`). It does **not** filter anything by itself. ### Context transition `CALCULATE` turns every row context that exists into an equivalent filter context. A measure reference is silently wrapped in `CALCULATE`, so: ```dax Customer Sales = SUMX ( Customer, [Total Sales] ) ``` Inside the iterator the row context is the current customer. Calling the measure `[Total Sales]` triggers a transition: that row becomes a filter on all the customer's columns, so `[Total Sales]` returns that customer's sales. Two consequences: 1. A **column** reference inside an iterator is just that row's value. A **measure** reference is the same row **as a filter**, which may hit more than one row if the table has duplicates. 2. Transition across a big table costs a lot. Each row becomes a filter and an evaluation. This is the single most common cause of slow iterators. ## How CALCULATE evaluates When `CALCULATE ( , , )` runs, in order: 1. The **filter arguments** are evaluated in the **original** filter context. 2. A row context in effect is turned into filters (context transition). 3. The filters are **applied** to the filter context: `ALL` and `REMOVEFILTERS` clear columns first, and other filters **replace** any existing filter on the same column. (Use `KEEPFILTERS` to intersect instead of replace.) 4. The expression is evaluated in the new context. Because step 1 uses the old context, a filter argument can read the current cell (for example with `SELECTEDVALUE`) and then use it to change the filter. Filter arguments such as `Sales[Region] = "West"` are shorthand for `FILTER ( ALL ( Sales[Region] ), Sales[Region] = "West" )`: a one-column table of allowed values. That is also why `CALCULATE ( [x], Sales[Region] = "West" )` **replaces** a region slicer rather than adding to it. ### ALL, ALLEXCEPT, ALLSELECTED, VALUES | Function | In a filter argument it... | |---|---| | `ALL ( Table[Col] )` | removes filters on that column (and, with `ALL(Table)`, all columns of the table) | | `ALLEXCEPT ( Table, Table[Col] )` | removes every filter on the table except those columns | | `ALLSELECTED ( ... )` | restores the filters coming from outside the visual (slicers, page filters) while ignoring the row and column the cell sits in | | `VALUES ( Table[Col] )` | the distinct values currently visible, plus a blank if one is needed | | `KEEPFILTERS ( ... )` | intersect with the existing filter instead of replacing it | `ALLSELECTED` is the usual choice for "percent of the visible total". `ALL` ignores slicers too, so a percent of `ALL` changes when you slice. ## Filter propagation and relationships A filter on a dimension reaches the fact table through the relationship, **from the one side to the many side**. A filter on the fact table does **not** reach the dimension unless the relationship is bidirectional (or you use `CROSSFILTER` or `TREATAS`). Under the hood the engine behaves as if the fact table were **expanded** to contain the columns of every table on its one side. That is why `RELATED` works in a row context on the many side, and why filtering `Product[Color]` is like filtering a column of the expanded Sales table. Know these tools: - `USERELATIONSHIP ( Sales[ShipDate], 'Date'[Date] )` activates an inactive relationship **inside** `CALCULATE` for that evaluation. It is how you get "sales by ship date" next to "sales by order date". - `TREATAS ( VALUES ( Other[Key] ), Sales[Key] )` applies the values of one column as a filter on an unrelated column. It is the way to use a **virtual relationship** without changing the model, and the usual fix for many-to-many without a bridge. - `CROSSFILTER ( Sales[Key], Dim[Key], BOTH )` changes the direction for one calculation. - **Lineage** tracks which column a filter comes from. `TREATAS` and `SELECTCOLUMNS` can change it. Many "my filter does nothing" bugs are lost lineage. ## Totals: why they differ At a grand total the filter context has no filter on the row column, so the measure sees **all** rows. A measure that is not additive (a ratio, a distinct count, a rank) does not equal the sum of its rows. ```dax Avg Price = DIVIDE ( [Total Sales], [Units Sold] ) ``` The total uses total sales over total units, which is correct. For a measure that must **act per row and add up**, iterate the row set so each row is calculated, then summed: ```dax Total Margin Rounded = SUMX ( VALUES ( Product[Product] ), ROUND ( [Margin], 0 ) ) ``` `ISINSCOPE ( Product[Product] )` returns true when the row being evaluated is a product, so you can behave differently on the subtotal and total rows. `HASONEVALUE` and `SELECTEDVALUE` answer "is there exactly one value in context". ## Variables and evaluation `VAR` computes an expression **once**, in the context where the `VAR` is written, and the `RETURN` expression reuses the result. ```dax Sales Growth = VAR Current = [Total Sales] VAR Prior = CALCULATE ( [Total Sales], SAMEPERIODLASTYEAR ( 'Date'[Date] ) ) RETURN DIVIDE ( Current - Prior, Prior ) ``` > **Watch out.** A variable's value is **fixed** at the point it was defined. A variable defined outside `CALCULATE` does not change when `CALCULATE` changes the filter context inside it. This makes variables useful (compute once) and sometimes the cause of "why is it not responding to my filter" (the value was frozen before the filter). ## Reading query plans: where time goes > **Tip.** Use **Performance Analyzer** in Power BI Desktop to find the slow visual and copy its DAX query. Run it in **DAX Studio** with **Server Timings** and **Query Plan** on. | You see | It means | What to try | |---|---|---| | Many fast **SE queries**, long **FE** time | Row-by-row work in the formula engine | Remove iterators over big tables, avoid `IF` inside iterators, pre-compute in the source | | One huge SE scan with large **materialization** | Pulled a lot of rows into the FE | Filter earlier; use columns of the dimension rather than the fact | | Many repeated SE queries differing only slightly | Transition or nested iterators | Reduce nesting; compute once in a variable | | **Callbacks** (`CallbackDataID`) in the SE query | The SE had to call the FE for each row (often `IF`, `SWITCH`, `DIVIDE` in an iterator, or a format) | Move the logic out of the iterator; use `DIVIDE` outside | | Big **cache** hits on the second run | The result was cached | Clear the cache between tests for honest timings | ### Habits that make DAX fast - **Filter columns, not tables.** `Sales[Region] = "West"` is a column filter the SE handles easily. `FILTER ( Sales, ... )` iterates the whole fact table. - **Use measures in the visual, not columns**, and hide columns people should not aggregate. - **`SUMMARIZECOLUMNS`** is what visuals use. It is usually faster than `SUMMARIZE` plus `ADDCOLUMNS` for building tables, and applies "auto-exist" so combinations that do not exist are not returned. - **Avoid bidirectional relationships** and many-to-many on large tables unless needed. - **Keep distinct counts to low-cardinality columns** when you can, or pre-aggregate. - **Reduce cardinality**: round decimals, split datetimes, drop unused columns. - **One date table**, marked as the date table, and relationships on date (or integer) keys. ## Patterns that look simple and are not | Request | Trap | Safer approach | |---|---|---| | "Percent of total" | `ALL` ignores slicers; users expect the visible total | `ALLSELECTED` for percent of visible, `ALL` for percent of everything | | "Top 5 customers" with a slicer | `TOPN` inside `CALCULATE` computes in the wrong context | Build the set in a `VAR`, then apply it with `TREATAS` or `KEEPFILTERS` | | "Prior year" on a partial year | Prior year full vs this year partial | `DATESYTD` and `SAMEPERIODLASTYEAR` together, or compare to the same elapsed days | | "Running total" | Window grows with every row, slow on big models | Use `WINDOW` or an index-based pattern; test timing | | "Distinct count of customers who bought X and Y" | Row context does not intersect two filters | Compute both sets as tables, intersect with `INTERSECT` | | "Average of a measure per month" | `AVERAGEX ( VALUES ( 'Date'[Month] ), [Measure] )` is right; averaging the cell values by hand is not | Iterate the level you want to average over | ## A debugging routine 1. **Reduce**: put the measure in a table visual with one column that shows the problem. 2. **Break it up**: return each `VAR` as the final result, one at a time. 3. **Inspect context**: add `COUNTROWS`, `VALUES` or `CONCATENATEX` measures that show what is filtered. 4. **Test the model first**: is the relationship active, the right direction, on the right key? 5. **Compare to SQL**: write the same question in [SQL](/analyze/sql-essentials) against the source. If it differs, the problem is the model, not the measure. ## Where next - [Pattern cookbook](/analyze/patterns): finished measures to adapt. - [Performance and troubleshooting](/analyze/performance): a shorter, practical version of the tuning section here. - [Function reference](/analyze/functions): look up any function. - [Relationships and the data model](/analyze/relationships): the model decisions that make everything else easy. --- # SQL for accountants Accountants ask the same questions every close: who owes us money, does the trial balance tie, did we bill what we delivered, did anything post twice. SQL answers those questions directly, and the answers can be checked. This guide gives you the queries a controller, auditor or finance analyst uses most. Each one was run against the LearnSQL sample databases before it appeared here. Every query is a read-only `SELECT` in SQL Server (T-SQL), with MariaDB notes where the syntax changes. New to SQL? Start with [SQL essentials](/analyze/sql-essentials). The window functions are explained in [SQL deep end](/analyze/sql-deep-end). ## The sample databases | Database | What it is | Tables used here | |---|---|---| | `classicmodels` | A model-car wholesaler | `customers`, `orders`, `orderdetails` | | `cpafirm` | A CPA firm: engagements, time, invoices, payments, a general ledger | `invoices`, `invoice_lines`, `payments`, `payment_applications`, `invoice_adjustments`, `engagements`, `time_entries`, `gl_journal_lines`, `gl_journal_entries`, `gl_accounts`, `dataset_info` | | `nonprofit` | A regional nonprofit with fund accounting | `gl_journal_lines`, `gl_journal_entries`, `gl_accounts`, `funds`, `grants` | `cpafirm` and `nonprofit` tables sit in the `dbo` schema, so `invoices` and `dbo.invoices` are the same. `classicmodels` uses its database name as the schema. In MariaDB, drop every `dbo.` prefix. **Dates.** `classicmodels` dates shift every day, so never hard-code them. `cpafirm` and `nonprofit` have a fixed as-of date, 2026-10-24, in `dataset_info`. Count the rows a total comes from, then tie it to a second number. A total you have not tied is an opinion. ## Sampling and audit trails A **systematic sample** takes a fixed slice of rows, so anyone can rerun it and get the same entries. This takes entries whose number leaves a remainder of 7 when divided by 1000, about 0.1 percent: ```sql SELECT journal_entry_id, entry_date, source_type, memo FROM dbo.gl_journal_entries WHERE journal_entry_id % 1000 = 7 ORDER BY journal_entry_id; ``` A **random sample** (`ORDER BY NEWID()`) differs on every run, so save the result. An **audit trail** rebuilds a posting from its date, source, poster and memo. In `nonprofit`, the monthly entries and one opening entry lack a source ID: ```sql SELECT source_type, COUNT(*) AS entries, COUNT(DISTINCT posted_by) AS posters, SUM(CASE WHEN source_id IS NULL THEN 1 ELSE 0 END) AS no_source_id FROM dbo.gl_journal_entries GROUP BY source_type ORDER BY entries DESC; ``` **First control: every entry balances.** Run this before any other ledger test. In `nonprofit` it checks 34,377 entries and finds none unbalanced. ```sql SELECT COUNT(*) AS entries_checked, SUM(CASE WHEN x.dr <> x.cr THEN 1 ELSE 0 END) AS unbalanced FROM (SELECT e.journal_entry_id, SUM(l.debit) AS dr, SUM(l.credit) AS cr FROM dbo.gl_journal_entries AS e JOIN dbo.gl_journal_lines AS l ON l.journal_entry_id = e.journal_entry_id GROUP BY e.journal_entry_id) AS x; ``` ## 1. A/R aging buckets **Question.** How much do clients owe, and how old is it? Build each open balance from the base tables (invoice total, less payments applied, less adjustments such as write-offs), then bucket it by days past the as-of date. ```sql WITH paid AS ( SELECT invoice_id, SUM(amount_applied) AS paid_amount FROM payment_applications GROUP BY invoice_id ), adj AS ( SELECT invoice_id, SUM(amount) AS adjusted_amount FROM invoice_adjustments GROUP BY invoice_id ), open_items AS ( SELECT i.invoice_id, i.due_date, i.total_amount - ISNULL(p.paid_amount, 0) - ISNULL(a.adjusted_amount, 0) AS open_amount FROM invoices AS i LEFT JOIN paid AS p ON p.invoice_id = i.invoice_id LEFT JOIN adj AS a ON a.invoice_id = i.invoice_id ), aged AS ( SELECT open_amount, DATEDIFF(day, due_date, (SELECT as_of_date FROM dataset_info)) AS days_past_due FROM open_items WHERE open_amount > 0.005 ) SELECT SUM(CASE WHEN days_past_due <= 0 THEN open_amount ELSE 0 END) AS current_amt, SUM(CASE WHEN days_past_due BETWEEN 1 AND 30 THEN open_amount ELSE 0 END) AS d1_30, SUM(CASE WHEN days_past_due BETWEEN 31 AND 60 THEN open_amount ELSE 0 END) AS d31_60, SUM(CASE WHEN days_past_due BETWEEN 61 AND 90 THEN open_amount ELSE 0 END) AS d61_90, SUM(CASE WHEN days_past_due > 90 THEN open_amount ELSE 0 END) AS over_90, SUM(open_amount) AS total_open, COUNT(*) AS open_invoices FROM aged; ``` **What it returns.** One row. Most of the open balance is current, and the over-90 bucket is a smaller but real share. The open total is about 566 thousand dollars across 658 invoices. **Control.** The buckets sum to the total in `vw_ar_aging`, and the open total matches `vw_invoice_balances`. **MariaDB.** `IFNULL` for `ISNULL`, and `DATEDIFF((SELECT as_of_date FROM dataset_info), due_date)`, because MariaDB's `DATEDIFF` takes two dates and no unit. > Exercise: Which clients have more than 1,000 that is over 90 days past due? Show the client name and the amount over 90 days, largest first. ## 2. Trial balance **Question.** Do the balances net to zero, and what is each balance at the as-of date? ```sql SELECT a.account_number, a.account_name, a.account_type, SUM(l.debit) AS debits, SUM(l.credit) AS credits, SUM(l.debit - l.credit) AS net FROM gl_journal_lines AS l JOIN gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE e.entry_date <= (SELECT as_of_date FROM dataset_info) GROUP BY a.account_number, a.account_name, a.account_type ORDER BY a.account_number; ``` **What it returns.** About twenty rows in account order. Assets and expenses net positive; liabilities, equity and revenue net negative. The receivables line equals the open total from query 1. > **Audit tip.** Total debits equal total credits, and the net column sums to zero. In `cpafirm` both hold to the cent. > Exercise: Across the whole general ledger, what are the total debits, the total credits and the difference? Round each figure to cents. Return one row. ## 3. Revenue by month **Question.** How much did we sell each month? Group by the first day of the month. Grouping by `MONTH()` alone merges January across years. ```sql SELECT DATEFROMPARTS(YEAR(o.orderDate), MONTH(o.orderDate), 1) AS month, COUNT(DISTINCT o.orderNumber) AS orders, SUM(d.quantityOrdered * d.priceEach) AS sales FROM classicmodels.orders AS o JOIN classicmodels.orderdetails AS d ON d.orderNumber = o.orderNumber GROUP BY DATEFROMPARTS(YEAR(o.orderDate), MONTH(o.orderDate), 1) ORDER BY month; ``` **What it returns.** About 29 monthly rows. The labels move with the sample dates, but the shape of the series holds. **Control.** The sales column sums to `SUM(quantityOrdered * priceEach)` over `orderdetails` with no grouping. **MariaDB.** Use `DATE_FORMAT(o.orderDate, '%Y-%m-01')` in both the select list and the `GROUP BY`. ## 4. Year over year **Question.** How did each month compare with the same month last year? Compare complete periods only. A partial year against a full prior year makes every month look like a decline. ```sql WITH monthly AS ( SELECT YEAR(issue_date) AS yr, MONTH(issue_date) AS mo, SUM(total_amount) AS billed FROM invoices WHERE YEAR(issue_date) IN (2024, 2025) GROUP BY YEAR(issue_date), MONTH(issue_date) ) SELECT cy.mo, py.billed AS billed_2024, cy.billed AS billed_2025, ROUND((cy.billed - py.billed) * 100.0 / NULLIF(py.billed, 0), 1) AS yoy_pct FROM monthly AS cy JOIN monthly AS py ON py.mo = cy.mo AND py.yr = 2024 WHERE cy.yr = 2025 ORDER BY cy.mo; ``` **What it returns.** Twelve rows. In `cpafirm` 2025 is above 2024 in every month, by about 4 to 40 percent, unevenly. > **Tip.** `NULLIF` stops a zero prior month from dividing by zero. **Control.** The two billed columns sum to the annual totals in `invoices`. ## 5. Duplicate detection **Question.** Did we record the same payment twice? Test two things, and never delete on one test alone. First, same client, date and amount. ```sql SELECT p.payment_id, p.client_id, p.payment_date, p.amount, p.payment_method, p.reference_number, p.received_by FROM payments AS p WHERE EXISTS (SELECT 1 FROM payments AS q WHERE q.client_id = p.client_id AND q.payment_date = p.payment_date AND q.amount = p.amount AND q.payment_id <> p.payment_id) ORDER BY p.client_id, p.payment_date; ``` **What it returns.** Pairs, usually with different methods and reference numbers. In `cpafirm` there are a few dozen rows. They may be split payments, so treat them as a review list. Second, a repeated reference number: ```sql SELECT reference_number, COUNT(*) AS copies, COUNT(DISTINCT client_id) AS clients, MIN(amount) AS min_amt, MAX(amount) AS max_amt FROM payments GROUP BY reference_number HAVING COUNT(*) > 1 ORDER BY copies DESC; ``` **What it returns.** In `cpafirm` every repeated reference belongs to different clients with different amounts. That is a problem with how references are generated, not a double payment. A true duplicate repeats the client and the amount, and this sample has none. > Exercise: Which payment reference numbers are used by three or more payments? Show the reference, the number of payments and the number of different clients. Most used first, then by reference. ## 6. Three-way match **Question.** For each fixed-fee engagement, do the agreed fee, the time worked and the billing agree? Compare the order (the engagement fee), the delivery (recorded time) and the bill (invoice lines). A billing above a fixed fee is an exception. ```sql WITH billed AS ( SELECT engagement_id, SUM(amount) AS billed_amount FROM invoice_lines GROUP BY engagement_id ), matched AS ( SELECT e.engagement_id, e.fee_amount AS agreed_fee, ISNULL(b.billed_amount, 0) AS billed_amount, CASE WHEN ISNULL(b.billed_amount, 0) = 0 THEN 'Work recorded, not billed' WHEN ISNULL(b.billed_amount, 0) > e.fee_amount THEN 'Billed above fixed fee' ELSE 'Billed within fixed fee' END AS match_status FROM engagements AS e LEFT JOIN billed AS b ON b.engagement_id = e.engagement_id WHERE e.fiscal_year = 2025 AND e.fee_type = 'Fixed' AND e.status = 'Complete' ) SELECT match_status, COUNT(*) AS engagements, SUM(agreed_fee) AS agreed, SUM(billed_amount) AS billed FROM matched GROUP BY match_status ORDER BY engagements DESC; ``` **What it returns.** Most completed fixed-fee engagements in fiscal 2025 were billed within the fee, and for that group the agreed and billed totals are equal. A smaller group was billed above the fee: those are the exceptions to review. The "not billed" row is empty for completed work. **Note.** To add hours, sum `time_entries` per engagement in its own CTE. Do not join time to invoice lines directly, or rows multiply (section 13). **MariaDB:** `IFNULL`. ## 7. Gaps in sequence numbers **Question.** Are numbered documents complete? `LEAD` looks at the next row in order, and a gap shows as a jump greater than one. ```sql SELECT x.orderNumber, x.nxt, x.nxt - x.orderNumber - 1 AS missing FROM (SELECT orderNumber, LEAD(orderNumber) OVER (ORDER BY orderNumber) AS nxt FROM classicmodels.orders) AS x WHERE x.nxt - x.orderNumber > 1 ORDER BY x.orderNumber; ``` **What it returns.** No rows. The order numbers are complete. An empty result is a result, and it is the one you want. **Control:** on `gl_journal_entries` in `cpafirm` the same test returns no rows, and the distinct count equals the maximum minus the minimum plus one. ## 8. Running balance **Question.** What is a client's balance after each invoice and payment? Put invoices and payments in one list, with payments negative, and sum from the first row to the current row. ```sql WITH lines AS ( SELECT i.issue_date AS txn_date, i.invoice_number AS ref, 'Invoice' AS kind, i.total_amount AS amount FROM invoices AS i WHERE i.client_id = 812 UNION ALL SELECT p.payment_date, p.reference_number, 'Payment', -p.amount FROM payments AS p WHERE p.client_id = 812 ) SELECT txn_date, kind, ref, amount, SUM(amount) OVER (ORDER BY txn_date, kind, ref ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_balance FROM lines ORDER BY txn_date, kind, ref; ``` **What it returns.** One row per invoice or payment. The balance rises with each invoice and falls with each payment, and this client returns to zero after most payments. **Control.** The last running balance equals the sum of all amounts. **Note.** The `kind` and `ref` tie-breakers keep the order repeatable when two rows share a date. ## 9. Top customers and concentration **Question.** How much revenue comes from the largest customers? `SUM(...) OVER ()` with no partition is the grand total, so each share is measured against all sales. ```sql WITH cust AS ( SELECT o.customerNumber, SUM(d.quantityOrdered * d.priceEach) AS sales FROM classicmodels.orders AS o JOIN classicmodels.orderdetails AS d ON d.orderNumber = o.orderNumber GROUP BY o.customerNumber ), ranked AS ( SELECT c.customerName, x.sales, ROW_NUMBER() OVER (ORDER BY x.sales DESC) AS rnk, x.sales * 100.0 / SUM(x.sales) OVER () AS share_pct, SUM(x.sales) OVER (ORDER BY x.sales DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) * 100.0 / SUM(x.sales) OVER () AS cum_pct FROM cust AS x JOIN classicmodels.customers AS c ON c.customerNumber = x.customerNumber ) SELECT rnk, customerName, ROUND(sales, 2) AS sales, ROUND(share_pct, 1) AS share_pct, ROUND(cum_pct, 1) AS cum_pct FROM ranked WHERE rnk <= 10 ORDER BY rnk; ``` **What it returns.** Ten rows. The largest customer is a little under a tenth of sales, the top two about 15 percent together, and the top ten about 28 percent. With about 98 customers, that is moderate concentration. **Control.** The cumulative share on the last customer is 100. **MariaDB:** use `LIMIT 10`. ## 10. Reconciling two lists **Question.** Does every payment received equal what we applied to invoices? Total each list by the shared key, join, and list each key that does not agree. ```sql WITH applied AS ( SELECT payment_id, SUM(amount_applied) AS applied_amount FROM payment_applications GROUP BY payment_id ) SELECT p.payment_id, p.client_id, p.payment_date, p.amount AS received, ISNULL(a.applied_amount, 0) AS applied, p.amount - ISNULL(a.applied_amount, 0) AS unapplied FROM payments AS p LEFT JOIN applied AS a ON a.payment_id = p.payment_id WHERE ABS(p.amount - ISNULL(a.applied_amount, 0)) > 0.005 ORDER BY unapplied DESC; ``` **What it returns.** Only payments that do not reconcile. In `cpafirm` there are none, so the result is empty, and that is the answer. **Control.** Count mismatches and total both sides. In `cpafirm` the received and applied totals are equal to the cent across about 20,000 payments, and no application points to a missing payment. ## 11. Accruals by month with a numbers table **Question.** How much of each grant should we recognize each month? A straight-line schedule spreads each award evenly over its term. A recursive CTE builds the list of months, so no calendar table is needed. Runs on `nonprofit`. ```sql WITH months AS ( SELECT CAST('2021-01-01' AS date) AS month_start UNION ALL SELECT DATEADD(month, 1, month_start) FROM months WHERE month_start < '2028-12-01' ), calc AS ( SELECT g.grant_id, g.award_amount, DATEFROMPARTS(YEAR(g.start_date), MONTH(g.start_date), 1) AS first_m, DATEFROMPARTS(YEAR(g.end_date), MONTH(g.end_date), 1) AS last_m FROM grants AS g ), spread AS ( SELECT m.month_start, c.grant_id, c.award_amount / (DATEDIFF(month, c.first_m, c.last_m) + 1) AS monthly_amount FROM months AS m JOIN calc AS c ON m.month_start BETWEEN c.first_m AND c.last_m ) SELECT YEAR(month_start) AS yr, MONTH(month_start) AS mo, COUNT(*) AS grant_months, ROUND(SUM(monthly_amount), 2) AS recognized FROM spread WHERE YEAR(month_start) = 2025 GROUP BY YEAR(month_start), MONTH(month_start) ORDER BY yr, mo OPTION (MAXRECURSION 200); ``` **What it returns.** Twelve rows for 2025. Recognition is highest early in the year and falls through the middle, and the grant-month count shows how many awards were active. **Control.** Run the same spread across all months and compare its total with `SUM(award_amount)` over `grants`, with no year filter. In `nonprofit` the two match across 94 grants. The month range must cover every grant: an earlier version stopped in 2027, and the totals came up short because some grants run into 2028. **MariaDB:** `WITH RECURSIVE`, `DATE_ADD(d, INTERVAL 1 MONTH)`, `TIMESTAMPDIFF(MONTH, a, b)`, `DATE_SUB(d, INTERVAL DAYOFMONTH(d) - 1 DAY)` in place of `DATEFROMPARTS`, and no `OPTION` clause. ## 12. Fund balances **Question.** What is the result for each fund, and does each fund balance on its own? Every fund must balance, so its lines net to zero. The result is revenue minus expense. Both types are `credit - debit`, because expense accounts net negative. Runs on `nonprofit`. ```sql SELECT f.fund_code, f.net_asset_class, SUM(CASE WHEN a.account_type IN ('Revenue', 'Expense') THEN l.credit - l.debit ELSE 0 END) AS result, SUM(l.debit - l.credit) AS all_lines_net FROM gl_journal_lines AS l JOIN gl_accounts AS a ON a.gl_account_id = l.gl_account_id JOIN funds AS f ON f.fund_id = l.fund_id GROUP BY f.fund_code, f.net_asset_class ORDER BY f.fund_code; ``` **What it returns.** One row per fund, 102 in the sample, with `all_lines_net` zero for every one. A negative result is a year-to-date deficit. **Control.** For the General Operating fund, `vw_fund_balances` shows opening net assets of 2,030,000 and ending net assets of about 763,000. This query's result is about negative 1.27 million, and opening plus result equals ending. **MariaDB:** the query runs unchanged. ## 13. How not to double count after joins A join multiplies rows. One invoice with three lines and two applications becomes six rows, so `SUM(total_amount)` counts it six times. The naive version joins everything and sums the invoice total. The second row sums the same client's invoices with no joins: ```sql SELECT 'naive join' AS method, SUM(i.total_amount) AS billed, COUNT(*) AS rows_in_join FROM invoices AS i JOIN invoice_lines AS l ON l.invoice_id = i.invoice_id JOIN payment_applications AS pa ON pa.invoice_id = i.invoice_id WHERE i.client_id = 812 UNION ALL SELECT 'aggregate first', SUM(i.total_amount), COUNT(*) FROM invoices AS i WHERE i.client_id = 812; ``` **What it returns.** For client 812 the naive join sums to about 13,490 over 29 rows. The correct total over 25 invoices is about 8,140. The naive figure is overstated by the fan-out. The fix aggregates each side to one row per invoice, then ties the result back: ```sql WITH lines AS ( SELECT invoice_id, SUM(amount) AS line_total FROM invoice_lines GROUP BY invoice_id ), paid AS ( SELECT invoice_id, SUM(amount_applied) AS paid_total FROM payment_applications GROUP BY invoice_id ) SELECT SUM(i.total_amount) AS billed, SUM(ISNULL(l.line_total, 0)) AS lines_tie, SUM(ISNULL(p.paid_total, 0)) AS paid FROM invoices AS i LEFT JOIN lines AS l ON l.invoice_id = i.invoice_id LEFT JOIN paid AS p ON p.invoice_id = i.invoice_id WHERE i.client_id = 812; ``` **What it returns.** `billed` and `lines_tie` are both about 8,140, and `paid` is about 7,890. **Control.** `lines_tie` must equal `billed`. Across the whole `cpafirm` sample the line total equals the invoice total exactly. **Rule for joins:** write down the grain of each table, aggregate one side when both are "many" on the key, and count rows before and after each join. ## Where next [SQL deep end](/analyze/sql-deep-end) covers the window functions used here. [CPA firm practice management](/analyze/cpafirm-practice-management) and [Nonprofit donors, grants and funds](/analyze/nonprofit-fund-accounting) describe the models behind these queries. [DAX for financial reporting](/analyze/dax-financial-reporting) and [Power Query for accounting data](/analyze/power-query-accounting-data-prep) do the same work in Power BI. --- # DAX for financial reporting This guide is for accountants, controllers and finance analysts who build income statements, balance sheets and budget reports in Power BI or Power Pivot. It assumes you know the basics of measures and filter context from [DAX fundamentals](/analyze/fundamentals). It also uses [CALCULATE](/functions/dax/calculate), [SUMX](/functions/dax/sumx) and [DIVIDE](/functions/dax/divide), which are listed in the [function reference](/analyze/functions). Financial reports have rules that sales reports do not. Debits and credits have signs. Balance sheet numbers are a snapshot at a date, not a total for a period. Subtotals must add up to totals. This guide shows how to get each of those right in DAX, and where the common mistakes come from. ## About the tables in this guide Most of the tables below are illustrative. They follow a simple general ledger design that many small and mid-size companies use, so you can see the pattern. Names such as `GL`, `Account` and `'Date'` are not from a sample database. The guide says so each time it introduces a table. The `nonprofit` sample database on LearnSQL has a real general ledger with the same shape (`gl_journal_lines`, `gl_accounts`, `fiscal_periods`). The [SQL for accountants](/analyze/sql-for-accountants) guide queries it directly. To map a real ledger to this model, rename the columns and keep the logic. | Illustrative table | Purpose | Key columns | |---|---|---| | `GL` | One row per journal line | `GL[Date]`, `GL[AccountKey]`, `GL[EntityKey]`, `GL[Amount]`, `GL[Debit]`, `GL[Credit]` | | `Account` | Chart of accounts | `Account[AccountKey]`, `Account[AccountNumber]`, `Account[AccountName]`, `Account[AccountType]`, `Account[Parent]`, `Account[Level1]` to `Account[Level4]` | | `'Date'` | Calendar with fiscal columns | `'Date'[Date]`, `'Date'[FiscalYear]`, `'Date'[FiscalPeriod]`, `'Date'[FiscalQuarter]` | | `Entity` | Legal entities or cost centers | `Entity[EntityKey]`, `Entity[EntityName]`, `Entity[Currency]` | | `Budget` | Budget by account and month | `Budget[AccountKey]`, `Budget[Month]`, `Budget[Amount]` | Use `Contoso` names only where the model is Contoso (`Sales`, `'Date'`, `Product`). The examples here are generic ledger names. ## The chart of accounts as a one-sided hierarchy A chart of accounts is a tree. Assets, liabilities, equity, revenue and expenses are the top level. Each account rolls up into a group, and each group rolls up into a statement line. The rule that makes reports foot is that **every account has exactly one parent**. An account that rolls up to two groups is counted twice. A one-sided hierarchy has one parent per row and no loops. Put the tree in `Account` as columns: | AccountNumber | AccountName | AccountType | Level1 | Level2 | Level3 | |---|---|---|---|---|---| | 4000 | Product sales | Revenue | Revenue | Product revenue | Product sales | | 4100 | Service sales | Revenue | Revenue | Service revenue | Service sales | | 5000 | Cost of goods | Expense | Cost of sales | Direct cost | Cost of goods | | 6100 | Salaries | Expense | Operating expenses | People | Salaries | Each level is a text column. You can use the columns as a hierarchy in a matrix (drag `Level1` then `Level2`), and DAX does not need a parent-child structure for this. The parent-child form is covered below, where it matters. ## Sign convention: keep one amount column Accountants think in debits and credits. Power BI sums numbers, so the model has to turn debits and credits into one signed number. Use a single column, `GL[Amount]`, with this rule: - `GL[Amount]` = debit minus credit, for every line, in the ledger's own currency. - Assets and expenses have a normal debit balance, so their total is positive when they are on the books. - Liabilities, equity and revenue have a normal credit balance, so their total is negative. That rule keeps the model honest. Every journal entry sums to zero, and so does the trial balance. You can check that with one measure: ```dax Ledger Check = ROUND(SUM(GL[Amount]), 2) ``` The result should be zero on the grand total. If it is not, a journal is out of balance or a line is missing from the load. Put this card on the first page of every financial report. Many reports need the other sign. Revenue is usually shown as a positive number. Flip the sign in the measure, not in the data, so the ledger stays as it was loaded: ```dax Revenue = -CALCULATE(SUM(GL[Amount]), Account[AccountType] = "Revenue") ``` When you flip signs, say so in the report title or footnote. A reader who sees a positive revenue and a negative expense in the same table will assume a sign error. ## A date table with fiscal periods Balance sheet and income statement measures depend on the calendar. Build a date table that includes the fiscal year, quarter and period. The fiscal year in the example starts in July. Name it by the year it ends, as many firms do: FY2026 runs July 2025 to June 2026. ```dax Date = ADDCOLUMNS( CALENDAR(DATE(2021, 7, 1), DATE(2027, 6, 30)), "FiscalYear", "FY" & (YEAR([Date]) + IF(MONTH([Date]) >= 7, 1, 0)), "FiscalPeriod", MOD(MONTH([Date]) - 7, 12) + 1, "FiscalQuarter", "Q" & (INT(MOD(MONTH([Date]) - 7, 12) / 3) + 1) ) ``` Mark the table as the date table in Power BI Desktop, and relate `'Date'[Date]` to `GL[Date]`. The fiscal period numbers start at 1 for July, so `FiscalPeriod` 12 is June. Use `FiscalPeriod` for sorting month names in visuals, or the names will sort alphabetically. > **Watch out.** The date table is the most important table in a finance model. If it has gaps or the wrong fiscal year, every period measure will be wrong, and the errors will look plausible. ## Income statement measures Build the statement from a small set of account-type measures. Each one is a filter on `Account[AccountType]` or on a statement group column. Put the business rules in one place. ```dax Revenue = -CALCULATE(SUM(GL[Amount]), Account[AccountType] = "Revenue") Cost of Goods = CALCULATE(SUM(GL[Amount]), Account[Level1] = "Cost of sales") Gross Margin = [Revenue] - [Cost of Goods] Gross Margin % = DIVIDE([Gross Margin], [Revenue]) Operating Expenses = CALCULATE(SUM(GL[Amount]), Account[Level1] = "Operating expenses") EBITDA = [Gross Margin] - [Operating Expenses] ``` > **Accounting note.** Read the signs carefully. `Cost of Goods` is a debit balance, so it is positive without a flip. `Revenue` is a credit balance, so it is flipped to a positive number. Gross margin is revenue minus cost, which is correct because both are positive now. `Gross Margin %` uses `DIVIDE`, not `/`. When there is no revenue in a period, `DIVIDE` returns blank instead of an error, and the visual shows nothing rather than a broken number. ### Why the EBITDA measure filters by group EBITDA is not a single account. It is revenue minus cost of sales minus operating expenses, before depreciation, amortization, interest and tax. The clean way is to tag each account with a statement group and sum the groups, because then the statement and the measure use the same rule. Add a column to `Account`: `StatementGroup`, with values such as `Revenue`, `Cost of sales`, `Operating expenses`, `Depreciation`, `Interest`, `Tax`. Then: ```dax EBITDA = -CALCULATE( SUM(GL[Amount]), Account[StatementGroup] IN { "Revenue", "Cost of sales", "Operating expenses" } ) ``` Here the sign is flipped once, for the whole set. In the ledger, revenue lines are negative and cost lines are positive. Their sum is therefore the negative of profit before depreciation, and the leading minus sign turns it back into a positive profit. This only works when every group in the list has the normal sign for its type. If a group with a reversed sign is added to the list, the result still looks reasonable and is wrong. Test EBITDA against a manual sum for one month before you trust it. A safer pattern is to build EBITDA from the measures you already check: `[Revenue] - [Cost of Goods] - [Operating Expenses]`. It is longer, but each step can be reconciled to the statement. ### Showing expenses as negatives Some reports show costs in parentheses, as a negative. Flip costs for display only: ```dax Cost of Goods (display) = -[Cost of Goods] ``` Do not use the display measure in other calculations. Keep the positive measure as the base, so `Gross Margin` is not accidentally computed on negative costs. ## Parent-child hierarchies and why we flatten them Some ledgers store the chart of accounts as a parent-child table, where each account has a `ParentAccountKey` that points to another account. The DAX functions `PATH`, `PATHITEM` and `PATHCONTAINS` read that structure directly. Here is how it works. ```dax Account Path = PATH(Account[AccountKey], Account[ParentKey]) ``` `PATH` returns a string such as `1000|1100|1110` for a cash account under a bank group under current assets. `PATHITEM(path, 2, INTEGER)` returns the second item. We recommend flattening the hierarchy into `Level1` to `Level4` columns, as in the table above, for four reasons: 1. **Reports need fixed levels.** Matrix visuals and the rest of the report expect named levels. A variable-depth path needs extra work for every level. 2. **Flattening is done once.** The path logic runs at refresh time in Power Query or in the source. The report then uses plain columns, which are faster and easier to test. 3. **Unbalanced trees show up at load time.** A flattened table with an empty level, or an account with two parents, is easy to spot in a table visual. A path measure hides the problem. 4. **Auditors can read it.** A flat list of accounts with their group names is easy to check against the chart of accounts that the controller signed. Use `PATHCONTAINS` only when you need to test whether an account sits under a given group and the depth varies. In most finance models that is not needed. ## Period comparisons Comparisons are where most financial reports break. The four common ones are month, year to date, prior year and budget variance. ### Month and year to date ```dax Revenue MTD = TOTALMTD([Revenue], 'Date'[Date]) Revenue YTD = CALCULATE( [Revenue], DATESYTD('Date'[Date], "6/30") ) ``` > **Watch out.** `DATESYTD` with `"6/30"` ends the year on June 30, which matches a July-to-June fiscal year. Without the year-end argument, the function assumes a calendar year, and every fiscal year-to-date number is wrong by six months. `TOTALMTD` and `DATESMTD` work the same way for months. Use `CALCULATE` with `DATESYTD` when you need to combine the period with other filters. ### Prior year and growth ```dax Revenue PY = CALCULATE([Revenue], SAMEPERIODLASTYEAR('Date'[Date])) Revenue Growth % = DIVIDE([Revenue] - [Revenue PY], [Revenue PY]) ``` `SAMEPERIODLASTYEAR` moves the date filter back one year. For a partial year, compare YTD to prior year YTD, not the full prior year to this year to date. The [DAX deep end](/analyze/dax-deep-end) guide covers the traps. ### Budget variance with favorable and unfavorable flags Budget and actual signs must match before you subtract. Budgets are often entered as positive numbers for both revenue and expense. If the budget table uses a different sign from the ledger, the variance will be wrong on the expense lines. ```dax Budget Amount = SUM(Budget[Amount]) Variance = [Revenue] - [Budget Amount] Variance Flag = VAR Diff = [Variance] RETURN SWITCH( TRUE(), ISBLANK(Diff), BLANK(), Diff >= 0, "Favorable", "Unfavorable" ) ``` For expenses, a positive variance is unfavorable, because the actual cost was higher than the budget. Write a separate expense flag so the sign logic is explicit: ```dax Budget Expense = SUM(Budget[Amount]) Expense Variance = [Operating Expenses] - [Budget Expense] Expense Variance Flag = VAR Diff = [Expense Variance] RETURN SWITCH( TRUE(), ISBLANK(Diff), BLANK(), Diff <= 0, "Favorable", "Unfavorable" ) ``` Keep revenue and expense flags as separate measures. A single flag that tries to handle both usually gets one of them backward. ## Balance sheet measures: closing balances are semi-additive Revenue and expense add up across months. A balance does not. The cash balance on March 31 is the balance on March 31, not the sum of balances for January, February and March. Balance sheet accounts are **semi-additive**: they add across accounts and not across time. The standard fix is a closing balance measure that takes the last date in the filter context: ```dax Balance = CALCULATE( SUM(GL[Amount]), FILTER( ALL('Date'[Date]), 'Date'[Date] <= MAX('Date'[Date]) ) ) ``` This sums all journal lines up to the last date visible. It is correct for cumulative balances, because a balance sheet account is the running total of everything posted to it. For a period end balance in a month column, use the built-in closing functions instead, which read the last date in the period: ```dax Closing Balance = CALCULATE( SUM(GL[Amount]), CLOSINGBALANCEMONTH(SUM(GL[Amount]), 'Date'[Date]) ) ``` If you use `CLOSINGBALANCEMONTH` or `CLOSINGBALANCEYEAR`, mark the date table and use the calendar columns consistently. A wrong fiscal year-end argument produces a balance at the wrong date. Two traps are common in balance sheet models: - **Summing balances across months.** A matrix with months across the top shows the cumulative balance in each cell if you use `SUM` on the amount. It is correct only if the measure is the closing balance. - **Totals that add monthly closing balances.** The grand total of a closing balance measure must show the last closing balance, not the sum of the months. Check the total column against a manual lookup. Use `ISBLANK` on the balance sheet to tell "no activity" from "zero balance". A new account with no postings should not show zero on a report that also has zero-balance accounts. ## Common ratios Ratios are measures, not columns. Build each one from measures that are already tested. ```dax Current Assets = CALCULATE([Balance], Account[StatementGroup] = "Current assets") Current Liabilities = -CALCULATE([Balance], Account[StatementGroup] = "Current liabilities") Current Ratio = DIVIDE([Current Assets], [Current Liabilities]) ``` The liabilities are flipped with a minus sign because they are credit-natural. Without the flip, the current ratio would be negative and most readers would not notice until a board meeting. Days sales outstanding (DSO) uses the receivables balance and the revenue of the period: ```dax Receivables = CALCULATE([Balance], Account[AccountName] = "Accounts receivable") Days Sales Outstanding = DIVIDE([Receivables], [Revenue]) * DAY(EOMONTH(MAX('Date'[Date]), 0)) ``` For a monthly view, DSO uses the number of days in the month. For a rolling 12-month view, replace the day count with 365. State the method in the report notes. Two companies with the same DSO can have different collection performance if they use different methods. Gross margin percent is the same pattern: `DIVIDE([Gross Margin], [Revenue])`, formatted as a percent. ## Consolidating entities and currencies at a glance A consolidated report adds several legal entities together. The usual trap is that intercompany transactions are counted twice. The model needs an elimination flag or a separate intercompany account. ```dax Consolidated Revenue = CALCULATE( [Revenue], Account[IsIntercompany] = FALSE() ) ``` Multi-currency models need one more rule: convert each line at a single rate, then sum. Store the rate in a table, `FxRate[Currency]`, `FxRate[Month]`, `FxRate[Rate]`, and convert in a measure: ```dax Revenue (Reporting Currency) = SUMX( GL, GL[Amount] * RELATED(FxRate[Rate]) ) ``` This only works if `GL` is related to a rate table that is already filtered to one rate per currency and month. If it is not, `SUMX` multiplies each line by a rate from a different month and the total is wrong. Test the conversion against a single entity in one currency first. For balance sheet accounts, the convention is to translate the closing balance at the period-end rate and income statement items at the average rate. Using one rate for both produces a translation difference that does not reconcile. Name the two rates separately: ```dax Balance (Closing Rate) = [Balance] * [Closing Rate] Revenue (Average Rate) = [Revenue] * [Average Rate] ``` ## Parent totals, subtotals and the footing check A report foots when every subtotal equals the sum of its children and every total equals the sum of its subtotals. A measure that does not foot is not always a bug in the measure. Sometimes the total row is computed differently, for example as a ratio or a closing balance. Use `ISINSCOPE` to find out which row of the matrix the measure is evaluated on. It returns TRUE on a row that is grouped by that column: ```dax Statement Row Type = IF(ISINSCOPE(Account[Level3]), "Account", IF(ISINSCOPE(Account[Level2]), "Subtotal", "Total")) ``` Build a footing check on the statement. For each level-2 row, it compares the revenue of the level-2 group with the sum of its level-3 children, which are computed one at a time: ```dax Footing Difference = VAR Parent = [Revenue] VAR Children = SUMX(VALUES(Account[Level3]), [Revenue]) RETURN ROUND(Parent - Children, 2) ``` The result should be zero on every row of the statement. A non-zero value means that the parent row uses a different filter than its children, or an account is missing a level. The usual fix is to make the parent total sum the children, instead of recalculating from the ledger. ## Pitfalls that show up in finance reports | Symptom | Likely cause | Check | |---|---|---| | Ledger check is not zero | A journal line is missing, or one side of a journal did not load | Count lines per journal in Power Query | | Revenue shows as negative | Sign flip is missing or applied twice | Compare to the trial balance in the ledger | | Gross margin percent is above 100% or negative on a small month | Revenue is near zero and the cost is not | Use `DIVIDE` with a minimum revenue test, or show the ratio as blank | | Total does not match the sum of months | A balance measure is summed across time | Use a closing balance measure for balance sheet lines | | Budget variance is the reverse of expectations | Budget sign convention differs from the ledger | Check one account by hand | | Blank in a row where you expected zero | `SUM` over no rows returns blank | Use `COALESCE([Measure], 0)` only on the display, not in ratios | | YTD number changes when you slice by a product | A filter on the account table removes the period | Filter the date table, not the account table, for period measures | ### Blanks versus zero A blank means "no rows matched". Zero means "rows matched and the total is zero". A balance sheet account with no postings in a month is blank. An account that was posted to and netted to zero is zero. Readers treat them differently. Use `COALESCE` for display, but keep the blank for ratios so `DIVIDE` can return blank for a division by no data. ### Totals that do not foot Most footing problems come from measures that use `ALL` or `REMOVEFILTERS` in one place and a plain filter in another. Check each total against its children after every change. A footing check measure, like the one above, catches the problem at the report layer. ### Hierarchies that do not balance A chart of accounts where an account has two parents, or where a level is empty for some accounts, produces totals that count some amounts twice or not at all. The fix belongs in the source chart of accounts. Add a check that every account has exactly one value in each level before you load the model. ## Testing your measures Before a financial measure goes into a board pack, test it against three figures: 1. **The trial balance** for one entity and one month. Totals by account type must match. 2. **The statement** from the accounting system for the same period. 3. **A hand calculation** in SQL. The [SQL for accountants](/analyze/sql-for-accountants) guide shows the trial balance and fund queries you can run against a ledger. Use `DAX Studio` or the query view in Power BI Desktop to see the measure result for one filter. Change one filter at a time, and keep the test in a table visual so you can see every row. ## Where next - [DAX fundamentals](/analyze/fundamentals): measures, context and the sign rules that this guide builds on. - [DAX deep end](/analyze/dax-deep-end): how CALCULATE and the storage engine decide what a measure returns. - [DAX pattern cookbook](/analyze/patterns): variance, prior year and balance patterns you can adapt. - [Power Query for accounting data](/analyze/power-query-accounting-data-prep): clean the ledger export before it reaches the model. - [SQL for accountants](/analyze/sql-for-accountants): query the trial balance and reconcile to the ledger. - [Function reference](/analyze/functions): look up any function used here. --- # Power Query for accounting data Accounting data arrives in bad shape. A bank export has a title block above the header. A trial balance from Excel has merged cells and subtotal rows. Monthly files have slightly different column names. Vendor names are spelled three ways. This guide works through the real problems that finance teams meet when they load data into Power Query, with a recipe for each one. Each recipe has the same parts: the problem, the M code, and what to check before you trust the result. The code is written for Power Query in Excel and Power BI. If you are new to M, read [M basics](/analyze/power-query-m-basics) first. The [M tables](/analyze/power-query-m-tables) guide covers the table functions used here. Some recipes read files you have on your own computer. Those files are not included on LearnSQL, so the file paths are parameters you set. Other recipes read the LearnSQL sample databases. For those, the server is `learnsql.cpetoday.com,1433`. The database name and schema differ by sample: | Sample database | Database name in `Sql.Database` | Schema in `Source{[Schema = ..., Item = ...]}` | |---|---|---| | QuickBooks Desktop style accounting | `qbd` | `QuickBooks` | | QuickBooks Online style accounting | `qbo` | `QuickBooksOnline` | | CPA firm practice management | `cpafirm` | `dbo` | | Nonprofit fund accounting | `nonprofit` | `dbo` | The schema name matters. `QuickBooks` is not `dbo`. If a `Source` lookup fails with "key not found", check the schema column in the Navigator. ## Set up parameters first Put the file locations and the reporting period in parameters, so you change them in one place. A parameter is a query that holds one value. Create each one in **Home > Manage Parameters**, or paste this into a blank query and set it as a parameter (the `meta` record marks it as one). ```powerquery "C:\Finance\Exports\2026" meta [IsParameterQuery = true, Type = "Text", IsParameterQueryRequired = true] ``` Create a second parameter for the first day of the period, as a date: ```powerquery #date(2026, 7, 1) meta [IsParameterQuery = true, Type = "Date", IsParameterQueryRequired = true] ``` Name the first parameter `FolderPath` and the second `StartPeriod`. Use them in any query like this: ```powerquery Table.SelectRows(Lines, each [Date] >= StartPeriod and [Date] < Date.AddMonths(StartPeriod, 12)) ``` Parameters keep a fiscal year close, the same for every export, from one copy of the query to the next. ## Recipe 1: a bank or credit card export **The problem.** The bank exports a CSV with four lines of account information above the header. Dates are text in `M/d/yyyy` form. Money is in two columns: `Debit` for money out and `Credit` for money in. Some rows have a blank in one of them. Some negative amounts come as `(1,234.50)`. **The M.** This query skips the title block, types the date, and makes one signed `Amount` column with money in positive. ```powerquery let Raw = Csv.Document(File.Contents(BankFile), [Delimiter = ",", Encoding = 65001]), NoTitle = Table.Skip(Raw, 4), Headers = Table.PromoteHeaders(NoTitle, [PromoteAllScalars = true]), WithDate = Table.AddColumn(Headers, "Date", each Date.FromText([Posted Date], [Format = "M/d/yyyy", Culture = "en-US"]), type date), ToAmount = (debit as nullable text, credit as nullable text) as number => (if credit = null or Text.Trim(credit) = "" then 0 else Number.FromText(Text.Remove(credit, {",", "$"}), "en-US")) - (if debit = null or Text.Trim(debit) = "" then 0 else Number.FromText(Text.Remove(debit, {",", "$"}), "en-US")), WithAmount = Table.AddColumn(WithDate, "Amount", each ToAmount([Debit], [Credit]), type number), Result = Table.SelectColumns(WithAmount, {"Date", "Description", "Amount"}) in Result ``` `BankFile` is a parameter that holds the path to the CSV. Set it to the full file path, not the folder. > **Audit tip.** Sum `Amount` and compare it with the closing balance minus opening balance on the statement. The totals must agree to the cent. **What to check.** Look at the first and last `Date` values to confirm the period. If the dates come through as `null`, the date format in the file is not the one in the `Format` option. ## Recipe 2: a trial balance from Excel **The problem.** The trial balance is an Excel sheet. The account group name is in a merged cell, so only the first row of each group has a value. Subtotal rows say "Total Cash" and similar. A blank row sits between sections. **The M.** Read the sheet, promote the header row, fill the account group down into the blank cells, and drop the subtotal and blank rows. ```powerquery let Source = Excel.Workbook(File.Contents(TrialBalanceFile), null, true), Sheet = Source{[Item = "Trial Balance", Kind = "Sheet"]}[Data], Promoted = Table.PromoteHeaders(Sheet, [PromoteAllScalars = true]), Filled = Table.FillDown(Promoted, {"Group"}), NoBlanks = Table.SelectRows(Filled, each [Account] <> null and [Account] <> ""), NoSubtotals = Table.SelectRows(NoBlanks, each not Text.StartsWith([Account], "Total")), Typed = Table.TransformColumnTypes(NoSubtotals, {{"Debit", type number}, {"Credit", type number}}), Net = Table.AddColumn(Typed, "Net", each [Debit] - [Credit], type number) in Net ``` `TrialBalanceFile` is a parameter that holds the workbook path. The sheet name `Trial Balance` must match the tab name exactly, including spaces. **What to check.** The sum of `Net` must be zero. A trial balance that does not net to zero means a row was dropped or a subtotal was kept. Count the rows before and after the filter steps and write the two numbers down, so you can explain the difference to a reviewer. ## Recipe 3: twelve monthly files in one folder **The problem.** Each month the bookkeeper saves a CSV named like `2026-07.csv`. You want all twelve in one table, with a column that says which file each row came from. The column headers should match in every file, and if one file has a different header, you want to find out. **The M.** List the files, read each one as a table with its own headers, tag each row with the file name, and combine them. The file list is filtered to CSV files only. ```powerquery let Files = Folder.Files(FolderPath), CsvOnly = Table.SelectRows(Files, each Text.EndsWith([Name], ".csv")), Tables = List.Transform(Table.ToRecords(CsvOnly), (f) => Table.AddColumn( Table.PromoteHeaders(Csv.Document(f[Content], [Delimiter = ",", Encoding = 65001]), [PromoteAllScalars = true]), "SourceFile", each f[Name], type text ) ), Combined = Table.Combine(Tables) in Combined ``` **What to check.** Run `Table.ColumnNames` on each file once by hand, or add a step that counts the distinct column lists. > **Watch out.** If a month has an extra or renamed column, `Table.Combine` produces a wider table with `null` cells for the missing columns, and nothing fails. Check that the row count per `SourceFile` looks reasonable, and that no month is missing. ## Recipe 4: mapping accounts with a fallback **The problem.** The accounting system has its own account types. The report needs a statement line for each account, such as Cash, Receivables or Revenue. Some accounts are new and not in the mapping table yet. Those must not disappear from the report, and they must be counted. **The M.** This uses the chart of accounts in the `qbd` sample. The mapping table is a small table written in the query. In practice it is an Excel sheet that the controller maintains. ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "qbd"), Accounts = Source{[Schema = "QuickBooks", Item = "Accounts"]}[Data], Chart = Table.SelectColumns(Accounts, {"FullName", "Type", "Number"}), Mapping = #table(type table [AccountType = text, StatementLine = text], { {"BANK", "Cash"}, {"ACCOUNTSRECEIVABLE", "Receivables"}, {"INCOME", "Revenue"}, {"COSTOFGOODSSOLD", "Cost of sales"}, {"EXPENSE", "Operating expenses"} }), Joined = Table.NestedJoin(Chart, {"Type"}, Mapping, {"AccountType"}, "Map", JoinKind.LeftOuter), Expanded = Table.ExpandTableColumn(Joined, "Map", {"StatementLine"}, {"StatementLine"}), WithFallback = Table.TransformColumns(Expanded, {{"StatementLine", each _ ?? "Unmapped", type text}}) in WithFallback ``` `JoinKind.LeftOuter` keeps every account, including the ones with no match. The `??` operator replaces a `null` with "Unmapped" when the account type is not in the table. **What to check.** Count the rows with `StatementLine = "Unmapped"`. Put that count on the report with the other data quality figures (recipe 11). Expect `Type` values such as `FIXEDASSET` and `OTHERCURRENTLIABILITY` to show up as unmapped until someone adds them to the mapping. Never let an unmapped account silently join a total. ## Recipe 5: unpivoting a monthly budget grid **The problem.** The budget is a grid. Accounts run down the rows, months run across the columns. Reports and the DAX model need one row per account per month. **The M.** The grid is written in the query so you can test it. Use `Table.UnpivotOtherColumns` to turn every column except the account into rows. ```powerquery let Grid = #table(type table [Account = text, Jan = number, Feb = number, Mar = number], { {"4000 Product sales", 120000, 125000, 130000}, {"6100 Salaries", -80000, -80000, -82000} }), Long = Table.UnpivotOtherColumns(Grid, {"Account"}, "Month", "Budget"), Typed = Table.TransformColumnTypes(Long, {{"Budget", type number}}) in Typed ``` `Table.UnpivotOtherColumns(table, {"Account"}, "Month", "Budget")` keeps `Account` as it is and creates two columns: `Month`, which holds the old column names, and `Budget`, which holds the values. **What to check.** The row count must equal the number of accounts times the number of months. Sum `Budget` by account and compare with the annual budget. The sign convention must match the ledger. In this example revenue is positive and cost is negative in the budget, so flip costs in the model if your ledger stores them as positive debits. See [DAX for financial reporting](/analyze/dax-financial-reporting) for how the signs are handled. ## Recipe 6: vendor names that are almost the same **The problem.** The same vendor appears as "C.U. Electric", "CU Electric" and "C. U. Electric" in different bills. Spend reports and 1099 totals are wrong until the names are matched. **The M.** Build a match key that ignores case, punctuation and spaces. Group by the key, and keep any key that has more than one name. This is the same technique for any name list. ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "qbd"), Vendors = Source{[Schema = "QuickBooks", Item = "Vendors"]}[Data], Keyed = Table.AddColumn(Vendors, "MatchKey", each Text.Remove(Text.Lower([Name]), {".", ",", "'", " ", "&", "-"}), type text), Groups = Table.Group(Keyed, {"MatchKey"}, { {"Count", each Table.RowCount(_), Int64.Type}, {"Names", each Text.Combine([Name], " | "), type text} }), Candidates = Table.SelectRows(Groups, each [Count] > 1) in Candidates ``` **What to check.** Read the candidate list. A match key is a search aid, not a decision. "Bayshore Water" and "Bayshore Water Co" may be different vendors with the same key only if the rule is too loose. Choose the vendor master record yourself, then map the other names to it. Do not merge two vendors with different tax IDs, even if the names match. The `qbd` vendors include a `Company` column as well as `Name`; repeat the key on `Company` to find matches that show up in only one of them. ## Recipe 7: splitting "1000 - Cash" **The problem.** The chart of accounts exports the account number and name in one text column, such as `1000 - Cash`. You need the number as a number, and the name on its own. **The M.** Use `Text.BeforeDelimiter` and `Text.AfterDelimiter`. They split at the first occurrence of the delimiter. ```powerquery let Accounts = #table(type table [Account = text], { {"1000 - Cash"}, {"1010 - Trust account cash"}, {"4100 - Tax preparation revenue"} }), Number = Table.AddColumn(Accounts, "Number", each Text.BeforeDelimiter([Account], " - "), type text), Name = Table.AddColumn(Number, "Name", each Text.AfterDelimiter([Account], " - "), type text), Typed = Table.TransformColumnTypes(Name, {{"Number", Int64.Type}}), Clean = Table.RemoveColumns(Typed, {"Account"}) in Clean ``` `Table.SplitColumn` with `Splitter.SplitTextByDelimiter(" - ", QuoteStyle.None)` also works. The `Text.` functions are easier to read when the delimiter may appear more than once in a name. **What to check.** Every `Number` must convert to an integer. If one row fails, the error will show on the column, and you can find the row with recipe 11. Account numbers with letters, such as `1000A`, need a different rule. ## Recipe 8: text numbers, parentheses and trailing minus **The problem.** Amounts arrive as text. Negative values come in three forms: `(1,234.50)`, `1,234.50-`, and `-1234.50`. Some have dollar signs and spaces. Excel will not sum these as numbers. **The M.** Write one function that handles all three forms, and call it on the column. The function is stored as a step, so you can read it. ```powerquery let ToNumber = (raw as nullable text) as nullable number => let trimmed = if raw = null then null else Text.Trim(raw), cleaned = if trimmed = null or trimmed = "" then null else Text.Remove(trimmed, {"$", ",", " "}), isNegative = cleaned <> null and ( (Text.StartsWith(cleaned, "(") and Text.EndsWith(cleaned, ")")) or Text.EndsWith(cleaned, "-") or Text.StartsWith(cleaned, "-") ), digits = if cleaned = null then null else Text.Remove(cleaned, {"(", ")", "-"}), value = if digits = null then null else Number.FromText(digits, "en-US"), signed = if value = null then null else if isNegative then -value else value in signed, Sample = #table(type table [Amount = text], { {"(1,234.50)"}, {"1,234.50-"}, {"-1234.50"}, {"$ 88.00"}, {""} }), Converted = Table.AddColumn(Sample, "Value", each ToNumber([Amount]), type nullable number) in Converted ``` Empty strings become `null`, not zero. That is deliberate. Turn them into zero only when you know a blank means zero in that file. **What to check.** The `Value` column for the sample rows should read -1234.5, -1234.5, -1234.5, 88 and blank. Check the sum of a real column against the total printed in the file. A rounding difference of a cent is usually a number that was rounded in the export, not a conversion error. ## Recipe 9: a clean general ledger table **The problem.** A general ledger export has one row per journal line. The entry date and source document live on the journal entry, and the account name lives on the account. The report needs one table with all of those columns, one row per line, and a check that each entry balances. **The M.** Read the three tables from the `cpafirm` sample, join them, and calculate a signed amount. The lines table has `debit` and `credit` columns. ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "cpafirm"), Lines = Source{[Schema = "dbo", Item = "gl_journal_lines"]}[Data], Entries = Source{[Schema = "dbo", Item = "gl_journal_entries"]}[Data], Accounts = Source{[Schema = "dbo", Item = "gl_accounts"]}[Data], LineCols = Table.SelectColumns(Lines, {"journal_entry_id", "gl_account_id", "debit", "credit"}), EntryCols = Table.SelectColumns(Entries, {"journal_entry_id", "entry_date", "source_type", "memo"}), AccountCols = Table.SelectColumns(Accounts, {"gl_account_id", "account_number", "account_name", "account_type"}), WithEntry = Table.NestedJoin(LineCols, {"journal_entry_id"}, EntryCols, {"journal_entry_id"}, "Entry", JoinKind.Inner), EntryExpanded = Table.ExpandTableColumn(WithEntry, "Entry", {"entry_date", "source_type", "memo"}), WithAccount = Table.NestedJoin(EntryExpanded, {"gl_account_id"}, AccountCols, {"gl_account_id"}, "Account", JoinKind.Inner), AccountExpanded = Table.ExpandTableColumn(WithAccount, "Account", {"account_number", "account_name", "account_type"}), Typed = Table.TransformColumnTypes(AccountExpanded, { {"entry_date", type date}, {"debit", type number}, {"credit", type number} }), WithAmount = Table.AddColumn(Typed, "Amount", each [debit] - [credit], type number), Result = Table.RenameColumns(WithAmount, {{"entry_date", "Date"}, {"account_number", "Account"}}) in Result ``` Notice that the join keys are the same name in each pair of tables. `JoinKind.Inner` drops lines with no matching entry or account. That is what you want for a report, but it hides orphan lines. Recipe 10 shows how to find them. **What to check.** Compare the row count with the count in the source table. The sum of `Amount` must be zero for the whole table, and zero for each `journal_entry_id`. The second check is the one that matters for audit: an entry that does not balance is a posting problem in the source system. ## Recipe 10: find unmatched rows with an anti join **The problem.** Every payment should be applied to at least one invoice, and every application should point to a real payment. Missing matches are where money goes missing. You need the rows on one side that have no partner on the other. **The M.** `Table.Join` with `JoinKind.LeftAnti` returns the rows from the left table that have no match in the right table. Build the right table first, as a total per key. ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "cpafirm"), Payments = Source{[Schema = "dbo", Item = "payments"]}[Data], Applications = Source{[Schema = "dbo", Item = "payment_applications"]}[Data], Applied = Table.Group(Applications, {"payment_id"}, { {"AppliedTotal", each List.Sum([amount_applied]), type number} }), NotApplied = Table.Join(Payments, {"payment_id"}, Applied, {"payment_id"}, JoinKind.LeftAnti) in NotApplied ``` `NotApplied` lists payments that have no application at all. For payments that are applied but for the wrong amount, use a left outer join and compare the two amounts: ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "cpafirm"), Payments = Source{[Schema = "dbo", Item = "payments"]}[Data], Applications = Source{[Schema = "dbo", Item = "payment_applications"]}[Data], Applied = Table.Group(Applications, {"payment_id"}, { {"AppliedTotal", each List.Sum([amount_applied]), type number} }), Joined = Table.NestedJoin(Payments, {"payment_id"}, Applied, {"payment_id"}, "Applied", JoinKind.LeftOuter), Expanded = Table.ExpandTableColumn(Joined, "Applied", {"AppliedTotal"}, {"AppliedTotal"}), Mismatch = Table.SelectRows(Expanded, each ([AppliedTotal] ?? 0) <> [amount]) in Mismatch ``` **What to check.** An anti join that returns rows is a list to investigate, not a conclusion. In the cpafirm sample, every payment has an application that matches its amount, so the mismatch list is empty. That is a result worth recording. The same pattern finds bills with no payment, or bank lines with no ledger entry. Write the count of unmatched rows down each time you refresh. ## Recipe 11: a data quality query that counts problems **The problem.** Before the report goes out, the controller wants a short list of checks with a count for each. Any count above zero must be explained. **The M.** Each check is a record with a name and a count. `Table.FromRecords` turns the list of records into a small table. Put it in its own query, and load it to a sheet. ```powerquery let Checks = { [Check = "Lines with no amount", Problems = Table.RowCount(Table.SelectRows(Lines, each [Amount] = null))], [Check = "Accounts not mapped", Problems = Table.RowCount(Table.SelectRows(Mapped, each [StatementLine] = "Unmapped"))], [Check = "Rows with a bad date", Problems = Table.RowCount(Table.SelectRows(Lines, each [Date] = null))], [Check = "Lines outside the period", Problems = Table.RowCount(Table.SelectRows(Lines, each [Date] < StartPeriod or [Date] >= Date.AddMonths(StartPeriod, 12)))] }, Result = Table.FromRecords(Checks, type table [Check = text, Problems = Int64.Type]) in Result ``` `Lines` and `Mapped` are the earlier queries from recipes 9 and 4. Reference them by query name, and keep them loaded to the data model or as connection-only queries. Add an entry balance check in the same way. Group the lines by entry, sum the amounts, and count the entries where the total is not zero: ```powerquery [Check = "Entries that do not balance", Problems = Table.RowCount(Table.SelectRows( Table.Group(Lines, {"journal_entry_id"}, {{"Net", each List.Sum([Amount]), type number}}), each [Net] <> 0))] ``` **What to check.** Every count should be zero, or have a note next to it that explains why it is not. Refresh the query each time you refresh the report. A check that changes from zero to a number after a refresh is the most useful signal in the workbook. ## Errors and bad rows Power Query shows an error in a cell as a special value. Two functions handle them: - `try expression otherwise fallback` gives a fallback value when the expression fails. Use it when a missing value is acceptable, and say so in the notes. - `Table.RemoveRowsWithErrors(table, {"Amount"})` drops rows whose named column has an error. Use it only after recipe 11 has counted those rows, so the drop is recorded. Do not use `try` to hide a conversion you have not checked. A date that fails to parse is a file problem, and the fix belongs in the file or in the format string. ## Refresh and audit trail Keep the steps in Applied Steps in order and name them, so a reviewer can follow them. Keep the source file names in the output (recipe 3 adds a `SourceFile` column), so each figure traces back to a file. Keep the raw export unchanged in its own folder. The query is the audit trail: it is repeatable, and it says exactly what was removed and why. ## Where next - [M basics](/analyze/power-query-m-basics): queries, values, lists and records. - [M tables](/analyze/power-query-m-tables): shaping rows, columns, joins and groups in more depth. - [M functions and patterns](/analyze/power-query-m-patterns): your own functions, error handling and date tables. - [M performance](/analyze/power-query-m-performance): query folding and when to buffer. - [DAX for financial reporting](/analyze/dax-financial-reporting): build the statements on top of the cleaned data. - [SQL for accountants](/analyze/sql-for-accountants): run the same checks in SQL against the source. - Function pages: [Table.Group](/functions/m/table-group), [Table.Join](/functions/m/table-join), [Table.NestedJoin](/functions/m/table-nestedjoin) and [Table.UnpivotOtherColumns](/functions/m/table-unpivotothercolumns). --- # Power BI for finance teams: ledger to dashboard A finance pack is a one-page answer to three questions: what did we earn, what did we spend, and how does that compare with the budget. This guide builds that pack from start to finish on a real general ledger. You connect to the data, shape it in Power Query, model it as a star, write the measures, lay out the page and publish it. The sample is `nonprofit`, a regional nonprofit with fund accounting. It has a general ledger, a budget by account and fund, fiscal periods, and cost centers. Every number in this guide came from a query run against that database. The DAX is standard and listed in the places it applies. DAX cannot run in this guide's environment, so each measure says to check it in DAX query view. If you are new to Power BI, read [Power BI Desktop how-to](/analyze/power-bi-desktop) first. The deeper DAX reference is in [DAX for financial reporting](/analyze/dax-financial-reporting). Time measures are in [Time intelligence](/analyze/time-intelligence). For the server and login details, see [Connect](/connect). ## The build at a glance Five steps, in order. Check each step before you start the next one. > Diagram: Five steps in order: Get Data from SQL Server, shape in Power Query, build the star model and DAX measures, lay out the report pages, then publish with a scheduled refresh. ## Step 1: Connect to the ledger 1. In Power BI Desktop, select **Home > Get Data > SQL Server**. 2. Server: `learnsql.cpetoday.com,1433`. Database: `nonprofit`. 3. Choose **Import**. Use Import for a finance pack. DirectQuery makes every visual a live query, which is slow for a page with twelve visuals. 4. In the Navigator, tick these tables under the `dbo` schema: `gl_journal_lines`, `gl_journal_entries`, `gl_accounts`, `funds`, `cost_centers`, `fiscal_periods`, and `budgets`. 5. Select **Transform Data**, not **Load**. You shape the data first. ## Step 2: Shape the data in Power Query Shaping means four things: join the entry date onto each line, type the columns, make one signed amount, and map each account to a type. **Join the date.** A line in `gl_journal_lines` has no date. The date is on the entry. Join the two tables on `journal_entry_id`. ```powerquery let Source = Sql.Database("learnsql.cpetoday.com,1433", "nonprofit"), Lines = Source{[Schema = "dbo", Item = "gl_journal_lines"]}[Data], Entries = Source{[Schema = "dbo", Item = "gl_journal_entries"]}[Data], Joined = Table.NestedJoin(Lines, {"journal_entry_id"}, Entries, {"journal_entry_id"}, "Entry", JoinKind.Inner), Expanded = Table.ExpandTableColumn(Joined, "Entry", {"entry_date", "fiscal_year", "fiscal_period_id"}, {"entry_date", "fiscal_year", "fiscal_period_id"}), Typed = Table.TransformColumnTypes(Expanded, {{"debit", Currency.Type}, {"credit", Currency.Type}, {"entry_date", type date}}), Signed = Table.AddColumn(Typed, "Amount", each [debit] - [credit], Currency.Type) in Signed ``` **Keep one sign rule.** The ledger stores debit minus credit. Assets and expenses are positive. Revenue, liabilities and net assets are negative. That is correct for the ledger, but a board reads revenue as a positive number. Keep `Amount` as debit minus credit, then add a display column for revenue. > Diagram: One revenue line with debit 0 and credit 1,000 becomes a signed amount of minus 1,000, which is multiplied by minus one for display as 1,000 of revenue. **Map the accounts.** Join `gl_accounts` to bring in the account type. Then use that type to flip the revenue sign. The account type also goes on the Accounts dimension, which Step 3 covers. Rename `account_type` to `Account Type` there, so the measures in Step 4 can find it. ```powerquery let Accounts = Source{[Schema = "dbo", Item = "gl_accounts"]}[Data], Merged = Table.NestedJoin(Signed, {"gl_account_id"}, Accounts, {"gl_account_id"}, "Account", JoinKind.LeftOuter), Mapped = Table.ExpandTableColumn(Merged, "Account", {"account_number", "account_name", "account_type"}, {"Account Number", "Account Name", "Account Type"}), WithDisplay = Table.AddColumn(Mapped, "Display Amount", each if [Account Type] = "Revenue" then [credit] - [debit] else [debit] - [credit], Currency.Type) in WithDisplay ``` Use `Source` from the first query. In Power Query, each step can refer to an earlier step by name, so paste the two blocks into one query, or build the second from the first in the Applied Steps pane. > **Audit tip.** The total of `Amount` across all lines must be zero for every entry. Run this check in SQL before you trust the model. In the sample it checks 34,377 entries and finds none that fail to net to zero. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT COUNT(*) AS entries_checked, SUM(CASE WHEN x.net <> 0 THEN 1 ELSE 0 END) AS not_zero FROM (SELECT e.journal_entry_id, SUM(l.debit - l.credit) AS net FROM dbo.gl_journal_entries AS e JOIN dbo.gl_journal_lines AS l ON l.journal_entry_id = e.journal_entry_id GROUP BY e.journal_entry_id) AS x; ``` ## Step 3: Model it as a star A star model has one fact table in the middle and dimension tables around it. Facts hold numbers. Dimensions hold the words you filter and group by. Power BI measures run fastest and are easiest to check on this layout. Here the model has one fact for actuals, one for budget, and four dimensions. > Diagram: A star model. The GL lines fact sits in the middle and joins to four dimensions: Date, Account, Fund and Cost center. A Budget fact sits below and shares the same dimensions. | Table in the model | Source | Role | |---|---|---| | GL Lines | Lines joined to entries (Step 2) | Fact: actual amounts | | Budget | `budgets` | Fact: budget by year, account, fund and cost center. Rename `budget_amount` to `Budget Amount` | | Accounts | `gl_accounts` | Dimension: number, name, and `account_type` renamed to `Account Type`. Relate it to GL Lines on `gl_account_id` | | Funds | `funds` | Dimension: fund, restriction class | | Cost Centers | `cost_centers` | Dimension: program or support function | | Date | Built in DAX (below) | Dimension: every day, with fiscal year and period | **Use a daily date table.** The `fiscal_periods` table has one row per month. Time intelligence functions need one row per day, so build a Date table in DAX. The fiscal year starts on July 1. That matches `fiscal_periods`, where fiscal 2026 runs from 2025-07-01 to 2026-06-30. ```dax Date = ADDCOLUMNS ( CALENDAR ( DATE ( 2021, 7, 1 ), DATE ( 2027, 6, 30 ) ), "Fiscal Year", YEAR ( [Date] ) + IF ( MONTH ( [Date] ) >= 7, 1, 0 ), "Fiscal Period", IF ( MONTH ( [Date] ) >= 7, MONTH ( [Date] ) - 6, MONTH ( [Date] ) + 6 ) ) ``` Mark the table as a date table in **Table tools > Mark as date table**, and relate `Date[Date]` to the entry date on the GL Lines table. Check in DAX query view that `Fiscal Period` gives 1 for July and 12 for June. **Expect blanks on cost center.** In the sample, every expense line has a cost center. Revenue, asset and liability lines have none. A slicer on cost center will hide revenue. That is correct, not an error. Show revenue on the summary page, and use cost center only for the expense detail. **The budget has no month.** The `budgets` table holds one amount per year, account, fund and cost center. It has no month. You can compare the full year, but you cannot compare a month with the budget unless you spread the budget across months with a rule, such as one twelfth each month. The sample has no spread rule, so the monthly view in this guide shows actuals only. ## Step 4: Write the core measures The measures below use the names from Step 3. Each measure is standard DAX. Enter them in the model, then check the results in DAX query view against the SQL totals in Step 6. ```dax Revenue = CALCULATE ( -SUM ( 'GL Lines'[Amount] ), Accounts[Account Type] = "Revenue" ) Expense = CALCULATE ( SUM ( 'GL Lines'[Amount] ), Accounts[Account Type] = "Expense" ) Net Surplus = [Revenue] - [Expense] Budget Revenue = CALCULATE ( SUM ( Budget[Budget Amount] ), Accounts[Account Type] = "Revenue" ) Budget Expense = CALCULATE ( SUM ( Budget[Budget Amount] ), Accounts[Account Type] = "Expense" ) Revenue Variance = [Revenue] - [Budget Revenue] Expense Variance = [Budget Expense] - [Expense] Expense Variance % = DIVIDE ( [Expense Variance], [Budget Expense] ) ``` **Read the variance signs.** Revenue variance is positive when you earned more than budget. Expense variance is positive when you spent less than budget. The expense measure is written as budget minus actual on purpose, so a positive number is good in both rows. Label it that way on the page, so a reader does not have to work it out. **Year to date and the prior year.** Use the fiscal year end, June 30, in `DATESYTD`. For the same period last year, use `SAMEPERIODLASTYEAR`. The time intelligence guide covers how these behave with a partial year. ```dax Revenue YTD = CALCULATE ( [Revenue], DATESYTD ( 'Date'[Date], "6/30" ) ) Revenue Prior Year = CALCULATE ( [Revenue], SAMEPERIODLASTYEAR ( 'Date'[Date] ) ) ``` **Balance as of a date.** Balance sheet accounts are running totals, not period totals. A cash balance at June 30 is every cash movement up to that date. The pattern below sums every day up to the latest date in the filter. ```dax Asset Balance = CALCULATE ( SUM ( 'GL Lines'[Amount] ), Accounts[Account Type] = "Asset", FILTER ( ALL ( 'Date' ), 'Date'[Date] <= MAX ( 'Date'[Date] ) ) ) ``` Check the cash balance against the SQL result in Step 6 for June 30, 2026. ## Step 5: Lay out the finance pack A one-page pack answers the three questions at the top and gives the detail below. The layout below is the standard. Each element uses one of the measures above. > Diagram: A one-page finance pack layout. A row of four KPI cards on top, an income statement matrix with actual, budget and variance on the left, a variance bar chart on the right, and a twelve-month trend along the bottom. | Element | Measure or field | Note | |---|---|---| | KPI cards (top row) | Revenue, Expense, Net Surplus, Expense Variance | Show the current fiscal year to date | | Income statement matrix (left) | Rows: Account Type, then Account Name. Columns: Actual, Budget, Variance | Use Revenue and Expense measures | | Variance bars (right) | Revenue Variance, Expense Variance, Net | Positive means favorable | | Trend (bottom) | Revenue and Expense by month | Show actuals only, since budget has no month | The FY2026 totals in this layout tie to the sample. Revenue $7.37M, expense $7.31M, net surplus about $0.06M, expense $19K under budget. Those are the numbers Step 6 checks. **Look at the trend before you publish.** December 2025 revenue is $1.34M, more than twice the typical month. It is probably a large grant or a year-end pledge. In a real pack, a month like that needs a note on the page, or the first question at the board meeting will be what happened in December. ## Step 6: Tie the pack to the ledger A report that does not tie to the ledger is wrong, however good it looks. Run the SQL below, then compare each figure with the matching measure in DAX query view. The figures should match to the cent. **Runs on:** `nonprofit`, SQL Server. This is the budget and actual tie for fiscal 2026. ```sql SELECT a.account_type, 'Budget' AS measure, SUM(b.budget_amount) AS amount FROM dbo.budgets AS b JOIN dbo.gl_accounts AS a ON a.gl_account_id = b.gl_account_id WHERE b.fiscal_year = 2026 GROUP BY a.account_type UNION ALL SELECT a.account_type, 'Actual (debit minus credit)', SUM(l.debit - l.credit) FROM dbo.gl_journal_lines AS l JOIN dbo.gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN dbo.gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE e.fiscal_year = 2026 GROUP BY a.account_type ORDER BY 1, 2; ``` **What it returns.** Expense actual 7,314,951.09 against budget 7,334,200.00. Revenue actual 7,374,811.79 (shown as negative, because it is a credit) against budget 7,342,100.00. Your `Expense` and `Budget Expense` measures should show those values. Revenue shows the sign flipped, which is why the Revenue measure uses a minus sign. The monthly trend should add up to the same annual totals. This query returns revenue and expense by month for fiscal 2026. Its twelve rows sum to the totals above. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT p.period_number, p.start_date, SUM(CASE WHEN a.account_type = 'Revenue' THEN l.credit - l.debit ELSE 0 END) AS revenue, SUM(CASE WHEN a.account_type = 'Expense' THEN l.debit - l.credit ELSE 0 END) AS expense FROM dbo.gl_journal_lines AS l JOIN dbo.gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN dbo.gl_accounts AS a ON a.gl_account_id = l.gl_account_id JOIN dbo.fiscal_periods AS p ON p.fiscal_period_id = e.fiscal_period_id WHERE e.fiscal_year = 2026 GROUP BY p.period_number, p.start_date ORDER BY p.period_number; ``` Balance sheet figures tie the same way. This query gives the asset balances at June 30, 2026, the end of fiscal 2026. Compare it with your `Asset Balance` measure set to that date. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT a.account_number, a.account_name, SUM(l.debit - l.credit) AS balance FROM dbo.gl_journal_lines AS l JOIN dbo.gl_journal_entries AS e ON e.journal_entry_id = l.journal_entry_id JOIN dbo.gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE a.account_type = 'Asset' AND e.entry_date <= '2026-06-30' GROUP BY a.account_number, a.account_name ORDER BY a.account_number; ``` **What it returns.** Seven asset accounts. Cash operating is $272,404.68, savings and money market is $900,000.00 and investments are $1,308,741.92. Pledges receivable, grants receivable and property are the rest. ## Step 7: Publish and refresh Publish from Power BI Desktop with **Home > Publish**, and pick a workspace. Once it is in the service, set a scheduled refresh under the dataset settings. Three checks before you schedule a refresh: - **Connection.** Scheduled refresh runs in the cloud. The server has to be reachable from the Power BI service, and the credentials have to be entered in the service. Check [Connect](/connect) for the server details before you schedule anything. - **Login.** Use a read-only login. A finance pack never needs write access to the ledger. - **Refresh time.** Schedule the refresh after the close. A refresh at 6 AM on the first business day shows whatever has been posted so far, so state the as-of date on the page. ## What the sample cannot show - **Budget by month.** The sample has one budget amount per year. A monthly budget needs a spread rule. - **Cost center on revenue.** Only expenses carry a cost center. A revenue-by-program view needs a different dimension, such as the fund's program. - **Approvals and drafts.** The ledger has posted entries only. A pack that includes draft entries needs a status field. - **Currency.** All amounts are in US dollars. Once you have the pack, the next step is the detailed variance work in [DAX for financial reporting](/analyze/dax-financial-reporting), and the period logic in [Time intelligence](/analyze/time-intelligence). --- # Audit and controls analytics in SQL Auditors test whole populations where they can. A journal entry test reads every entry in the period and flags the ones that match a risk. A duplicate test reads every bill and finds the pairs. This guide gives you the tests that matter most, the SQL for each one, what the sample data showed, and what to do with the hits. Every query here was run against the LearnSQL sample databases. The results describe sample data, not any real client. The point is the method. Use the same steps on your own ledger. ## How the tests fit together A test has five parts: a population, a risk, a query, a list of hits, and a follow-up. The query only finds candidates. A person decides what the hits mean. > Diagram: Three columns. Four sample data sources feed four groups of audit tests, and every test flows into one exception list that is then sampled and followed up. Start every test with the population. Count the rows, then tie that count to a second source, such as the trial balance or a control total. A test on a partial population proves nothing about the rest. ## Which sample data supports which test Not every database has every column. Check first. This table lists what each test needs and where the sample has it. | Test | Sample database | Tables and columns used | Supported? | |---|---|---|---| | Weekend and holiday postings | `nonprofit` | `gl_journal_entries.entry_date` | Yes, by date only | | After-hours postings | `qbd` | `JournalEntries.TimeCreated` | Mechanically yes, but the timestamps are old (see below) | | Round amounts and threshold | `nonprofit` | `gl_journal_lines.debit` | Yes. There is no approval table, so the threshold is an assumption | | Posting user and unusual users | `nonprofit` | `gl_journal_entries.posted_by` | Yes | | Entries with no source document | `nonprofit` | `gl_journal_entries.source_id` | Yes | | Rarely used accounts | `nonprofit` | `gl_journal_lines`, `gl_accounts` | Yes | | Unbalanced entries | `nonprofit` | Entries and lines | Yes | | Duplicate bills | `qbd`, `nonprofit`, `bigbear` | `QuickBooks.Bills`, `bills`, `invoices` | Yes | | Vendor tax ID | `qbd` | `QuickBooks.Vendors.TaxIdentity` | Yes | | Vendor and employee shared address | `qbd`, `bigbear`, `nonprofit` | `Vendors.Line1`, `Employees.Line1` | Partly. Only `qbd` has street addresses for employees | | Benford's law | `nonprofit` | `bills.amount` | Yes | | Cutoff | `nonprofit` | Entry dates and fiscal periods | Only the period tag. The data has no service or receipt dates | | Check number gaps | `qbd` | `QuickBooks.Checks.ReferenceNumber` | Yes | | AP aging | `nonprofit` | `bills.due_date`, `status`, `amount` | Yes | The `posted_by` column exists only in `nonprofit`. The `bigbear` journal has no user column, and `qbd` journal entries have a created time but no user. > **Audit tip.** A real user-access test needs the system's audit log, which the samples do not include. **Runs on:** all queries in this guide are written and tested in SQL Server (T-SQL). The qbd database is on SQL Server only. The nonprofit and bigbear databases are also on MariaDB, but there you need MariaDB date functions, such as `DAYOFWEEK()` in place of `DATEPART(weekday, ...)` and `LIMIT` in place of `TOP`. The `DATEPART(weekday, ...)` results below assume the US English default, where Sunday is day 1 and Saturday is day 7. ## Journal entry tests Journal entries carry the most audit risk, because management can post them directly. The five tests below run on the `nonprofit` ledger. ### Entries on weekends and holidays **Question.** Did anyone post on a Saturday or Sunday, or on a fixed holiday? Weekend work is not wrong. It is a population to explain. The fixed holidays here are New Year's Day, Independence Day and Christmas. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT e.source_type, COUNT(*) AS entries, SUM(CASE WHEN DATEPART(weekday, e.entry_date) IN (1, 7) THEN 1 ELSE 0 END) AS weekend_entries, SUM(CASE WHEN (MONTH(e.entry_date) = 1 AND DAY(e.entry_date) = 1) OR (MONTH(e.entry_date) = 7 AND DAY(e.entry_date) = 4) OR (MONTH(e.entry_date) = 12 AND DAY(e.entry_date) = 25) THEN 1 ELSE 0 END) AS fixed_holiday_entries FROM dbo.gl_journal_entries AS e GROUP BY e.source_type ORDER BY entries DESC; ``` **What it returns.** About 6,600 of 34,377 entries fall on a weekend, roughly 19 percent. Most of them are bill payments and gift batches. Bill payments alone account for 3,879 weekend entries. That is probably a date rule in the system, such as the date a payment file was created. Ask. Do not assume fraud. The useful hit list is different. The `Bill` source has 130 entries dated on a fixed holiday. A bill should not carry a holiday date unless the vendor invoiced that day. Sample those first. **What to do.** Ask the owner of the source system how each date is set. If the date is the system date, the test is looking at processing, not at people. If the date is typed in by a person, read the support. ### Round-dollar amounts **Question.** Are there lots of round amounts, such as exactly $5,000.00, where the support should show cents? Round amounts can mean estimates, plugs or manual entries. Test by account type, because some round numbers are normal (for example, a fixed monthly rent). **Runs on:** `nonprofit`, SQL Server. ```sql SELECT a.account_type, COUNT(*) AS debit_lines, SUM(CASE WHEN l.debit % 1000 = 0 THEN 1 ELSE 0 END) AS round_thousands, SUM(CASE WHEN l.debit % 100 = 0 THEN 1 ELSE 0 END) AS round_hundreds FROM dbo.gl_journal_lines AS l JOIN dbo.gl_accounts AS a ON a.gl_account_id = l.gl_account_id WHERE l.debit >= 1000 GROUP BY a.account_type ORDER BY debit_lines DESC; ``` **What it returns.** Asset lines have 219 round-thousand debits out of 7,965. Expense lines have 5 out of 8,836. Round amounts are common in cash and investment movements (transfers between accounts) and rare in expenses. A round expense is worth a look. A round transfer usually is not. ### Amounts just below an approval threshold **Question.** Does the population bunch just under an approval limit? This sample has no approval table, so assume a limit of $10,000 and test the bands beneath it. If a band just under the limit is much bigger than the bands around it, people may be splitting or shaving amounts. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT FLOOR(l.debit / 500) * 500 AS band_start, COUNT(*) AS lines FROM dbo.gl_journal_lines AS l WHERE l.debit BETWEEN 8000 AND 10999.99 GROUP BY FLOOR(l.debit / 500) * 500 ORDER BY band_start; ``` **What it returns.** The six $500 bands from $8,000 to $10,999 hold 100, 116, 102, 87, 86 and 56 lines. The $9,500 band (87 lines) sits just under the limit and the $10,000 band (86) just above it. The counts fall steadily as amounts rise. There is no spike below $10,000. This population does not show threshold avoidance at this limit. **What to do.** Replace the assumed limit with your real one. A limit on purchase orders, a signing authority, or a board limit is the number to use. Run the test again at that number. ### Entries by poster and entries with no source **Question.** Who posts, and do some people post entries no one else would? Two tests help. First, look at posters with few entries, since a rarely used user posting in a sensitive account is a classic flag. Second, look for entries with no source document (a blank `source_id`). **Runs on:** `nonprofit`, SQL Server. ```sql SELECT e.posted_by, COUNT(*) AS entries, SUM(CASE WHEN DATEPART(weekday, e.entry_date) IN (1, 7) THEN 1 ELSE 0 END) AS weekend_entries, COUNT(DISTINCT e.source_type) AS source_types FROM dbo.gl_journal_entries AS e GROUP BY e.posted_by HAVING COUNT(*) <= 20 ORDER BY entries, e.posted_by; ``` **What it returns.** Fifteen posters have 20 or fewer entries. They post in one to four source types, and most of their entries fall on weekdays. Nothing about them stands out as a group. The no-source test is more useful when you know which entries should have no source. Run it like this: **Runs on:** `nonprofit`, SQL Server. ```sql SELECT e.source_type, COUNT(*) AS entries_without_source_id FROM dbo.gl_journal_entries AS e WHERE e.source_id IS NULL GROUP BY e.source_type; ``` **What it returns.** All 189 `Monthly` entries and the one `Opening` entry have no source ID. The monthly entries are the system's allocation entries, so they are expected. Filter known types out, then read what is left. In this sample nothing is left. ### Rarely used accounts **Question.** Which accounts have very few postings? A new or seldom-used account, especially one tied to cash, revenue or an estimate, deserves a look. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT a.account_number, a.account_name, a.account_type, COUNT(*) AS lines, SUM(l.debit) AS debits, SUM(l.credit) AS credits FROM dbo.gl_journal_lines AS l JOIN dbo.gl_accounts AS a ON a.gl_account_id = l.gl_account_id GROUP BY a.account_number, a.account_name, a.account_type HAVING COUNT(*) <= 50 ORDER BY lines, a.account_number; ``` **What it returns.** Four accounts have 50 lines or fewer. Two of them are opening balances (1010 with one line for $900,000 and 3000 with two lines). Pledge Write-offs (6900) has 29 lines and $62,615 of debits. Write-offs are rare by design, so the answer is to read the 29 entries and check that each one has an approval. ### Unbalanced entries **Question.** Does every entry balance, with total debits equal to total credits? This is the first control. > **Audit tip.** Run it before anything else. If an entry does not balance, the ledger is wrong, and every other test is in doubt. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT COUNT(*) AS entries_checked, SUM(CASE WHEN x.dr <> x.cr THEN 1 ELSE 0 END) AS unbalanced FROM (SELECT e.journal_entry_id, SUM(l.debit) AS dr, SUM(l.credit) AS cr FROM dbo.gl_journal_entries AS e JOIN dbo.gl_journal_lines AS l ON l.journal_entry_id = e.journal_entry_id GROUP BY e.journal_entry_id) AS x; ``` **What it returns.** 34,377 entries checked and none unbalanced. **After-hours postings.** The `qbd` sample has a created time on each journal entry, so you could test for postings between 7 PM and 7 AM. The test runs, but this sample is not usable for it. Its timestamps were created between 2003 and 2011, while the entries they describe are dated 2021 to 2023. The 18 night-time hits are opening balance entries created in 2007. Run the same test on a live system with current timestamps, and check the time zone first. **Runs on:** `qbd`, SQL Server. This query shows the created-year gap that makes the after-hours test unreliable on this sample. ```sql SELECT YEAR(j.TimeCreated) AS created_year, YEAR(j.Date) AS entry_year, COUNT(*) AS entries, SUM(CASE WHEN DATEPART(hour, j.TimeCreated) >= 19 OR DATEPART(hour, j.TimeCreated) < 7 THEN 1 ELSE 0 END) AS outside_7_to_7 FROM QuickBooks.JournalEntries AS j GROUP BY YEAR(j.TimeCreated), YEAR(j.Date) ORDER BY 1, 2; ``` > Exercise: Which posters entered journal entries dated on a Saturday or Sunday? Show the posted_by value and the number of weekend entries. Order does not matter. ## Duplicate payments and invoices A duplicate payment is the same bill paid twice, or the same work billed twice under two numbers. Auditors test three ways: an exact match on vendor, amount and date; a near match within a short window; and a match on the invoice number. > Diagram: Two bill cards with the same vendor, amount and date, joined by a line labeled as a duplicate candidate, then three steps to clear the candidate. **Exact match, `qbd` bills.** This query finds bills from the same vendor with the same amount on the same date. **Runs on:** `qbd`, SQL Server. ```sql SELECT b.VendorName, b.Amount, b.Date, COUNT(*) AS bills, STRING_AGG(b.ReferenceNumber, ', ') AS reference_numbers FROM QuickBooks.Bills AS b GROUP BY b.VendorName, b.Amount, b.Date HAVING COUNT(*) > 1; ``` **What it returns.** One pair: East Bayshore Auto Mall, $532.97, 2023-04-03, bill references 028 and 029. This is a candidate, not a finding. Read both bills. Two bills with consecutive numbers on the same day may be two real jobs, or one job entered twice. **Near match, `nonprofit` bills.** This query finds the same vendor and amount within 30 days, which catches duplicates that were entered on different dates. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT a.bill_id, b.bill_id AS other_bill_id, a.vendor_id, a.amount, a.bill_date, b.bill_date AS other_bill_date FROM dbo.bills AS a JOIN dbo.bills AS b ON b.vendor_id = a.vendor_id AND b.amount = a.amount AND b.bill_id > a.bill_id AND ABS(DATEDIFF(day, a.bill_date, b.bill_date)) <= 30; ``` **What it returns.** No rows. No vendor in this ledger was billed the same amount twice within 30 days. **Invoice number, `bigbear` invoices.** Two invoices with the same vendor and invoice number are a classic duplicate. **Runs on:** `bigbear`, SQL Server. ```sql SELECT i.vendor_id, i.invoice_number, COUNT(*) AS invoices, SUM(i.amount) AS total FROM dbo.invoices AS i GROUP BY i.vendor_id, i.invoice_number HAVING COUNT(*) > 1; ``` **What it returns.** No rows. Each vendor and invoice number pair appears once. **What to do.** For each candidate, check four things: the vendor's remittance address, the payment date and check or ACH number, whether the vendor sent one invoice or two, and the approval on each. If the second payment exists, get a credit or refund in writing. Record the result on the exception list. ## Vendor master checks The vendor master is the list of people you pay. A bad record there is how fake vendors and duplicate payments get in. Check four things. **Blank tax IDs.** Each vendor paid in a year needs a tax ID for Form 1099 reporting. This query counts vendors with no tax ID, and vendors with no bills or checks at all. **Runs on:** `qbd`, SQL Server. ```sql WITH flags AS ( SELECT v.ID, v.TaxIdentity, CASE WHEN EXISTS (SELECT 1 FROM QuickBooks.Bills AS b WHERE b.VendorId = v.ID) OR EXISTS (SELECT 1 FROM QuickBooks.Checks AS c WHERE c.PayeeId = v.ID) THEN 1 ELSE 0 END AS has_activity FROM QuickBooks.Vendors AS v ) SELECT COUNT(*) AS vendors, SUM(CASE WHEN TaxIdentity IS NULL OR LTRIM(RTRIM(TaxIdentity)) = '' THEN 1 ELSE 0 END) AS no_tax_id, SUM(CASE WHEN has_activity = 0 THEN 1 ELSE 0 END) AS no_bills_or_checks FROM flags; ``` **What it returns.** Of 54 vendors, 43 have no tax ID and 9 have no bills or checks. A vendor with no activity is not a problem by itself, but a vendor with no activity and an active status is a candidate for cleanup. A vendor with no tax ID that you paid is a real reporting gap. **Same name, different record.** The `bigbear` vendor list has several records for one company. This query finds vendor records that share a street address. **Runs on:** `bigbear`, SQL Server. ```sql SELECT a.address_line1, COUNT(*) AS vendor_records, STRING_AGG(a.vendor_name, ' | ') AS vendors FROM dbo.vendors AS a WHERE a.address_line1 IS NOT NULL GROUP BY a.address_line1 HAVING COUNT(*) > 1 ORDER BY vendor_records DESC, a.address_line1; ``` **What it returns.** Ten addresses with two records each. In nine of them the two names are the same company written two ways, such as `Cedar Security` and `CEDAR SECURITY`, or a name with and without `Inc`. One pair, Valley Equipment and Pioneer Equipment, has different names at one address and needs a question to the owner. Duplicate vendor records are how a payment gets sent twice. **Vendor and employee address match.** Compare the vendor address with the employee address. A shared address is a classic conflict-of-interest flag. **Runs on:** `qbd`, SQL Server. ```sql SELECT v.Name AS vendor_name, v.Line1 AS vendor_address, e.Name AS employee_name, e.Line1 AS employee_address FROM QuickBooks.Vendors AS v JOIN QuickBooks.Employees AS e ON UPPER(LTRIM(RTRIM(v.Line1))) = UPPER(LTRIM(RTRIM(e.Line1))); ``` **What it returns.** No rows. The three `qbd` employees have home street addresses, and none matches a vendor. The `bigbear` employees table has only a home city and postal code, and the `nonprofit` employees table has no address. This test can only run where both sides have street addresses. In this sample, only `qbd` qualifies. ## Benford's law Benford's law says that in many natural sets of numbers, the first digit is 1 about 30 percent of the time and 9 about 5 percent. It is a screening test, not proof of fraud. Human-made amounts, fixed prices and rounded estimates break the pattern for innocent reasons. Use it to decide where to look. > Diagram: Bar pairs for first digits 1 to 9. Benford's expected share is an outline bar and the actual share of nonprofit bill amounts is a filled bar. Digit 1 runs high and digit 2 runs low. **Runs on:** `nonprofit`, SQL Server. ```sql WITH digits AS ( SELECT CAST(LEFT(CONVERT(varchar(40), CAST(b.amount AS decimal(18, 2))), 1) AS int) AS first_digit FROM dbo.bills AS b WHERE b.amount >= 1 ) SELECT v.digit AS first_digit, COUNT(d.first_digit) AS actual_count, ROUND(100.0 * COUNT(d.first_digit) / (SELECT COUNT(*) FROM digits), 1) AS actual_pct, ROUND(100 * LOG10(1 + 1.0 / v.digit), 1) AS benford_pct FROM (VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS v(digit) LEFT JOIN digits AS d ON d.first_digit = v.digit GROUP BY v.digit ORDER BY v.digit; ``` **What it returns.** Across 13,871 bills, the first digit is 1 in 35.5 percent of bills (Benford expects 30.1) and 2 in 10.1 percent (Benford expects 17.6). Digits 3 to 9 are within about 2.5 points of the expected shares. Digits 1 and 2 are the outliers. **What to do.** An excess of 1s with a shortage of 2s can come from a fixed price list, a vendor that bills in round dollar blocks, or a standing monthly charge. Sample bills whose amounts start with 1, beginning with the largest, and check whether the same vendor billed the same figure repeatedly. Do not conclude anything from the test alone. > Exercise: Count the journal lines whose debit is a round amount: a whole multiple of 10,000 and at least 10,000. Return the count and the total of those debits. ## Cutoff around period end Cutoff asks whether transactions landed in the right period. A bill for June services posted in July overstates July expenses and understates June. A real cutoff test compares the service date with the posting date. This sample has no service or receipt date, so the test below only checks the period tags. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT COUNT(*) AS entries_outside_their_period FROM dbo.gl_journal_entries AS e JOIN dbo.fiscal_periods AS p ON p.fiscal_period_id = e.fiscal_period_id WHERE e.entry_date < p.start_date OR e.entry_date > p.end_date; ``` **What it returns.** Zero. Every entry sits inside the fiscal period it was tagged to. **What to do.** Zero is a good result, but it does not test cutoff. Ask for the receipt date on the largest bills in the last five days of each period. Compare that date with the period tag. That is the test auditors run, and it needs a column the sample does not have. ## Gaps in check numbers A check sequence should have no holes, and no number should appear twice. A gap can mean a voided check that was never recorded, a check that was pulled from the sequence, or a check printed and never entered. Count the gaps, then account for every one. > Diagram: A bar for the check number range 93 to 515. About 185 of the 423 numbers in that range are used, and 238 are missing. **Runs on:** `qbd`, SQL Server. ```sql WITH nums AS ( SELECT CAST(c.ReferenceNumber AS int) AS check_no FROM QuickBooks.Checks AS c WHERE c.ReferenceNumber NOT LIKE '%[^0-9]%' ) SELECT COUNT(*) AS numeric_checks, COUNT(DISTINCT check_no) AS distinct_numbers, MIN(check_no) AS first_number, MAX(check_no) AS last_number, MAX(check_no) - MIN(check_no) + 1 - COUNT(DISTINCT check_no) AS numbers_missing_in_range FROM nums; ``` **What it returns.** 185 numeric checks, all distinct, running from 93 to 515. That leaves 238 numbers missing from the range. No number is used twice. Before you call those 238 gaps, clear the non-numeric entries. The next query shows them. **Runs on:** `qbd`, SQL Server. ```sql SELECT c.ReferenceNumber, COUNT(*) AS checks, SUM(c.Amount) AS amount FROM QuickBooks.Checks AS c WHERE c.ReferenceNumber IS NULL OR c.ReferenceNumber LIKE '%[^0-9]%' GROUP BY c.ReferenceNumber; ``` **What it returns.** The reference `DRAFT` on 22 checks, totaling $275.00. Each check is for $12.50, and the payee on most of them is a bad check charge. These are not check numbers. A check reference of `DRAFT` should never reach a printed check. Ask why the system allowed it, and make sure a bank charge like this is posted once. **What to do.** Get the voided-check log and the bank's list of cleared checks. Each missing number should appear in one of them. Any number that appears in neither is an exception. > Exercise: How many checks were written in each calendar year, and what was their total amount rounded to cents? Oldest year first. Skip checks with no date. ## AR and AP aging reasonableness Aging is a reasonableness test. The total should agree to the ledger, and the buckets should agree to the due dates. The test below ages open bills against the as-of date in the sample, 2026-10-24. **Runs on:** `nonprofit`, SQL Server. ```sql SELECT CASE WHEN days_past_due <= 0 THEN '1. Not yet due' WHEN days_past_due <= 30 THEN '2. 1 to 30 days' WHEN days_past_due <= 60 THEN '3. 31 to 60 days' WHEN days_past_due <= 90 THEN '4. 61 to 90 days' ELSE '5. Over 90 days' END AS bucket, COUNT(*) AS bills, SUM(amount) AS amount FROM (SELECT amount, DATEDIFF(day, due_date, '2026-10-24') AS days_past_due FROM dbo.bills WHERE status IN ('Open', 'Overdue')) AS x GROUP BY CASE WHEN days_past_due <= 0 THEN '1. Not yet due' WHEN days_past_due <= 30 THEN '2. 1 to 30 days' WHEN days_past_due <= 60 THEN '3. 31 to 60 days' WHEN days_past_due <= 90 THEN '4. 61 to 90 days' ELSE '5. Over 90 days' END ORDER BY 1; ``` **What it returns.** 226 bills not yet due ($258,948.23), 30 bills one to 30 days late ($24,599.24) and 2 bills 31 to 60 days late ($1,782.07). Total open: $285,329.54. The 32 past-due bills match the 32 bills flagged `Overdue` in the status column. The aging and the status agree. For receivables, the same pattern works on the `cpafirm` invoice balances. The [SQL for accountants](/analyze/sql-for-accountants) guide builds that aging step by step. ## What to do with exceptions Every test produces a list of hits. Most hits are explained by a normal process. Some are not. The work is to sort them, sample the ones you cannot explain, and follow up in writing. > Diagram: Four steps in a row: define the population, filter for the risky items, sample and read the support, then conclude and follow up. Use these rules to decide the next step: - **Explain the population first.** A weekend bill payment may be a system date. Confirm that before you read a single entry. - **Sample the unexplained hits.** Read the support for a sample. Pick the largest hits, the newest, and any that repeat a pattern. If the sample shows a problem, widen it. If it does not, document the sample and the result. - **Record each hit.** Keep the test name, the key (bill ID, check number or entry number), the amount, the owner and the result. Save the query so the test can be rerun. - **Follow up with the owner, in writing.** Ask the person who posted or approved the item. Their answer is evidence, but it is not the end. Check it against the support. - **Escalate when the answer does not match the support.** A missing invoice, a second payment, or an approval that cannot be found goes to the person who owns the control. | Test | Typical hit | First step | Escalate when | |---|---|---|---| | Weekend and holiday | Bill dated on a holiday | Ask how the date is set | A person typed a date that has no vendor invoice | | Round amounts | Round expense | Read the invoice | The amount has no invoice or matches no order | | Threshold | A band just under the limit | Compare with the real limit | Several entries from one person, one day | | Duplicate bill | Same vendor, amount and date | Check both bills and payments | Two payments went out | | Vendor tax ID | Paid vendor with no tax ID | Request a W-9 | No W-9 for a vendor over the reporting limit | | Check gap | Missing number | Check the void log and bank list | A missing check has no void record | | Benford | Excess of 1s | Sample the top of the range | A repeated amount with no contract | ## What the sample data cannot test These tests need data the samples do not have. Each one is worth running on a real system. - **After-hours postings** need current timestamps and a time zone. The `qbd` timestamps are from 2003 to 2011. - **Entry approval** needs an approval table, a signing authority list, and the approver's name on each entry. Nothing in the samples records who approved an entry. - **Who created or changed a record** needs an audit log. The `nonprofit` ledger has `posted_by`, but the `bigbear` and `qbd` ledgers do not. - **Real cutoff** needs service dates and receipt dates on bills. - **Employee and vendor address conflicts** need street addresses on the employee record. Only `qbd` has them, and none matched. - **Approval thresholds** need your real limits, not a guess. If your system has these fields, use them. The method stays the same: population, test, hits, sample, follow-up. --- # CPA firm practice management (cpafirm) `cpafirm` is the practice-management database of a fictional small-business CPA firm: 17 people in two offices (Bellevue and Portland), about 1,000 clients, and nearly six years of work, billing, collections and bookkeeping from January 2021 to October 2026. It has 34 tables and roughly 400,000 rows, and the data is identical on SQL Server and MariaDB. It is built to look like the reports a real firm runs: realization by partner, WIP, AR aging, staff utilization, late returns, retainer balances and a general ledger that balances. ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `cpafirm`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** open [the database page](/databases/cpafirm) for the Playground, the SQL editor and the schema explorer. - **Dates:** the data is as of **2026-10-24**. That date is stored in `dataset_info.as_of_date`, and the reporting views use it instead of today's date, so results do not drift. ## How the firm works Follow the money and you have the whole model. 1. **Clients** belong to a partner and a manager, have an entity type (individual, S corporation, partnership, C corporation, nonprofit, trust) and an industry. 2. The firm sells **services** in six service lines: Tax Preparation, Tax Planning, Bookkeeping, Payroll, Assurance and Advisory. 3. Each piece of work is an **engagement**. Tax and assurance work is one engagement per client per tax year. Recurring services (bookkeeping, payroll, virtual CFO) are one engagement per client per calendar year, billed every month. 4. Staff record **time entries** against engagements, at the rate for their role in that year. 5. **Invoices** (with **invoice lines**) bill the work. Time entries point at the invoice that billed them; unbilled time has no invoice. 6. **Payments** arrive and are applied to one or more invoices. Some clients prepay through a **trust** account (a retainer) and the firm applies it to invoices. Small balances are adjusted off. 7. Every invoice, payment, adjustment and trust movement posts to the **general ledger**, together with monthly operating expenses. ## Tables at a glance | Area | Tables | What they hold | |---|---|---| | Firm | `offices`, `staff_roles`, `staff` | Two offices; six roles with standard billing rates; 17 staff (one has left, one joined recently) with a manager chain | | Lookups | `entity_types`, `industries`, `referral_sources`, `service_lines`, `services` | Reference lists; `services` has 17 services with default fees | | Clients | `clients`, `client_relationships`, `contacts` | 1,000 clients (900 active, 60 inactive, 40 prospects); ownership links between people and their entities; 1,500 contacts | | Work | `engagements`, `engagement_staff`, `tasks`, `tax_returns`, `time_entries`, `client_visits` | 7,494 engagements, who is on each, a checklist of tasks, 6,635 federal and state returns, 70,044 time entries, 2,362 trips to client sites | | Billing | `invoices`, `invoice_lines`, `payments`, `payment_applications`, `invoice_adjustments`, `trust_transactions` | 21,282 invoices, 20,222 payments, how payments are applied, write-offs and discounts, and retainer activity | | General ledger | `gl_accounts`, `gl_journal_entries`, `gl_journal_lines` | A 24-account chart and about 93,600 balanced lines | | Documents | `document_requests`, `tax_organizers`, `esign_requests` | Document requests to clients, tax organizers and e-signature requests (engagement letters, Form 8879) | | Geography and fiscal | `geo_states`, `geo_places`, `geo_postal_codes`, `fiscal_calendar` | Reference geography and a July-June fiscal calendar with the tax-season rhythm | | Meta | `dataset_info` | The as-of date | ## How the tables join ``` clients ──< engagements >── services >── service_lines │ │ │ │ │ ├──< time_entries >── staff (time_entries.invoice_id → invoices; NULL = unbilled) │ │ ├──< tax_returns (federal and state rows per tax-prep engagement) │ │ ├──< tasks, document_requests │ │ └──< engagement_staff │ └──< invoice_lines >── invoices >── clients └──< payments ──< payment_applications >── invoices invoices ──< invoice_adjustments clients ──< trust_transactions gl_journal_lines >── gl_journal_entries (source_type + source_id point back to the invoice, payment, ...) ``` The `engagements.partner_id` and `manager_id` columns are copied from the client so you can report by partner without an extra join. ## Rules built into the data - **Fixed fees versus hours.** Tax and assurance work is billed at a fixed fee, so billed amount and hours worked do not match. That is what makes **realization** (billed divided by hours at standard rates) interesting. Overall it is about 87%. - **Work in process (WIP).** Billable time with no invoice (`time_entries.invoice_id IS NULL`) is unbilled WIP. Engagements still in progress have some. - **One payment, many invoices.** A payment can cover several invoices, and an invoice can be paid in parts. Use `payment_applications` to follow the money. - **Aging.** An invoice is overdue when its balance (`total_amount` minus applied payments minus adjustments) is above zero and `due_date` is before the as-of date. - **Ledger.** Debits equal credits in every entry. Revenue accounts are by service line, and `4900` holds billing adjustments and discounts. - **Prospects.** About 40 clients have status `Prospect` and no activity at all. Filter them out of client counts. ## Views | View | Use | |---|---| | `vw_invoice_balances` | Paid, adjusted and open amount for every invoice, plus days past due | | `vw_ar_aging` | Open receivables by client in current, 1-30, 31-60, 61-90 and over-90 buckets | | `vw_unbilled_wip` | Unbilled hours and value by engagement | | `vw_engagement_realization` | Billed amount, hours, standard value, labor cost and realization per engagement | | `vw_staff_monthly_hours` | Total and billable hours per person per month | ## Questions to try **Which service lines earned the most in 2025?** ```sql SELECT sl.line_name, SUM(l.amount) AS billed FROM invoice_lines l JOIN invoices i ON i.invoice_id = l.invoice_id JOIN engagements e ON e.engagement_id = l.engagement_id JOIN services s ON s.service_id = e.service_id JOIN service_lines sl ON sl.service_line_id = s.service_line_id WHERE YEAR(i.issue_date) = 2025 GROUP BY sl.line_name ORDER BY billed DESC; ``` Bookkeeping comes first at about $1.3 million, then Tax Preparation at about $1.0 million. **Who owes the most, and how much of it is more than 90 days old?** ```sql SELECT client_name, total_due, days_over_90 FROM vw_ar_aging ORDER BY total_due DESC OFFSET 0 ROWS FETCH FIRST 10 ROWS ONLY; ``` **What is each partner's realization?** ```sql SELECT CONCAT(st.first_name, ' ', st.last_name) AS partner, SUM(r.billed_amount) AS billed, SUM(r.standard_value) AS standard_value, ROUND(SUM(r.billed_amount) / SUM(r.standard_value), 3) AS realization FROM vw_engagement_realization r JOIN engagements e ON e.engagement_id = r.engagement_id JOIN staff st ON st.staff_id = e.partner_id WHERE r.standard_value > 0 GROUP BY st.staff_id, st.first_name, st.last_name ORDER BY realization; ``` **Which returns were filed after their extended due date?** ```sql SELECT c.client_name, t.form_type, t.tax_year, t.extended_due_date, t.filed_date FROM tax_returns t JOIN clients c ON c.client_id = t.client_id WHERE t.filed_date > COALESCE(t.extended_due_date, t.due_date) ORDER BY t.filed_date DESC; ``` **How busy was everyone in 2025?** ```sql SELECT first_name, last_name, SUM(total_hours) AS hours, ROUND(SUM(billable_hours) * 1.0 / SUM(total_hours), 3) AS utilization FROM vw_staff_monthly_hours WHERE work_year = 2025 GROUP BY staff_id, first_name, last_name ORDER BY utilization DESC; ``` **Where is unbilled time piling up?** ```sql SELECT c.client_name, w.unbilled_hours, w.unbilled_value FROM vw_unbilled_wip w JOIN engagements e ON e.engagement_id = w.engagement_id JOIN clients c ON c.client_id = e.client_id ORDER BY w.unbilled_value DESC OFFSET 0 ROWS FETCH FIRST 10 ROWS ONLY; ``` These queries run on both SQL Server and MariaDB. Other ideas: write-offs by partner, revenue per client by entity type, days from engagement open to invoice, collection time by client, and trust balances by client. ## Excel and Power BI - **Star model.** Use `invoice_lines`, `time_entries` and `payments` as fact tables, with `clients`, `services`, `staff` and a calendar table as dimensions. Build the calendar from `invoices.issue_date`, or from the date range with `dataset_info.as_of_date` as the end. - **Measures to write.** Billed, Hours, Billable hours, Realization % (billed over standard value), Utilization %, Open AR, Days sales outstanding, WIP value. - **Power Query.** Merge `time_entries` to `staff` for the rate, and `invoices` to `payment_applications` to build your own aging. The [modeling guide](/analyze/learnsql-models) shows the same recipe for the other databases. ## Maps and fiscal years **Mapping.** Every client and both offices have a full address, `latitude`, `longitude` and `geocode_quality`, and `client_visits` has the origin and destination of each trip from an office to a client, with round-trip miles and drive time. See [maps and fiscal calendars](/analyze/maps-and-fiscal) for the shared pattern. **The fiscal year runs July to June**, named by the year it ends (FY2026 is July 2025 to June 2026). `engagements`, `invoices`, `payments`, `time_entries`, `gl_journal_entries` and `client_visits` carry `fiscal_year`, `fiscal_quarter` and `fiscal_period`. `fiscal_calendar` adds `tax_season` (Busy season, Extension season, Off season) and `tax_deadline`. **What was billed in each fiscal quarter?** ```sql SELECT fiscal_year, fiscal_quarter, COUNT(*) AS invoices, SUM(total_amount) AS billed FROM invoices WHERE fiscal_year BETWEEN 2024 AND 2026 GROUP BY fiscal_year, fiscal_quarter ORDER BY fiscal_year, fiscal_quarter; ``` **How much do we bill in busy season versus extension season?** ```sql SELECT f.tax_season, COUNT(*) AS invoices, SUM(i.total_amount) AS billed FROM invoices i JOIN fiscal_calendar f ON f.calendar_date = i.issue_date WHERE i.fiscal_year = 2025 GROUP BY f.tax_season ORDER BY billed DESC; ``` **How many miles did each person drive to client sites in fiscal 2025?** ```sql SELECT s.first_name, s.last_name, COUNT(*) AS visits, ROUND(SUM(v.miles_round_trip), 0) AS miles FROM client_visits v JOIN staff s ON s.staff_id = v.staff_id WHERE v.fiscal_year = 2025 GROUP BY s.staff_id, s.first_name, s.last_name ORDER BY miles DESC OFFSET 0 ROWS FETCH FIRST 5 ROWS ONLY; ``` **Which Washington counties have the most active clients?** ```sql SELECT c.county, COUNT(*) AS clients FROM clients c WHERE c.status = 'Active' AND c.state = 'WA' GROUP BY c.county ORDER BY clients DESC OFFSET 0 ROWS FETCH FIRST 5 ROWS ONLY; ``` Clients in Arizona, California and Idaho are served remotely, so filter by distance (see the haversine query in the maps guide) before you map visits. ## Realistic imperfections Some of the mess is deliberate, so your reports have something to find: about 6% of engagements have no engagement letter on file, a few returns are filed late, some invoices are overdue or written off, some document requests are still outstanding, and the trust account holds a few unapplied retainers. --- # Nonprofit donors, grants and funds (nonprofit) `nonprofit` is the database of Cascade Community Alliance, a fictional regional nonprofit with about $7 million a year in revenue and 42 employees. It runs four programs (Food Security, Youth Opportunity, Housing Stability and Community Health Outreach) and is funded by individual donors, foundation and government grants, events and program fees. It covers what a nonprofit actually keeps: **donor management** (CRM), **fundraising** (campaigns, pledges, events), **grants and proposals**, **fund accounting** with a general ledger, **personnel** records and **volunteers**. There are 49 tables and roughly 411,000 rows, identical on SQL Server and MariaDB. The fiscal year runs **July to June**, from FY2022 (July 2021) to October 2026, which is the first four months of FY2027. ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `nonprofit`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** the [database page](/databases/nonprofit) opens the Playground, the SQL editor and the schema explorer. - **As-of date:** 2026-10-24, stored in `dataset_info`. ## How a nonprofit differs Three ideas explain most of the design. 1. **Funds, not just accounts.** Money is tracked by *fund* as well as by account. A fund is either *without donor restrictions* (General Operating, a board-designated reserve) or *with donor restrictions* (a capital campaign, an endowment, program funds, and one fund per grant). Every ledger line has a fund. The same accounts show a different story when you filter by fund. 2. **Functional expenses.** Costs are classed as Program, Management and General, or Fundraising, through the cost center on each line. That is what feeds the statement of functional expenses. 3. **Restricted money has to be spent as promised.** Grants carry a budget by category, payments, and deliverables with due dates. Staff time is allocated across funds by percent of effort. ## Tables at a glance | Area | Tables | What they hold | |---|---|---| | Organization | `fiscal_periods`, `programs`, `cost_centers`, `funds` | 64 monthly periods, four programs, eight cost centers with a functional class, 103 funds | | Donors and CRM | `constituents`, `constituent_relationships`, `interests`, `constituent_interests`, `interactions`, `opportunities` | 20,887 people and organizations, spouse and employer links, interests, 6,400 interactions, and a major-gift pipeline | | Giving | `campaigns`, `appeals`, `events`, `event_registrations`, `gift_batches`, `gifts`, `soft_credits`, `gift_acknowledgments`, `pledges`, `recurring_gifts` | 103,286 gifts from 17,237 donors, 238 pledges, 1,976 monthly donors, 24 events | | Grants | `funders`, `proposals`, `grants`, `grant_budgets`, `grant_payments`, `grant_deliverables` | 44 funders, 166 proposals, 94 awarded grants with budgets, receipts and reports | | People | `job_titles`, `employees`, `salary_history`, `pay_periods`, `payroll_records`, `timesheet_allocations` | 58 employees over time, pay history, semi-monthly payroll, and effort split by fund | | Volunteers | `volunteers`, `volunteer_shifts` | 1,420 volunteers and 21,000 shifts | | Accounting | `gl_accounts`, `gl_journal_entries`, `gl_journal_lines`, `budgets`, `vendors`, `bills` | A 44-account chart, about 104,000 balanced lines, annual budgets, 113 vendors and 13,900 bills | | Sites and field work | `program_sites`, `mobile_market_routes`, `mobile_market_stops`, `mobile_market_visits` | Seven program sites, a mobile food market with 3 routes and 24 stops, and 6,300 weekly visits | | Geography and fiscal | `geo_states`, `geo_places`, `geo_postal_codes`, `fiscal_calendar` | Reference geography and a July-June fiscal calendar with giving seasons | | Meta | `dataset_info` | The as-of date | ## How the tables join ``` constituents ──< gifts >── funds gifts → campaigns / appeals / pledges / recurring_gifts / gift_batches │ └──< soft_credits >── constituents (matching gifts, donor-advised funds, household) ├──< opportunities ──< interactions >── employees (major-gift pipeline) └──< event_registrations >── events >── campaigns funders ──< proposals >── programs proposals ──1:1── grants >── funds grants ──< grant_budgets, grant_payments, grant_deliverables employees ──< payroll_records >── pay_periods employees ──< timesheet_allocations >── funds (percent of effort by pay period) gl_journal_lines >── gl_journal_entries >── fiscal_periods gl_journal_lines >── gl_accounts, funds, cost_centers bills >── vendors, gl_accounts, funds, cost_centers, grants ``` ## Rules built into the data - **Ledger.** Every journal entry balances, and so does every fund on its own. Cash and payables sit in the General fund; other funds balance through account `1900` (due from or to other funds). Pledges are recognized when made (`1100`), and later gifts reduce the receivable. In-kind gifts post revenue and an equal expense. - **Grants.** Non-reimbursable grants are recognized at award. Government grants are *reimbursable*: revenue is recognized as costs are incurred, plus indirect, and collected quarterly. The indirect share goes to the General fund. - **Payroll.** Payroll posts every pay period, split across funds by `timesheet_allocations`. Grant-funded staff are charged to the grant fund until its personnel budget is used up. - **Simplified net assets.** Each fund carries its own balance. There are no release-from-restriction entries. - **Soft credits.** Matching gifts are credited to the employee, donor-advised fund gifts to the advisor, and household gifts to the spouse. Use `soft_credits` when you want to credit the person and `gifts` when you want the cash. - **Fiscal year.** FY2025 is July 2024 to June 2025. `fiscal_periods` and `pay_periods.fiscal_year` carry it, and gifts need `CASE WHEN MONTH(gift_date) >= 7 THEN YEAR(gift_date) + 1 ELSE YEAR(gift_date) END`. ## Views | View | Use | |---|---| | `vw_donor_summary` | Gift count, lifetime giving, first and last gift per donor | | `vw_pledge_balances` | Paid, written-off and open balance per pledge | | `vw_fund_balances` | Opening net assets, revenue, expenses and ending net assets by fund | | `vw_grant_budget_vs_actual` | Budget against actual spending by grant and category | | `vw_functional_expenses` | Expenses by fiscal year, function class, cost center and account | ## Questions to try **What share of donors give again the next year?** ```sql WITH gift_fy AS ( SELECT constituent_id, CASE WHEN MONTH(gift_date) >= 7 THEN YEAR(gift_date) + 1 ELSE YEAR(gift_date) END AS fy FROM gifts WHERE gift_type NOT IN ('In-Kind', 'Matching Gift') ), donors AS ( SELECT DISTINCT constituent_id, fy FROM gift_fy ) SELECT a.fy AS from_fy, COUNT(*) AS donors, SUM(CASE WHEN b.constituent_id IS NOT NULL THEN 1 ELSE 0 END) AS retained, ROUND(SUM(CASE WHEN b.constituent_id IS NOT NULL THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 1) AS retention_pct FROM donors a LEFT JOIN donors b ON b.constituent_id = a.constituent_id AND b.fy = a.fy + 1 WHERE a.fy BETWEEN 2022 AND 2025 GROUP BY a.fy ORDER BY a.fy; ``` Retention is about 54% in FY2022 and 64-65% in the later years. **How did each FY2025 campaign do against its goal?** ```sql SELECT c.campaign_name, c.goal_amount, SUM(g.amount) AS raised, ROUND(SUM(g.amount) * 100.0 / c.goal_amount, 1) AS pct_of_goal FROM campaigns c JOIN gifts g ON g.campaign_id = c.campaign_id WHERE c.fiscal_year = 2025 GROUP BY c.campaign_id, c.campaign_name, c.goal_amount ORDER BY raised DESC; ``` **What are the largest fund balances?** ```sql SELECT fund_code, fund_name, net_asset_class, ending_net_assets FROM vw_fund_balances ORDER BY ending_net_assets DESC OFFSET 0 ROWS FETCH FIRST 10 ROWS ONLY; ``` **How much of each grant's personnel budget has been spent?** ```sql SELECT grant_number, budgeted_amount, actual_amount FROM vw_grant_budget_vs_actual WHERE budget_category = 'Personnel' ORDER BY budgeted_amount DESC OFFSET 0 ROWS FETCH FIRST 10 ROWS ONLY; ``` **What is the split between program, administration and fundraising?** ```sql SELECT function_class, SUM(expense_amount) AS expenses, ROUND(SUM(expense_amount) * 100.0 / SUM(SUM(expense_amount)) OVER (), 1) AS pct FROM vw_functional_expenses WHERE fiscal_year = 2025 GROUP BY function_class ORDER BY expenses DESC; ``` In FY2025 about 64% of expenses are program, 19% management and general, and 17% fundraising. **Which kinds of funders say yes most often?** ```sql SELECT f.funder_type, COUNT(*) AS decided, SUM(CASE WHEN p.status = 'Awarded' THEN 1 ELSE 0 END) AS won, ROUND(SUM(CASE WHEN p.status = 'Awarded' THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 1) AS win_rate_pct FROM proposals p JOIN funders f ON f.funder_id = p.funder_id WHERE p.status IN ('Awarded', 'Declined') GROUP BY f.funder_type ORDER BY decided DESC; ``` **What is in each major-gift officer's pipeline?** ```sql SELECT CONCAT(e.first_name, ' ', e.last_name) AS officer, o.stage, COUNT(*) AS opportunities, SUM(o.ask_amount) AS asked FROM opportunities o JOIN employees e ON e.employee_id = o.officer_id WHERE o.stage NOT LIKE 'Closed%' GROUP BY e.employee_id, e.first_name, e.last_name, o.stage ORDER BY officer, o.stage; ``` All of these run on SQL Server and MariaDB. More ideas: lapsed donors who gave $1,000 or more, cost to raise a dollar by appeal, overdue grant deliverables, personnel cost by fund from effort allocations, volunteer hours by program, and bills that are overdue. ## Excel and Power BI - **Giving model.** `gifts` is the fact table, with `constituents`, `funds`, `campaigns` and a date table as dimensions. Add a fiscal year column to the date table (July start), and set up time intelligence with a fiscal year end of June 30. - **Fund accounting model.** `gl_journal_lines` is the fact table, with `gl_accounts`, `funds`, `cost_centers` and `fiscal_periods`. Revenue is credit minus debit, and expense is debit minus credit. - **Measures to write.** Total giving, Donors, Retention %, Average gift, Raised vs goal, Net assets by fund, Program expense ratio, Grant spend vs budget. ## Maps and fiscal years **Mapping.** About 18,500 constituents have a full address with `latitude`, `longitude` and `geocode_quality`, and so do funders, vendors, program sites and mobile-market stops. The **mobile food market** has three weekly routes with eight stops each, and `mobile_market_visits` records the households served and the pounds distributed, which is ready for a map of reach over time. See [maps and fiscal calendars](/analyze/maps-and-fiscal). **Fiscal columns.** `gifts`, `pledges`, `bills`, `grant_payments`, `interactions`, `opportunities`, `gl_journal_entries`, `volunteer_shifts` and `mobile_market_visits` carry `fiscal_year`, `fiscal_quarter` and `fiscal_period` (July is period 1), matching `fiscal_periods`. `fiscal_calendar` adds `giving_season`, holidays and Giving Tuesday, so you no longer need the CASE expression to derive the fiscal year of a gift. **When do we raise the money?** ```sql SELECT f.giving_season, COUNT(*) AS gifts, SUM(g.amount) AS raised FROM gifts g JOIN fiscal_calendar f ON f.calendar_date = g.gift_date WHERE g.fiscal_year = 2025 GROUP BY f.giving_season ORDER BY raised DESC; ``` **Giving by fiscal quarter:** ```sql SELECT fiscal_year, fiscal_quarter, SUM(amount) AS raised FROM gifts WHERE fiscal_year BETWEEN 2024 AND 2026 GROUP BY fiscal_year, fiscal_quarter ORDER BY fiscal_year, fiscal_quarter; ``` **Which states give the most?** ```sql SELECT s.state_name, COUNT(DISTINCT g.constituent_id) AS donors, SUM(g.amount) AS raised FROM gifts g JOIN constituents c ON c.constituent_id = g.constituent_id JOIN geo_states s ON s.state_code = c.state WHERE g.fiscal_year = 2025 GROUP BY s.state_name ORDER BY raised DESC OFFSET 0 ROWS FETCH FIRST 5 ROWS ONLY; ``` **How many households did each mobile market route serve?** ```sql SELECT r.route_name, v.fiscal_year, SUM(v.households_served) AS households, SUM(v.pounds_distributed) AS pounds FROM mobile_market_visits v JOIN mobile_market_stops s ON s.stop_id = v.stop_id JOIN mobile_market_routes r ON r.route_id = s.route_id WHERE v.fiscal_year IN (2025, 2026) GROUP BY r.route_name, v.fiscal_year ORDER BY r.route_name, v.fiscal_year; ``` **Where do volunteers work?** ```sql SELECT p.site_name, p.city, COUNT(*) AS shifts, SUM(vs.hours) AS hours FROM volunteer_shifts vs JOIN program_sites p ON p.site_id = vs.site_id WHERE vs.fiscal_year = 2025 GROUP BY p.site_name, p.city ORDER BY hours DESC; ``` Funders were originally given a random state; each funder now uses the state of its city so its address is consistent. ## Realistic imperfections Acknowledgments are sometimes late or missing, a few grant deliverables are overdue, some pledges are written off, some bills are open or overdue, and some constituents are deceased or marked do-not-contact. FY2027 is only four months old, and many grants are recognized when awarded, so its revenue looks high next to its expenses. --- # Power Query playground (pqlab) `pqlab` is a database made for learning and showing off **Power Query**, the data-shaping tool in Excel, Power BI and Fabric. It holds 86 tables and about 142,000 rows, identical on SQL Server and MariaDB, and everything in it is fictional data for a made-up outdoor-gear retailer, Harbor & Pine Outfitters. It has two layers, and the data is **deliberately imperfect**, the way real extracts are. | Layer | Tables | What it is for | |---|---|---| | `core_*` | 17 | A joinable retail backbone with customers, products, categories, stores, employees, orders, order lines, shipments, returns, inventory, promotions, suppliers, a calendar, a fiscal calendar and reference geography. Mostly clean, with realistic gaps. Use it to build real reports and to practice merges, grouping and custom columns. | | `lab_*` | 68 plus `lab_catalog` | One table per Power Query skill, each deliberately messy: pasted reports, text in every format, JSON and XML, dates and numbers as text, schema drift, duplicates, null markers, ugly column names. | `lab_catalog` is an index of every table with its category, level (Beginner, Intermediate or Advanced), techniques and a how-to. The full catalog is at the end of this page. ## Connect from Excel or Power BI 1. In Excel choose **Data > Get Data > From Database > From SQL Server Database**. In Power BI Desktop choose **Get data > SQL Server**. 2. Server: `learnsql.cpetoday.com,1433`. Database: `pqlab`. Choose **Import**. 3. In the Navigator, search for a prefix such as `lab_hdr` or `lab_mrg` to find a family of tables. Pick one table and choose **Transform Data** to open Power Query. See [Connect Excel and Power BI](/connect) for the sign-in details. The tables are in the `dbo` schema. ## Things to know first - **Sort by `row_id` first on the spreadsheet-style tables** (`lab_hdr_*`). A SQL source does not guarantee row order, and steps like Remove Top Rows or Use First Row as Headers depend on it. Every lab table has an integer key: `row_id`, or `line_no`. - **Several tables are all text on purpose.** Setting the types is part of the exercise. - **Answer keys.** Some tables include the true value so you can check your work: `lab_dt_formats.date_true`, `lab_dt_events_tz.start_utc` and `lab_num_formats.value_true`. - **`core_*` is a sample.** Orders run from January 2023 to October 2026 (14,000 orders, about $11 million) and are all in US dollars. - **Known imperfections in `core_*`:** about 45 near-duplicate customers, state spellings that vary (WA, Wash., Washington), tier casing, missing emails, phones, weights and ratings, and a few trailing spaces in product names. - **Fiscal year.** The retailer's fiscal year starts on February 1. `core_calendar.fiscal_year` is the year in which the fiscal year starts. - **Odd column names** (spaces, symbols, upper case) exist on purpose in `lab_app_orders_2024`, `lab_piv_budget_matrix` and `lab_err_wide`. ## "I want to practice..." | Power Query skill | Start with | |---|---| | Remove top and bottom rows, use first row as headers | `lab_hdr_title_rows`, `lab_hdr_first_row` | | Two-level headers, fill right, transpose | `lab_hdr_two_row`, `lab_piv_kpi_rows` | | Fill down, conditional column on layout | `lab_hdr_blocks`, `lab_hdr_ledger` | | Stacked records, index and modulo, pivot | `lab_hdr_stacked`, `lab_hdr_alternating` | | Unpivot columns | `lab_piv_sales_wide`, `lab_piv_survey_wide`, `lab_piv_budget_matrix` | | Pivot column, with or without aggregation | `lab_piv_sales_long`, `lab_piv_attributes` | | Split a column (delimiter, position, into rows) | `lab_txt_contacts`, `lab_piv_tags`, `lab_txt_addresses`, `lab_hier_category_paths` | | Trim, clean, case, replace, extract | `lab_txt_contacts`, `lab_txt_sku_parse`, `lab_txt_names_accents` | | Add column from examples, merge columns | `lab_txt_sku_parse`, `lab_txt_contacts` | | Parse JSON, expand records and lists | `lab_json_orders`, `lab_json_product_specs` | | Parse XML and HTML | `lab_xml_shipments`, `lab_txt_html` | | Base64 and URL decoding | `lab_txt_encoded` | | Types with a locale, dates in many formats | `lab_dt_formats`, `lab_num_formats` | | Time zones and durations | `lab_dt_events_tz` | | Fill, group by hour, de-duplicate, rolling windows | `lab_dt_sensor` | | Group by (every aggregation, all rows) and rank | `lab_num_scores`, `core_orders` | | Append tables whose columns drift | `lab_app_orders_2023` to `_2026`, `lab_app_web_2026_01` to `_05` | | Merge join kinds and anti joins | `lab_mrg_customers_crm`, `lab_mrg_customers_billing` | | Fuzzy merge and fuzzy grouping | `lab_mrg_company_variants`, `lab_mrg_company_master` | | Replace values through a lookup table | `lab_mrg_state_aliases`, `lab_mrg_state_lookup` | | Merge on keys of different types | `lab_mrg_sales_textkeys` | | Range and date-effective joins | `lab_mrg_price_history`, `lab_dt_fiscal_periods`, `lab_mrg_commission_tiers` | | Cross join | `lab_mrg_sizes` with `lab_mrg_colors` | | Currency conversion (merge and fill down) | `lab_dt_fx_rates` with `lab_app_orders_2025` | | Remove duplicates and keep the latest | `lab_dup_contacts`, `lab_dup_customer_versions` | | Parameters and custom functions | `lab_fn_parameters`, `lab_fn_report_requests`, `lab_json_api_pages` | | Recursive hierarchy and self-merge | `lab_hier_bom`, `core_employees`, `core_categories` | | Null markers, errors and column profiling | `lab_err_null_markers`, `lab_err_mixed_types`, `lab_err_impossible_values` | | Rename, choose and reorder columns | `lab_err_wide` | | Standardize categories | `lab_err_category_variants` | | Query folding on larger tables | `core_orders`, `core_order_lines`, `core_inventory` | | Create a custom data type (Excel) | `lab_type_trails`, `core_products`, `core_customers`, `core_employees`, `core_stores` | | Parse coordinates written many ways | `lab_geo_coordinates_text` | | Geocode messy addresses by merging to a reference table | `lab_geo_addresses_to_geocode`, `core_geo_postal_codes` | | Distances between points, GPS breadcrumbs | `lab_geo_route_points`, `lab_geo_distance_pairs` | | Fiscal years, 4-5-4 retail weeks | `core_fiscal_calendar` | ## Recipes These are starting points in M, the Power Query language. Paste one into **Home > Advanced Editor** in a blank query. In each recipe, `Source` points at the table named in its `Item`. ### Pasted report: remove the title block, promote headers, drop footers ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_hdr_title_rows"]}[Data], Sorted = Table.Sort(Source, {{"line_no", Order.Ascending}}), Dropped = Table.RemoveColumns(Sorted, {"line_no"}), Skipped = Table.Skip(Dropped, 4), Promoted = Table.PromoteHeaders(Skipped, [PromoteAllScalars = true]), NoRepeat = Table.SelectRows(Promoted, each [Product] <> null and [Product] <> "Product" and not Text.StartsWith([Product], "Page ")), NoFooter = Table.SelectRows(NoRepeat, each [Product] <> "Total" and not Text.StartsWith([Product], "--")) in NoFooter ``` > **Watch out.** The report repeats its header at every page break and ends with a total row and an end-of-report line, so filtering is needed after promoting the headers. Then remove the `$` and commas and set the types. ### Twelve month columns: unpivot ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_piv_sales_wide"]}[Data], Trimmed = Table.RemoveColumns(Source, {"row_id", "ytd_total"}), Unpivoted = Table.UnpivotOtherColumns(Trimmed, {"product_id", "product_name", "region", "sales_year"}, "month", "sales"), NoNulls = Table.SelectRows(Unpivoted, each [sales] <> null) in NoNulls ``` Remove `ytd_total` first, or it would unpivot as if it were a month. ### Budget and actual in the column names: unpivot, split, pivot back ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_piv_budget_matrix"]}[Data], Trimmed = Table.RemoveColumns(Source, {"row_id"}), Unpivoted = Table.UnpivotOtherColumns(Trimmed, {"department"}, "col", "amount"), Split = Table.SplitColumn(Unpivoted, "col", Splitter.SplitTextByDelimiter(" ", QuoteStyle.None), {"month", "measure"}), Pivoted = Table.Pivot(Split, List.Distinct(Split[measure]), "measure", "amount") in Pivoted ``` ### JSON documents: parse safely and expand ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_json_orders"]}[Data], Parsed = Table.AddColumn(Source, "doc", each try Json.Document([payload]) otherwise null), Valid = Table.SelectRows(Parsed, each [doc] <> null), WithItem = Table.AddColumn(Valid, "items", each [doc][items]), Expanded = Table.ExpandListColumn(WithItem, "items"), Lines = Table.ExpandRecordColumn(Expanded, "items", {"sku", "qty", "unit_price"}, {"sku", "qty", "unit_price"}) in Lines ``` About 1% of the documents are broken on purpose, so the `try ... otherwise` keeps the query from failing. Compare the rows you keep against the 1,500 you started with. ### Dates in many formats: type with a locale ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_dt_formats"]}[Data], Us = Table.TransformColumnTypes(Source, {{"us_text", type date}}, "en-US"), Eu = Table.TransformColumnTypes(Us, {{"eu_text", type date}}, "en-GB"), Serial = Table.AddColumn(Eu, "from_serial", each try Date.From(Number.FromText([serial_text])) otherwise null, type date), Epoch = Table.AddColumn(Serial, "from_epoch", each try #datetime(1970, 1, 1, 0, 0, 0) + #duration(0, 0, 0, Number.FromText([epoch_text])) otherwise null, type datetime) in Epoch ``` Compare each result with `date_true`. The same text, `03/04/2026`, is March 4 in `en-US` and April 3 in `en-GB`, which is why the locale matters. ### Append many same-shaped tables ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"), Parts = Table.SelectRows(Source, each Text.StartsWith([Item], "lab_app_web_2026_")), Tidy = Table.AddColumn(Parts, "T", each Table.TransformColumnNames([Data], each Text.Replace(Text.Lower(_), " ", "_"))), Combined = Table.Combine(Tidy[T]) in Combined ``` The five monthly tables differ on purpose: one has an extra column, one has spaced column names and one stores the total as text. Fix the totals after combining. ### Fuzzy merge company names to a master list ```m let Pqlab = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"), Variants = Pqlab{[Schema = "dbo", Item = "lab_mrg_company_variants"]}[Data], Master = Pqlab{[Schema = "dbo", Item = "lab_mrg_company_master"]}[Data], Joined = Table.FuzzyNestedJoin(Variants, {"company_raw"}, Master, {"company_name"}, "match", JoinKind.LeftOuter, [IgnoreCase = true, IgnoreSpace = true, Threshold = 0.8]), Result = Table.ExpandTableColumn(Joined, "match", {"company_id", "company_name"}) in Result ``` Forty variants belong to no master company. They are the ones that stay unmatched at a high threshold. ### A custom function over a list of API pages First create a query named `fnPage`: ```m (page as number) as table => let Pages = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_json_api_pages"]}[Data], Row = Table.SelectRows(Pages, each [page_number] = page){0}, Doc = Json.Document(Row[payload]), Items = Table.FromList(Doc[results], Splitter.SplitByNothing(), {"item"}) in Items ``` Then, in a second query, invoke it for every page: ```m let Pages = List.Numbers(1, 40), AsTbl = Table.FromList(Pages, Splitter.SplitByNothing(), {"page"}), Called = Table.AddColumn(AsTbl, "data", each fnPage([page])), All = Table.ExpandTableColumn(Called, "data", {"item"}) in All ``` ### Coordinates in nine text formats: classify, then parse ```m let Source = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"){[Schema = "dbo", Item = "lab_geo_coordinates_text"]}[Data], Clean = Table.TransformColumns(Source, {{"coordinate_text", each Text.Trim(Text.Replace(Text.Replace(_, "(", ""), ")", "")), type text}}), Kind = Table.AddColumn(Clean, "style", each if Text.Contains([coordinate_text], "°") then "dms" else if Text.Contains([coordinate_text], ";") then "european" else if Text.StartsWith([coordinate_text], "N ") or Text.StartsWith([coordinate_text], "S ") then "ddm" else if Text.Contains([coordinate_text], "N,") or Text.Contains([coordinate_text], "S,") then "hemisphere" else "decimal", type text) in Kind ``` Parse each style on its own (split on the delimiter, replace the decimal comma, convert degrees-minutes-seconds with a small function), then compare with `lat_true` and `lon_true`. Some "decimal" rows have longitude first: a latitude must be between -90 and 90, so a value beyond that means the pair is swapped. ### Geocode an address by merging to a reference table ```m let Pqlab = Sql.Database("learnsql.cpetoday.com,1433", "pqlab"), Addr = Pqlab{[Schema = "dbo", Item = "lab_geo_addresses_to_geocode"]}[Data], Zips = Pqlab{[Schema = "dbo", Item = "core_geo_postal_codes"]}[Data], WithZip = Table.AddColumn(Addr, "zip", each try Text.Select(Text.End([raw_address], 6), {"0".."9"}) otherwise null, type text), Joined = Table.NestedJoin(WithZip, {"zip"}, Zips, {"postal_code"}, "geo", JoinKind.LeftOuter), Result = Table.ExpandTableColumn(Joined, "geo", {"latitude", "longitude"}, {"lat_found", "lon_found"}) in Result ``` Rows in the "no zip" style have no postal code, so they stay unmatched; fall back to a merge on city and state with `core_geo_places`. Measure how far each guess is from `lat_true` and `lon_true`. ### Create a custom data type (Excel) In the Power Query editor, load `lab_type_trails` (or `core_products`, `core_employees`, `core_stores`), select the columns that should become fields, choose **Transform > Create Data Type**, and pick a display column such as `trail_name`. After you load it to a worksheet, you can read a field with a formula such as `=A2.Rating`. ## Full catalog The same list is in the `lab_catalog` table, which you can query and shape in Power Query. Each row says what the table is and which techniques it teaches. ### Core backbone | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `core_categories` | 58 | Beginner | Self-referencing hierarchy (parent_category_id); Merge to self; Split path; Fill down levels | Three-level product category tree. | | `core_suppliers` | 25 | Beginner | Merge; Group by; Filter; Conditional column | Supplier list with lead times and quality ratings. | | `core_promotions` | 24 | Beginner | Merge; Date range filter; Conditional column | Promotion codes with discount and date window. | | `core_products` | 220 | Beginner | Create data type; Merge to categories; Add custom column (margin); Conditional column; Image URLs; Dates | 220 products with cost, price, launch/discontinue dates, image and page URLs. A few trailing spaces and missing weights/ratings. | | `core_stores` | 9 | Beginner | Create data type (location); Merge; Latitude/longitude; Filter nulls | Nine locations including an online store (no address) and a distribution center. | | `core_employees` | 66 | Intermediate | Self-merge for manager name; Hierarchy flatten; Create data type; Date arithmetic (tenure); Conditional column | Staff with manager_id, store, hire/termination dates and pay type. | | `core_customers` | 1,545 | Intermediate | Remove duplicates (~45 near-duplicate customers); Clean state variants; Trim/Case; Null handling; Age from birth_date; Group by tier | 1,500 customers. Mostly clean, but ~3% state spellings vary, tier casing varies, some emails/phones are missing and ~45 customers are duplicates. | | `core_orders` | 14,000 | Beginner | Merge to customers/stores/employees; Group by month/channel; Date parts; Conditional column; Query folding on a 14k-row table | 14,000 orders from 2023-01 to 2026-10 with channel, status, payment method, promo and totals (all USD). | | `core_order_lines` | 27,933 | Beginner | Merge to products; Group by order; Custom column (net price); Large-table folding; Index/rank within order | About 36,000 order lines with quantity, unit price and discount. | | `core_shipments` | 5,774 | Beginner | Merge; Date difference (ship days); Null delivered_date handling; Conditional column | Shipments for web and phone orders with carrier, dates and status. | | `core_returns` | 856 | Beginner | Merge to lines/orders/products; Group by reason; Percent of total | Returns with reason, condition and refund amount. | | `core_inventory` | 10,575 | Intermediate | Unpivot/pivot by snapshot date; Group by; Merge to products and stores; Filter latest snapshot | Quarter-end inventory snapshots by store and product. | | `core_calendar` | 2,191 | Beginner | Date dimension; Merge on date; Fiscal year (starts Feb 1) columns; Holiday flags | Calendar 2022-2027 with fiscal year, quarter, month and holiday names. | | `core_fiscal_calendar` | 2,191 | Intermediate | Fiscal year, quarter and period; 4-5-4 retail weeks; Merge on date; Map by fiscal period; Seasons | Daily calendar with the February-start fiscal year (named by the year it starts), fiscal weeks, a true 4-5-4 retail calendar and shopping seasons. | ### Headers & layout | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_hdr_title_rows` | 197 | Beginner | Remove top rows; Use first row as headers; Remove bottom rows; Remove blank rows; Filter out repeated header rows; Change type; Replace values ($ and commas); Remove columns | A pasted report export: title block, blank lines, repeated page headers, a total row and an end-of-report footer. Everything is text. | | `lab_hdr_first_row` | 301 | Beginner | Use first row as headers; Duplicate column names (Amount.1); Rename columns; Trim; Change type; Split column | The classic: the real headers are the first data row. Includes a duplicate header, stray spaces and mixed case. | | `lab_hdr_two_row` | 10 | Intermediate | Fill right (year across quarters); Transpose; Merge columns (Year-Quarter); Use first row as headers; Unpivot other columns; Split column by delimiter | A crosstab whose header is two rows: years on top (only in the first column of each group) and quarters below. | | `lab_hdr_blocks` | 129 | Intermediate | Add conditional column; Fill down; Filter rows; Remove blank rows; Remove duplicates (repeated headers); Extract text after delimiter | A grouped report: each store has a title line, its own header line, detail rows and a subtotal, all in the same columns. | | `lab_hdr_ledger` | 280 | Intermediate | Fill down (dates); Use first row as headers; Parentheses negatives; Merge Debit/Credit into one signed Amount; Replace nulls with 0; Change type with locale | A general-ledger style export: the date appears only on the first line of each day, debits and credits are in separate text columns, balances use parentheses for negatives. | | `lab_hdr_stacked` | 745 | Advanced | Add index column; Conditional column (record number); Split column by delimiter; Fill down; Pivot column (no aggregation); Remove blank rows | One text column holding 'Label: value' lines, one blank line between records. Records have different fields. | | `lab_hdr_alternating` | 200 | Intermediate | Add index column; Modulo / Integer-divide index; Pivot column; Keep/Remove alternate rows; Merge queries (odd with even) | Records stacked on alternating lines: name, then phone, then the next name, and so on. | ### Pivot & unpivot | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_piv_sales_wide` | 1,920 | Beginner | Unpivot columns; Unpivot other columns; Rename attribute to Month; Replace nulls; Sort; Merge with calendar; Remove total column | Twelve month columns per product, region and year (2026 is year-to-date, so late months are null). A ready-made unpivot exercise. | | `lab_piv_sales_long` | 4,719 | Beginner | Pivot column (Sum / Don't aggregate); Group by; Add column from examples (quarter); Merge columns; Percent of total | Tall, tidy data: category x region x channel x month. Good for building a matrix (months across, categories down). | | `lab_piv_attributes` | 1,083 | Intermediate | Pivot column (Don't aggregate); Replace values (Colour/color); Clean names first; Change type per column; Conditional columns | Entity-attribute-value: one row per product attribute, with inconsistent attribute names (Color / Colour / color). | | `lab_piv_kpi_rows` | 12 | Intermediate | Transpose; Use first row as headers; Unpivot columns; Replace values (n/a, TBD); Change type with mixed formats; Add custom column | A finance-style scorecard: metrics down the side, fiscal years across the top, values in mixed text formats ($41.2M, 41,200,000, 44.3%). | | `lab_piv_survey_wide` | 800 | Intermediate | Unpivot other columns; Split column by delimiter (q9_hiking -> hiking); Replace values (Likert to 1-5); Count rows / Group by; Pivot to cross-tab | Survey export with Likert text answers (q1-q8) and one Yes/blank column per checkbox option (q9_*). | | `lab_piv_tags` | 1,545 | Beginner | Split column by delimiter into rows; Trim; Clean; Capitalize each word; Remove duplicates; Group by (count); Pivot | Multi-value columns: interests separated by ';' (sometimes '; ' or ','), purchase categories by ', '. Casing is inconsistent. | | `lab_piv_budget_matrix` | 15 | Advanced | Unpivot other columns; Split column by delimiter (Month / Measure); Pivot column back to Budget and Actual; Add column (variance); Date from month name | Budget and actual side by side for every month: 24 columns named like 'Jan Budget' and 'Jan Actual'. | ### Text | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_txt_contacts` | 1,200 | Beginner | Trim; Clean (tabs); Replace non-breaking spaces; Split column by delimiter (name, address); Proper case; Extract (first/last chars, length); Replace values; Add column from examples (area code); Pad zip codes with zeros; Filter invalid emails | A messy contact list: seven name styles, ten phone formats, mixed-case emails with spaces and typos, one-string addresses, state spellings, lost leading zeros on ZIPs, tabs and non-breaking spaces. | | `lab_txt_sku_parse` | 800 | Beginner | Split column by delimiter (SKU parts); Extract text between delimiters; Text before/after delimiter; First/last N characters; Length; Upper/Lower/Proper; Column from examples; Sort (natural order trap: Item 10 before Item 2) | Coded identifiers to take apart: SKUs with category, sequence, size and color segments (plus lowercase, underscore and padded variants), 'size / color' strings, titles in assorted casing. | | `lab_txt_html` | 120 | Advanced | Html.Table with CSS selectors; Text.Replace of entities; Extract text between delimiters; Split rows from
  • ; Parse HTML table into rows | Product-detail HTML fragments: a name, price span, description with
    , a feature list, a specs table and a link, with & entities. | | `lab_txt_logs` | 5,000 | Advanced | Split column by delimiter; Extract between delimiters; Json.Document on JSON lines; Conditional parsing (try ... otherwise); Parse key=value pairs; Datetime.FromText; Filter rows (contains); Group by hour/level | Five thousand application log lines in four different formats: key=value, JSON lines, syslog style and free text. | | `lab_txt_addresses` | 600 | Intermediate | Split column by delimiter (comma, pipe, line feed); Split by number of characters; Extract text after last delimiter; Replace line feeds; Conditional column to detect PO boxes; Trim; Standardize country | Free-form address blobs: line-feed separated, comma separated, pipe separated, PO boxes and ALL CAPS, with five spellings of USA. | | `lab_txt_encoded` | 300 | Advanced | Binary.FromText (Base64); Text.FromBinary; Uri.Parts / Uri.BuildQueryString; Split query string by & then =; Text.Replace for HTML entities; Percent-decoding | Encoded text to decode: Base64 notes, URL query strings with %20, URLs with parameters and HTML-encoded strings. | | `lab_txt_names_accents` | 400 | Intermediate | Proper-case traps (McDonald, O'Neil, van der Berg); Text.Normalize / remove diacritics via Replace; Split on spaces and hyphens; Trim; Detect non-ASCII with Text.Select; Unicode data types | International names with accents, apostrophes, hyphens, particles (van der, de la) and mixed casing. | ### JSON & XML | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_json_orders` | 1,500 | Advanced | Json.Document; Expand record columns; Expand list to rows; Nested expand (items > attributes); Optional fields (MissingField.UseNull); try ... otherwise for malformed JSON; Type conversion after expand; Column from record | Order documents as an API would deliver them: nested customer (with optional loyalty), an items array with nested attributes, shipping that is null for store sales, a few missing keys, string-typed quantities and about 1% malformed JSON. | | `lab_json_product_specs` | 220 | Intermediate | Json.Document; Expand record with different fields per row; Record.FieldNames to discover columns; Expand list to rows; Combine lists (Text.Combine); Dynamic column expansion | One JSON object per product whose keys differ by category (tents have capacity and dimensions, boots have sizes and drop, bags have temperature rating). | | `lab_xml_shipments` | 400 | Advanced | Xml.Tables / Xml.Document; Expand tables with namespaces; Attributes vs elements; Expand nested tables (packages > items); Combine events; Type conversion | Shipment XML with a namespace, attributes (weight_kg, qty), repeating packages and events. | | `lab_json_api_pages` | 40 | Advanced | Custom function over a list of pages; Invoke custom function; List.Generate / List.Accumulate for pagination; Expand list; next_url loop stop condition; Combine results | Forty pages of a paginated API response. Each row holds one page's JSON, its URL and the next URL (null on the last page). | ### Dates & times | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_dt_formats` | 600 | Beginner | Change type using locale; Date.FromText; Excel serial to date (Date.From of number); Unix epoch to datetime; DateTimeZone.FromText; Time and Duration parsing; Extract year/quarter/month/day/week/day name; Age and Duration.Days; Replace errors | The same date in eleven text formats (ISO, US, EU, long, short, compact, Excel serial, Unix epoch, ISO with offset), plus time and duration strings. date_true is the answer key. | | `lab_dt_events_tz` | 800 | Advanced | DateTimeZone.FromText; DateTimeZone.SwitchZone / ToLocal; Parse offsets; Duration between start and end; Time zone abbreviations (PT, ET) as text; Datetime to Date and Time; Duration.TotalMinutes | Event start and end times written four different ways (local with a zone abbreviation, ISO with offset, UTC 'Z', US 12-hour) with an IANA zone name and the true UTC start as the answer key. | | `lab_dt_sensor` | 6,604 | Intermediate | Group by (device, date / hour); Fill down / fill up; Remove duplicates; Replace values (-999 sentinel); Replace errors; Sort; Index column for rolling windows; Date/Time split; Detect gaps; Percent-style text to number | Fifteen-minute readings from five sensors over 14 days with gaps, duplicate rows, out-of-order delivery, sentinel values (-999), blanks, and humidity stored as text ('45%', '45.2', '--'). | | `lab_dt_fx_rates` | 3,891 | Intermediate | Merge queries on two keys (date + currency); Fill down for weekends/holidays; Generate a date list (List.Dates) and left-join; Pivot currency to columns; Custom function for latest-rate lookup | Daily FX rates (USD per unit) for CAD, EUR, GBP and JPY. Weekdays only, plus a few random missing days, so a naive join leaves weekend orders without a rate. | | `lab_dt_fiscal_periods` | 72 | Intermediate | Merge on a date range (custom column filtering period_start <= date <= period_end); Table.SelectRows inside a custom column; Expand; Cross join; Date arithmetic | Fiscal periods (the fiscal year starts in February) with start and end dates, for range-based lookups. | ### Numbers | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_num_formats` | 800 | Intermediate | Change type using locale (en-US / de-DE / fr-FR); Replace values ($ , parentheses, trailing minus, currency words); Percent text to number; Abbreviations (K/M/B) via conditional column; Scientific notation text; Number.FromText with culture; Replace errors; Round / Number.Mod | Money, quantities and percentages written every way a spreadsheet or a European colleague might: $1,234.50, 1.234,50 €, (1,234.50), 1,234.50-, 1.2K, 12%, 8.50E+11 UPCs. value_true is the answer key. | | `lab_num_scores` | 560 | Beginner | Group by (sum, average, median, std dev, count distinct, all rows); Rank / Index within group; Percent of total; Add column (attainment %); Number bins via conditional column; Statistics (Transform > Statistics); Running total | Sales reps by quarter with sales, quota, calls and deals. Built for aggregation, ranking, attainment %, and percentile or bucket work. | ### Append | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_app_orders_2023` | 3,391 | Beginner | Append queries; Rename columns to match; Change type; Table.Combine across a list of tables | Order summary for 2023: the oldest layout (cust_id, total). | | `lab_app_orders_2024` | 3,550 | Intermediate | Append with mismatched column names; Rename; Dates and money stored as text; Change type with locale | 2024 orders: column names with spaces and camel case (Order ID, CustomerID), dates as MM/DD/YYYY text, totals as '$1,234.50' text. | | `lab_app_orders_2025` | 3,941 | Intermediate | Append; Added columns (currency); Currency conversion with lab_dt_fx_rates; Merge on date + currency | 2025 orders with a currency column; about 6% of rows are billed in CAD or EUR and the amount is in that currency. | | `lab_app_orders_2026` | 3,118 | Intermediate | Append with uppercase names and different column order; Reorder columns; Rename; Table.Combine with MissingField handling | 2026 year-to-date orders: UPPER_CASE column names, columns in a different order and an extra STATUS column. | | `lab_app_web_2026_01` | 247 | Intermediate | Navigation table filter on name (Text.StartsWith); Table.Combine over many tables; Custom function to normalize each table; Missing/extra columns; Rename drift; Type drift | Web-store orders for 2026-01. One of several same-shaped monthly tables. | | `lab_app_web_2026_02` | 281 | Intermediate | Navigation table filter on name (Text.StartsWith); Table.Combine over many tables; Custom function to normalize each table; Missing/extra columns; Rename drift; Type drift | Web-store orders for 2026-02. One of several same-shaped monthly tables. | | `lab_app_web_2026_03` | 236 | Advanced | Navigation table filter on name (Text.StartsWith); Table.Combine over many tables; Custom function to normalize each table; Missing/extra columns; Rename drift; Type drift | Web-store orders for 2026-03. One of several same-shaped monthly tables (this one has an extra coupon_code column). | | `lab_app_web_2026_04` | 274 | Advanced | Navigation table filter on name (Text.StartsWith); Table.Combine over many tables; Custom function to normalize each table; Missing/extra columns; Rename drift; Type drift | Web-store orders for 2026-04. One of several same-shaped monthly tables (this one has spaced, title-case column names). | | `lab_app_web_2026_05` | 324 | Advanced | Navigation table filter on name (Text.StartsWith); Table.Combine over many tables; Custom function to normalize each table; Missing/extra columns; Rename drift; Type drift | Web-store orders for 2026-05. One of several same-shaped monthly tables (this one stores order_total as text). | ### Merge & join | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_mrg_customers_crm` | 800 | Intermediate | Merge queries (Inner, Left outer, Right outer, Full outer, Left anti, Right anti); Merge on email; Trim/lowercase keys before merging; Expand merged column | A CRM customer list (800 customers). About 560 also exist in lab_mrg_customers_billing; 240 CRM customers have no billing account. | | `lab_mrg_customers_billing` | 760 | Intermediate | Fuzzy merge (similarity threshold, transformation table); Merge on cleaned email; Right anti; Full outer to find mismatches; Group by to find duplicates | A billing system's accounts: names in different styles with occasional typos, emails in mixed case or with spaces, some changed emails. 200 accounts are not in the CRM. | | `lab_mrg_company_variants` | 502 | Advanced | Fuzzy grouping (Group By with fuzzy matching); Fuzzy merge to master list; Transformation table; Similarity threshold; Trim/Clean/Case before fuzzy; Remove duplicates | About 500 company names that are really ~120 companies: typos, swapped letters, suffix variants (Inc / Incorporated), casing and extra spaces. Forty don't belong to any master company. | | `lab_mrg_company_master` | 120 | Intermediate | Fuzzy merge target; Lookup table; Merge and expand; Left outer join to find unmatched | The clean list of 120 companies to match the variants against. | | `lab_mrg_state_aliases` | 268 | Intermediate | Replace values using a lookup table (Merge instead of 40 Replace steps); Trim; Case-insensitive merge; List.ReplaceMatchingItems; Fix state spelling variants | Every spelling of a state seen in the wild (WA, wa, Washington, Wash., Washinton) mapped to its two-letter code. | | `lab_mrg_state_lookup` | 50 | Beginner | Merge (left outer) to add state name and region; Lookup table; Expand; Group by region | Fifty states with region and approximate sales tax rate. | | `lab_mrg_sales_textkeys` | 2,000 | Intermediate | Merge on keys of different types (text with leading zeros vs integer); Text.PadStart / Number.FromText; Extract number from 'C-00123'; Left anti join to find orphan keys; Date from compact text | Sales whose keys are text: product_key '0042' and customer_key 'C-00123'. About 5% point at products that don't exist; some keys are null. | | `lab_mrg_price_history` | 782 | Advanced | Range join (date between effective_from and effective_to); Custom column with Table.SelectRows; Open-ended ranges (null effective_to); Merge then filter; Slowly changing dimension | Price changes per product with effective_from / effective_to. The current price has a null end date. | | `lab_mrg_commission_tiers` | 5 | Intermediate | Approximate (range) lookup; Cross join then filter; Custom column over a table; Conditional column alternative; Last-row lookup with List.Max | Commission tiers with minimum and maximum sales (open-ended top tier). | | `lab_mrg_sizes` | 7 | Beginner | Cross join (Custom column holding another table, then expand); Generate all combinations; Sort order lookup; Merge | Seven apparel sizes with a sort order. | | `lab_mrg_colors` | 10 | Beginner | Cross join; Merge; Add custom column from hex code; Data type color text | Ten product colors with codes and hex values. | ### Duplicates | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_dup_contacts` | 900 | Beginner | Remove duplicates (exact vs after cleaning); Keep duplicates; Trim/Lowercase before dedupe; Group by (count) to find duplicates; Sort then Remove Duplicates (keep newest); Table.Buffer | 900 contact rows for 600 people: 300 duplicates that differ only by email casing/spacing, first-name casing or initials, source and created date. | | `lab_dup_customer_versions` | 3,340 | Intermediate | Group by customer with Max(updated_at) then merge back; Sort descending + Remove Duplicates on customer_id (keep latest, with Table.Buffer); Rank within group; Index; Keep top N rows per group (all rows); Compare previous row | Up to five versions per customer (history of changes), delivered in random order. Pick the current record, or compare consecutive versions. | ### Functions & parameters | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_fn_parameters` | 12 | Intermediate | Parameters table pattern; Drill down to a single value (Table.SelectRows then {0}[param_value]); Convert text to typed parameter; Reference queries; Dynamic filter values | Report settings stored as name / value / type rows. Drive your queries from here instead of hard-coding filters. | | `lab_fn_report_requests` | 40 | Advanced | Create a custom function; Invoke custom function per row; Pass columns as parameters; Group and combine results; Table.AddColumn with function call; Error handling per row | A work list of report requests (store, month, report type). Run a function once per row to build each report. | ### Hierarchy | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_hier_bom` | 33 | Advanced | Recursive expansion with a custom function or List.Generate; Self-merge on parent/child; Flatten to levels; Table.Buffer; Quantity roll-up (multiply down the tree); Detect leaves vs assemblies | A bill of materials: kits contain sub-assemblies, which contain products (up to three levels). Component can itself be an assembly. | | `lab_hier_category_paths` | 58 | Beginner | Split column by delimiter into columns (uneven depth); Extract last level; Count levels (Text.Split + List.Count); Fill down parent levels; Trim; Handle '>' with and without spaces | Category breadcrumbs ('Camping > Tents > Backpacking Tents') of different depths, some written without spaces around the arrow. | ### Errors & nulls | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_err_null_markers` | 1,000 | Beginner | Replace values (many null markers to null); Replace errors; Remove errors; Remove empty; Column quality / distribution / profile; Fill down/up; Change type with error handling; Keep rows with errors | Thirteen different ways of saying 'no value' (NULL, n/a, #N/A, '-', blanks) across eight columns, plus junk in numeric and date columns ('ERR', '#DIV/0!', 0000-00-00). | | `lab_err_mixed_types` | 800 | Intermediate | Change type errors; Conditional column to detect type (try Number.From otherwise ...); Replace values for booleans (Y/yes/TRUE/1); Remove errors; Number.From with culture; Detect with Value.Is | One column holding integers, decimals, words, dates, booleans, blanks and thousands-separated numbers; plus boolean and integer columns in many spellings. | | `lab_err_wide` | 300 | Beginner | Choose columns; Remove other columns; Rename columns (clean names, remove symbols); Reorder columns; Table.TransformColumnNames; Duplicate column; Data profiling on wide tables; Promote/demote headers | A 40-column extract with ugly column names (double spaces, trailing spaces, #, $, /, parentheses, dashes) to practice column management. | | `lab_err_impossible_values` | 600 | Intermediate | Filter rows (between, greater than, equals); Replace values with null; Conditional column for flags; Keep rows / Remove rows; Date sentinel handling; Statistics for outliers; Group by to count problems; Add column (is_valid) | Sales rows with logically impossible values: negative or huge quantities, zero and negative prices, discounts over 100%, ship dates before order dates, sentinel dates (1900, 2099, 9999). | | `lab_err_category_variants` | 1,000 | Intermediate | Replace values; Trim/Clean/Capitalize Each Word; Group by to find distinct spellings; Fuzzy grouping; Merge to a mapping table; Conditional columns; List.Distinct; Column distribution | Five categorical columns each spelled several ways (casing, spaces, typos, abbreviations like CC/Visa). | ### Geospatial | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `core_geo_states` | 51 | Beginner | Merge to customers and stores; Map by state; Region grouping; Filled map | The 50 states and DC with FIPS codes, Census region and division, centroid coordinates and capital. | | `core_geo_places` | 29 | Beginner | Merge on city and state; Bubble map; County and metro rollups; Time zones | Cities used in the data with county, centroid coordinates, population, time zone and metro area. | | `core_geo_postal_codes` | 895 | Intermediate | Merge on postal code; Postal-centroid maps; Fix leading zeros first | A centroid for every postal code in core_customers and core_stores. The codes are fictitious (a real city prefix plus random digits), so centroids are synthetic. | | `lab_geo_coordinates_text` | 600 | Intermediate | Split column by delimiter; Parse degrees-minutes-seconds with a custom function; Detect swapped latitude/longitude with a range check; Replace decimal commas; Trim; Extract numbers; Conditional column by format | Coordinates written nine ways (decimal, swapped, DMS, degrees-decimal-minutes, with N/W letters, European commas, padded, in parentheses). lat_true and lon_true are the answer key. | | `lab_geo_addresses_to_geocode` | 800 | Intermediate | Standardize addresses (case, abbreviations, suites); Split into street, city, state, ZIP; Merge to core_geo_postal_codes or core_geo_places to geocode; Compare to the answer key; Fuzzy merge on street with typos | Eight hundred customer addresses in six messy styles (lowercase, spelled-out, no ZIP, suite added, typo), each with the true coordinates and quality. | | `lab_geo_route_points` | 6,989 | Advanced | Sort by route and sequence; Index within group; Distance between consecutive points with a custom function (haversine); Remove duplicates; Detect GPS glitches (impossible jumps); Group by route (total miles, duration, stops); Fill gaps; Lines on a map | GPS breadcrumbs for 40 delivery routes from the Kent distribution center, with dropped points, duplicate timestamps and a few GPS glitches. Stops carry the customer served. | | `lab_geo_distance_pairs` | 2,100 | Intermediate | Cross join stores and customers; Group by and keep the minimum (nearest store); Rank within group; Distance bands via conditional column; Catchment analysis; Lines from origin to destination on a map | Every physical store paired with 300 customers: straight-line miles, estimated drive miles and minutes, and a flag for each customer's nearest store. | ### Custom data types | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_type_trails` | 120 | Intermediate | Create data type (Excel: Transform > Create Data Type); Record columns; Display column choice; Linked data type fields (latitude/longitude, url, image); Table.ToRecords; Add column from record fields | Trailheads with name, park, location, ratings, URL and image: a natural 'card' entity for an Excel data type. | ### Index | Table | Rows | Level | Techniques | What it is | |---|---:|---|---|---| | `lab_catalog` | 86 | Beginner | Split column by delimiter into rows (techniques); Group by category; Filter by difficulty; Count distinct techniques | Index of all tables: category, difficulty, techniques (';' separated), description, row count and how to use each table. | --- # IoT and smart home (smarthome) `smarthome` is the data of one connected family home, the Rivera household in Bellevue, WA: two adults, a teen, a child and a dog, with 83 devices in 12 rooms, rooftop solar, a home battery and an electric car. It covers 24 September 2025 to 24 October 2026 and holds about 1.9 million rows in 39 tables, identical on SQL Server and MariaDB. It is built for **personal household data**: sensor time series, events, energy, bills, weather, security and routines, with a geospatial layer (the home, the places the family drives to, GPS trips and a floor plan) and a fiscal calendar (the October to September "water year" that utilities use). ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `smarthome`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** open [the database page](/databases/smarthome) for the Playground, the SQL editor and the schema explorer. - **Time:** every timestamp is the home's local *standard* time all year, so there are no daylight-saving gaps or repeats. The as-of time is 2026-10-24 12:00. ## Everything is connected The data is generated so that the numbers agree with each other: - **Weather drives the house.** Outdoor temperature runs the heat pump (heating through winter, cooling in the July heat wave) and sunlight drives the solar array. - **People drive the rest.** Who is home, and in which room, determines lights, motion, CO2, humidity, hot-water use and appliance cycles. - **Energy reconciles.** Grid import plus solar plus battery discharge equals grid export plus household load plus battery charge. The hourly meters are integrated from the 15-minute power channels. - **Bills are priced from the meters.** The electric bill applies the time-of-use rates to the hourly import and export, and the gas and water bills come from the gas and water meters. ## Tables at a glance | Area | Tables | What they hold | |---|---|---| | Home | `homes`, `rooms`, `people` | One home with a full address, 12 rooms with floor-plan rectangles, and five household members (the dog included) | | Devices | `device_types`, `device_models`, `devices`, `sensor_channels`, `energy_meters` | 83 devices (thermostats, sensors, lights, locks, cameras, appliances, energy gear), their models and positions, 79 sensor channels, 8 energy meters | | Readings and events | `readings`, `events`, `commands` | 1.6 million readings every 15 to 60 minutes, 153,000 events (motion, doors, locks, lights, appliance cycles), 10,800 commands | | Automations | `automations`, `automation_runs` | 17 automations and 6,200 runs with their outcomes | | Alerts and upkeep | `alerts`, `firmware_updates`, `maintenance_log` | 98 alerts, 146 firmware updates, battery and filter changes | | Energy and bills | `energy_hourly`, `device_energy_daily`, `ev_charging_sessions`, `rate_plans`, `utility_bills` | Hourly meters, daily energy per appliance, 167 EV sessions, time-of-use rate plans, electric, gas, water and internet bills | | Weather | `weather_hourly`, `weather_daily` | Hourly and daily weather with sunrise and sunset | | People | `presence_log`, `routines`, `sleep_sessions`, `access_codes` | Home and away history, routines, 1,500 nights of sleep with the bedroom's temperature and CO2, door codes for family, guests and services | | Security | `camera_clips`, `alarm_sessions` | 3,800 camera clips and 580 alarm sessions | | Places and trips | `points_of_interest`, `geofences`, `trips`, `trip_points` | 11 places, 7 geofences, 704 EV trips and their GPS breadcrumbs | | Reference | `geo_states`, `geo_places`, `geo_postal_codes`, `calendar_dates`, `fiscal_calendar`, `dataset_info` | Geography and calendars | ## How the tables join ``` homes ──< rooms ──< devices >── device_models >── device_types │ ├──< sensor_channels ──< readings (the big time series) ├──< events (motion, doors, locks, lights, cycles) ├──< commands >── automation_runs >── automations ├──< alerts, firmware_updates, maintenance_log └──< camera_clips, alarm_sessions, ev_charging_sessions energy_meters ──< energy_hourly devices ──< device_energy_daily people ──< presence_log, sleep_sessions, routines people ──< trips ──< trip_points points_of_interest ──< geofences fiscal_calendar / calendar_dates join on the date ``` ## Rules built into the data - **Readings have a quality flag.** About 1% of values are missing, about 0.3% are `suspect` (spikes, zeros and `-999`), and some are `gap-filled`. Filter on `quality` before you average. - **Different channels have different rhythms.** Most channels report every 15 minutes, four report every 30, soil moisture and gas every hour, and battery levels once a day (see `sensor_channels.sample_interval_s`). - **Grid power is signed.** In `grid_power_w`, positive means buying from the utility and negative means selling to it. - **Fiscal year.** The fiscal year is October to September, named by the year it ends, so FY2026 is October 2025 to September 2026 and fiscal period 1 is October. - **Rate change.** On 1 April 2026 the time-of-use rates rise and the export credit falls (see `rate_plans`). ## Things that happen in the data - A **heat wave** in late July and early August 2026, a **cold snap** in mid-January 2026, and **wildfire smoke** on 10 to 13 September 2026. - A **windstorm outage** on 14 January 2026 (03:10 to 09:40): the battery carries the house and the freezer warms to 14F. - A **laundry-room leak** on 17 February 2026: the automatic shutoff fails because the valve was offline, and Jordan closes it by hand 29 minutes later. - A **thermostat offline** for two days in March, batteries that run low and are replaced, and a few firmware updates that fail. - A **family vacation** from 10 to 24 July 2026 with the house empty, a Thanksgiving trip, grandparents at Christmas, and three business trips. ## Questions to try **How much of the house's electricity comes from the sun, month by month?** ```sql SELECT fiscal_period, ROUND(SUM(consumption_kwh), 0) AS used_kwh, ROUND(SUM(solar_kwh), 0) AS solar_kwh, ROUND(SUM(grid_import_kwh), 0) AS imported_kwh, ROUND(SUM(grid_export_kwh), 0) AS exported_kwh, ROUND((1 - SUM(grid_import_kwh) / SUM(consumption_kwh)) * 100, 1) AS self_sufficiency_pct FROM vw_daily_energy WHERE fiscal_year = 2026 GROUP BY fiscal_period ORDER BY fiscal_period; ``` Self-sufficiency is about 6% in January (fiscal period 4) and about 60% in June and July (periods 9 and 10). **What did each utility cost in fiscal 2026?** ```sql SELECT utility_type, COUNT(*) AS bills, ROUND(SUM(amount_due), 2) AS total_due FROM utility_bills WHERE fiscal_year = 2026 GROUP BY utility_type ORDER BY total_due DESC; ``` **Which rooms were warmest in July?** ```sql SELECT room_name, ROUND(AVG(avg_value), 1) AS avg_temp_f, ROUND(MIN(min_value), 1) AS coldest, ROUND(MAX(max_value), 1) AS warmest FROM vw_room_comfort_daily WHERE metric = 'temperature_f' AND reading_date >= '2026-07-01' AND reading_date < '2026-08-01' GROUP BY room_name ORDER BY avg_temp_f DESC; ``` **Does stuffy bedroom air hurt sleep?** ```sql SELECT CASE WHEN avg_bedroom_co2_ppm < 800 THEN 'under 800 ppm' WHEN avg_bedroom_co2_ppm < 1100 THEN '800-1,100 ppm' ELSE 'over 1,100 ppm' END AS co2_band, COUNT(*) AS nights, ROUND(AVG(sleep_score * 1.0), 1) AS avg_sleep_score FROM sleep_sessions GROUP BY CASE WHEN avg_bedroom_co2_ppm < 800 THEN 'under 800 ppm' WHEN avg_bedroom_co2_ppm < 1100 THEN '800-1,100 ppm' ELSE 'over 1,100 ppm' END ORDER BY avg_sleep_score DESC; ``` Nights with the highest CO2 score lower than nights with fresh air. **How far did the EV go, and for what?** ```sql SELECT purpose, COUNT(*) AS trips, ROUND(SUM(miles), 0) AS miles, ROUND(SUM(kwh_used), 0) AS kwh FROM trips WHERE fiscal_year = 2026 GROUP BY purpose ORDER BY miles DESC; ``` **Which alerts come up most?** ```sql SELECT alert_type, severity, COUNT(*) AS alerts FROM alerts GROUP BY alert_type, severity ORDER BY alerts DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` **Where is the motion?** Count motion events by room from the room sensors: ```sql SELECT r.room_name, COUNT(*) AS motion_events FROM events e JOIN devices d ON d.device_id = e.device_id JOIN rooms r ON r.room_id = d.room_id WHERE e.event_type = 'motion' AND e.fiscal_year = 2026 AND d.device_name LIKE 'Room Sensor%' GROUP BY r.room_name ORDER BY motion_events DESC OFFSET 0 ROWS FETCH FIRST 5 ROWS ONLY; ``` All of these run on SQL Server and MariaDB. More ideas: the outage hour by hour (`readings` for `battery_soc_pct` and `grid_power_w`), the leak minute by minute, empty-house energy from `presence_log`, battery health from the `battery_pct` channels, appliance cost per cycle (`device_energy_daily` with the rate plan), and a floor-plan heat map. ## Maps, floor plans and fiscal periods - **Floor plan.** `rooms` have `x_ft`, `y_ft`, `width_ft` and `depth_ft` and `devices` have `x_ft`, `y_ft` and `z_ft`. A scatter chart of `x_ft` against `y_ft`, sized by motion events, is a heat map of the house. - **Trips.** `trips` have origin and destination coordinates, and `trip_points` have the GPS breadcrumbs. A map with latitude and longitude, ordered by `point_seq`, draws each route. - **Fiscal year.** `fiscal_year`, `fiscal_quarter` and `fiscal_period` sit on `events`, `alerts`, `energy_hourly`, `device_energy_daily`, `weather_daily`, `utility_bills`, `ev_charging_sessions` and `trips`. See [maps and fiscal calendars](/analyze/maps-and-fiscal) for the shared pattern. ## Excel and Power BI - **Star model.** Use `readings`, `events` and `energy_hourly` as fact tables, with `devices`, `rooms`, `people` and a date table as dimensions. `fiscal_calendar` is a ready-made date table. - **Keep the big table light.** `readings` has 1.6 million rows. Load it filtered to the channels and months you need, or use the daily views. - **Measures to write.** Energy used, Solar produced, Self-sufficiency %, Net grid kWh, Cost per kWh, Average room temperature, Comfort hours, Automation success %. ## Realistic imperfections The data has the usual smart-home mess: missing and suspect readings, devices that drop off the network, failed automations and firmware updates, low batteries, duplicate-looking alerts, and a few sensors that are wrong in the same way for a day. It is clean enough to reconcile, and messy enough to need filtering. --- # City government operations (bigbear) `bigbear` is **municipal-style data**: the operating records of a city, in the way a city manager, finance director or public works superintendent would see them. The city is Big Bear Lake, California, a small mountain resort city in the San Bernardino Mountains at 6,750 feet. About 75 employees run streets, a municipal water utility and development services; police and fire are contracted; and the economy is built on visitors, second homes and short-term rentals. It covers fiscal 2022 through 24 October 2026 (the first months of fiscal 2027) in 61 tables and about 586,000 rows. It is built for **local-government analytics**: 311 requests, snow operations, dispatch, permits and code enforcement, a water utility, the lodging tax, property and parcels, fund accounting and council records, all on a map and on a July-June fiscal year. > **These are invented records.** The place, the seasons and the kind of work are real and recognizable, but every parcel, owner, employee, vendor, call, permit and dollar amount is made up. Neighborhoods and the lake road follow the real layout only approximately, street names are typical of the area but house numbers and lot lines are invented, and tax rates, fees, budgets and ballot measures are fictional. Use it for learning, not for navigation or for any question about a real property or the real city's finances. ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `bigbear`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** open [the database page](/databases/bigbear) for the Playground, the SQL editor and the schema explorer. - **Time:** timestamps are local time. The as-of date is 2026-10-24. ## One engine drives the city: visitors and weather `weather_daily` has a synthetic mountain climate and a `visitor_index` for every day (1.0 is an average day). Winter weekends, powder days, the summer season and holiday weeks run high; a blizzard or a road closure drops it. Almost everything else follows it, so the data behaves like a real tourist town: - **Visitors fill the rentals** (`tot_filings`), generate noise, trash and parking complaints (`service_requests`), dispatch calls (`cad_calls`) and water use in rentals (`utility_bills`). - **Snow runs the public works budget.** `storm_events` are derived from daily snowfall; storms drive `snow_operations`, storm overtime in `payroll_records`, plow breakdowns in `fleet_maintenance`, snow-related 311 requests, slower dispatch response, estimated water meter reads (buried meter boxes) and the emergency declaration behind disaster reimbursement. - **Freezes and thaws** cause main breaks and bursts in empty second homes. - **Season** drives potholes in spring and defensible-space work in summer. ## Tables at a glance | Area | Tables | What they hold | |---|---|---| | Geography and calendar | `neighborhoods`, `zoning_districts`, `fiscal_calendar`, `geo_states`, `geo_places`, `geo_postal_codes` | 12 neighborhoods in the city and 6 outside it, zoning, the July-June calendar with budget phases, tax installment dates and seasons | | Organization | `funds`, `departments`, `facilities` | 10 funds, 13 departments (two merged during the period), 32 facilities with addresses and coordinates | | People | `job_classes`, `positions`, `employees`, `pay_periods`, `payroll_records` | Authorized positions, 174 employees over the years (including seasonal snow crews), biweekly payroll with overtime split into storm overtime | | Governance | `council_members`, `council_meetings`, `agenda_items`, `council_votes` | Five seats with turnover, about 137 meetings, agenda items and every member's vote | | Finance | `gl_accounts`, `gl_journal_entries`, `gl_journal_lines`, `budgets`, `vendors`, `invoices`, `grants`, `capital_projects` | A balanced fund ledger, adopted and amended budgets, accounts payable, 12 grants and 29 capital projects | | Streets and property | `streets`, `street_condition_surveys`, `snow_plow_routes`, `property_owners`, `parcels`, `parcel_assessments`, `property_transfers` | 886 street segments with endpoints, pavement condition surveys, 24 plow routes, 8,643 parcels with owners, six years of assessed value and 2,000 transfers | | Assets | `asset_types`, `assets` | Hydrants, valves, streetlights, signals, drains, signs, restrooms and more, each with coordinates | | Water utility | `water_rates`, `utility_accounts`, `utility_bills` | Tiered rates by fiscal year, accounts that turn over when a home sells, 260,000 bimonthly bills | | Weather and snow | `weather_daily`, `storm_events`, `snow_operations`, `fleet_vehicles`, `fleet_maintenance` | Daily weather and visitors, 30 storms, plow shifts, the fleet | | 311 and work orders | `service_request_types`, `service_requests`, `work_orders` | 18,600 requests with channel and location, and the work orders they create plus preventive programs | | Dispatch | `cad_call_types`, `cad_units`, `cad_calls` | 30,000 Fire/EMS and Sheriff calls with every timestamp from receipt to clear | | Community development | `permit_types`, `permits`, `inspections`, `code_violation_types`, `code_cases` | 5,800 permits, 27,000 inspections, code cases from 311, patrols and defensible-space inspections | | Tourism economy | `short_term_rentals`, `tot_operators`, `tot_filings`, `businesses`, `business_licenses`, `special_events` | 1,400 rentals, lodging operators, 78,000 monthly tax filings, local businesses and licenses, festivals and parades | `dataset_info` holds the as-of date and a one-line description. Nine views save you the common joins: `vw_budget_vs_actual`, `vw_fund_balances`, `vw_response_times`, `vw_service_request_sla`, `vw_monthly_tot`, `vw_parcel_current`, `vw_permit_cycle_times`, `vw_water_billing_monthly` and `vw_storm_cost_summary`. ## How the tables join ``` neighborhoods ──< parcels >── zoning_districts property_owners ──< parcels ──< parcel_assessments │ └──< property_transfers streets ──< parcels streets ──< assets >── asset_types snow_plow_routes ──< streets │ parcels ──< utility_accounts ──< utility_bills parcels ──< short_term_rentals ──< tot_operators ──< tot_filings businesses ──< tot_operators, business_licenses parcels ──< permits ──< inspections parcels ──< code_cases ──< inspections violation types ──< code_cases service_request_types ──< service_requests ──< work_orders >── assets service_requests ──< code_cases cad_call_types ──< cad_calls >── cad_units storm_events ──< snow_operations, service_requests, work_orders funds, departments, gl_accounts ──< gl_journal_lines >── gl_journal_entries vendors ──< invoices >── capital_projects >── grants council_meetings ──< agenda_items ──< council_votes >── council_members employees ──< payroll_records >── pay_periods positions ──< employees fiscal_calendar joins on any date ``` ## Rules built into the data - **Fiscal year.** July 1 to June 30, named by the year it ends: FY2026 is July 2025 to June 2026 and fiscal period 1 is July. Fact tables carry `fiscal_year`, `fiscal_quarter` and `fiscal_period`. - **The ledger reconciles.** TOT revenue equals the TOT payments received, water sales equal the bills issued, permit and license revenue equal the fees paid, salaries and overtime equal payroll, and grant revenue equals `grants.reimbursed_amount`. Every journal entry belongs to one fund and balances; transfers between funds are a pair of entries. - **Property tax follows the roll.** Each parcel's assessed value grows 2 percent a year until a reassessing transfer resets it to the sale price (Proposition 13 style). `base_year` shows when it was last reset. - **One department history, two names.** Parks and Facilities merged into Public Works on 1 July 2024 and Building and Safety into Community Development on 1 July 2023. `departments.successor_department_id` links them, and `payroll_records.department_id` is the department on the pay date. - **Locations.** Parcels, assets, requests, calls, permits and cases carry `latitude`, `longitude` and `geocode_quality` (`rooftop`, `gps`, `street`, `address_match`, `neighborhood_centroid`, `failed`). About a third of dispatch calls are outside the city limits (`jurisdiction`, `neighborhoods.is_in_city`). ## Things that happen in the data - **A historic winter.** FY2023 is modeled as exceptionally heavy and FY2022 as dry. The largest storm, in February 2023, drops over 100 inches, closes roads, drives record overtime and 311 volume, and triggers an emergency declaration and a state reimbursement grant. - **A rate increase.** The TOT rate rises from 10 to 12 percent on 1 January 2025 (a modeled ballot measure). - **A drought surcharge** on water bills from June 2022 to April 2023, and a **smart-meter rollout** from late 2023 to spring 2026 that changes how reads are taken. - **A permit backlog** in spring and summer 2023 as storm repairs flood the counter, and a council fight over short-term rental rules in 2022 to 2026. - **Mergers and turnover:** the two department mergers, a council seat filled by appointment, and an employee base that turns over about 9 percent a year. ## Questions to try **Does a busier town mean more dispatch calls?** Group days by the visitor index: ```sql SELECT CASE WHEN w.visitor_index < 0.7 THEN '1 quiet (under 0.7)' WHEN w.visitor_index < 1.3 THEN '2 normal' ELSE '3 busy (1.3 and up)' END AS day_type, COUNT(*) AS days, ROUND(AVG(c.calls * 1.0), 1) AS avg_dispatch_calls, ROUND(AVG(w.visitor_index), 2) AS avg_visitor_index FROM (SELECT CAST(received_at AS DATE) AS call_date, COUNT(*) AS calls FROM cad_calls GROUP BY CAST(received_at AS DATE)) c JOIN weather_daily w ON w.weather_date = c.call_date GROUP BY CASE WHEN w.visitor_index < 0.7 THEN '1 quiet (under 0.7)' WHEN w.visitor_index < 1.3 THEN '2 normal' ELSE '3 busy (1.3 and up)' END ORDER BY day_type; ``` Busy days see roughly twice the calls of quiet days. **What does the lodging tax look like month by month?** The view already rolls up filings: ```sql SELECT period_month, SUM(filings) AS filings, ROUND(SUM(tax_due), 0) AS tax_due, ROUND(SUM(occupied_nights) * 100.0 / SUM(available_nights), 1) AS occupancy_pct FROM vw_monthly_tot WHERE fiscal_year = 2026 GROUP BY period_month ORDER BY period_month; ``` The winter holidays and the summer peak stand out; April and the fall are the troughs. **Where does the General Fund's money come from?** ```sql SELECT a.account_number, a.account_name, ROUND(SUM(l.credit - l.debit), 0) AS revenue FROM gl_journal_lines l JOIN gl_journal_entries e ON e.entry_id = l.entry_id JOIN gl_accounts a ON a.account_id = l.account_id WHERE e.fiscal_year = 2026 AND l.fund_id = 1 AND a.account_type = 'Revenue' AND a.account_number <> '4900' GROUP BY a.account_number, a.account_name ORDER BY revenue DESC; ``` The lodging tax is the largest single source, ahead of property tax and sales tax. **Did departments stay inside their budgets?** ```sql SELECT department_name, ROUND(SUM(adopted_amount), 0) AS adopted, ROUND(SUM(amended_amount), 0) AS amended, ROUND(SUM(actual_amount), 0) AS actual, ROUND(100.0 * SUM(actual_amount) / SUM(amended_amount), 1) AS pct_of_amended FROM vw_budget_vs_actual WHERE fiscal_year = 2026 AND fund_code = '001' AND account_type = 'Expense' GROUP BY department_name ORDER BY actual DESC; ``` **Who owns the homes?** Share of owner-occupied and absentee owners by neighborhood, with the active rentals: ```sql SELECT n.neighborhood_name, COUNT(*) AS homes, ROUND(100.0 * SUM(CASE WHEN p.is_owner_occupied = 1 THEN 1 ELSE 0 END) / COUNT(*), 1) AS owner_occupied_pct, ROUND(100.0 * SUM(CASE WHEN o.is_local = 0 THEN 1 ELSE 0 END) / COUNT(*), 1) AS absentee_owner_pct, SUM(CASE WHEN s.str_id IS NOT NULL THEN 1 ELSE 0 END) AS short_term_rentals FROM parcels p JOIN neighborhoods n ON n.neighborhood_id = p.neighborhood_id JOIN property_owners o ON o.owner_id = p.owner_id LEFT JOIN short_term_rentals s ON s.parcel_id = p.parcel_id AND s.status = 'Active' WHERE p.land_use IN ('Single Family Residence', 'Condominium', 'Townhome') GROUP BY n.neighborhood_name ORDER BY absentee_owner_pct DESC; ``` Resort areas such as Moonridge and the lakefront are mostly absentee; the residential hillsides are the most local. **Where do the owners live?** ```sql SELECT o.city AS owner_city, o.state AS owner_state, COUNT(*) AS parcels, ROUND(SUM(a.net_assessed_value) / 1000000.0, 1) AS assessed_value_millions FROM parcels p JOIN property_owners o ON o.owner_id = p.owner_id JOIN parcel_assessments a ON a.parcel_id = p.parcel_id AND a.fiscal_year = 2026 WHERE o.is_local = 0 GROUP BY o.city, o.state ORDER BY parcels DESC OFFSET 0 ROWS FETCH FIRST 8 ROWS ONLY; ``` **What did each big storm cost?** ```sql SELECT storm_name, severity, total_snowfall_in, plow_shifts, ROUND(overtime_hours, 0) AS overtime_hours, ROUND(abrasive_tons, 0) AS abrasive_tons, work_orders, ROUND(work_order_cost, 0) AS work_order_cost, service_requests FROM vw_storm_cost_summary ORDER BY total_snowfall_in DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` **How much of public works overtime is storm overtime?** ```sql SELECT d.department_name, p.fiscal_year, ROUND(SUM(p.overtime_pay), 0) AS overtime_pay, ROUND(100.0 * SUM(p.storm_overtime_hours) / NULLIF(SUM(p.hours_overtime), 0), 1) AS storm_share_pct FROM payroll_records p JOIN departments d ON d.department_id = p.department_id WHERE d.department_code IN ('PW', 'DWP', 'PRK') GROUP BY d.department_name, p.fiscal_year ORDER BY d.department_name, p.fiscal_year; ``` **How fast do police and fire respond, and what does snow do to it?** ```sql SELECT agency, CASE WHEN weather_snowfall_in >= 4 THEN 'snow day (4+ in)' ELSE 'otherwise' END AS conditions, COUNT(*) AS calls, ROUND(AVG(response_seconds) / 60.0, 1) AS avg_response_min, ROUND(100.0 * SUM(CASE WHEN response_seconds <= 480 THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_within_8_min FROM vw_response_times WHERE priority <= 2 AND response_seconds IS NOT NULL AND jurisdiction = 'City of Big Bear Lake' GROUP BY agency, CASE WHEN weather_snowfall_in >= 4 THEN 'snow day (4+ in)' ELSE 'otherwise' END ORDER BY agency, conditions; ``` **Which request types meet their service level?** ```sql SELECT category, COUNT(*) AS requests, ROUND(100.0 * SUM(met_sla) / COUNT(*), 1) AS pct_met_sla, ROUND(AVG(hours_to_close), 1) AS avg_hours_to_close FROM vw_service_request_sla WHERE fiscal_year = 2026 AND hours_to_close IS NOT NULL GROUP BY category ORDER BY requests DESC; ``` **Which short-term rentals draw the most noise complaints?** ```sql SELECT s.str_permit_number, p.address_line1, n.neighborhood_name, s.max_occupancy, COUNT(*) AS noise_complaints FROM service_requests q JOIN service_request_types t ON t.type_id = q.type_id JOIN parcels p ON p.parcel_id = q.parcel_id JOIN neighborhoods n ON n.neighborhood_id = p.neighborhood_id JOIN short_term_rentals s ON s.parcel_id = p.parcel_id WHERE t.type_name LIKE 'Noise complaint%' AND q.status <> 'Duplicate' GROUP BY s.str_permit_number, p.address_line1, n.neighborhood_name, s.max_occupancy ORDER BY noise_complaints DESC OFFSET 0 ROWS FETCH FIRST 8 ROWS ONLY; ``` **How long do permits take compared with the target?** ```sql SELECT type_name, COUNT(*) AS permits, ROUND(AVG(review_days * 1.0), 1) AS avg_review_days, MAX(review_days_target) AS target_days, ROUND(100.0 * SUM(CASE WHEN review_days <= review_days_target THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_on_target FROM vw_permit_cycle_times WHERE review_days IS NOT NULL AND fiscal_year IN (2024, 2025, 2026) GROUP BY type_name ORDER BY permits DESC OFFSET 0 ROWS FETCH FIRST 8 ROWS ONLY; ``` **Which neighborhoods have the worst roads?** Length-weighted pavement condition (0 to 100) from the latest survey: ```sql SELECT n.neighborhood_name, COUNT(*) AS segments, ROUND(SUM(s.length_ft) / 5280.0, 1) AS miles, ROUND(SUM(s.last_pci * 1.0 * s.length_ft) / SUM(s.length_ft), 1) AS length_weighted_pci FROM streets s JOIN neighborhoods n ON n.neighborhood_id = s.neighborhood_id WHERE s.maintained_by = 'City' AND s.last_pci IS NOT NULL GROUP BY n.neighborhood_name ORDER BY length_weighted_pci; ``` **Did the snow cause estimated water reads?** Buried meter boxes after the big storms show up as estimates: ```sql SELECT YEAR(bill_date) AS yr, MONTH(bill_date) AS mo, COUNT(*) AS bills, ROUND(100.0 * SUM(CASE WHEN read_type = 'Estimated' THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_estimated FROM utility_bills WHERE bill_date >= '2023-01-01' AND bill_date < '2023-08-01' GROUP BY YEAR(bill_date), MONTH(bill_date) ORDER BY yr, mo; ``` Estimates jump in the January and March 2023 billing cycles, right after the big storms, and fall back to the usual 2 percent in the cycles between. **Which agenda items split the council?** ```sql SELECT i.title, m.meeting_date, SUM(CASE WHEN v.vote = 'Yes' THEN 1 ELSE 0 END) AS yes_votes, SUM(CASE WHEN v.vote = 'No' THEN 1 ELSE 0 END) AS no_votes, i.outcome FROM agenda_items i JOIN council_meetings m ON m.meeting_id = i.meeting_id JOIN council_votes v ON v.agenda_item_id = i.agenda_item_id GROUP BY i.agenda_item_id, i.title, m.meeting_date, i.outcome HAVING SUM(CASE WHEN v.vote = 'No' THEN 1 ELSE 0 END) >= 2 ORDER BY m.meeting_date DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` **When are property tax payments due?** The fiscal calendar carries the installment and apportionment dates: ```sql SELECT calendar_date, fiscal_period_name, tax_installment, budget_phase FROM fiscal_calendar WHERE fiscal_year = 2026 AND tax_installment IS NOT NULL ORDER BY calendar_date; ``` All of these run on SQL Server and MariaDB. ## Maps and fiscal periods - **Bin the complaints.** Round the coordinates and group to get a heat map of where noise complaints concentrate: ```sql SELECT ROUND(latitude, 3) AS lat, ROUND(longitude, 3) AS lon, COUNT(*) AS complaints FROM service_requests q JOIN service_request_types t ON t.type_id = q.type_id WHERE t.type_name LIKE 'Noise complaint%' AND q.latitude IS NOT NULL AND q.fiscal_year = 2026 GROUP BY ROUND(latitude, 3), ROUND(longitude, 3) ORDER BY complaints DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` - **Map the streets.** `streets` has `from_latitude`/`from_longitude` and `to_latitude`/`to_longitude`, so a line layer can draw the network, colored by `last_pci`, `snow_priority` or `route_id`. - **Points of every kind.** Parcels (colored by `land_use`, owner origin or assessed value), assets, 311 requests, dispatch calls, permits, code cases and capital projects all carry latitude and longitude, and `neighborhoods` give a roll-up level. - **Fiscal periods.** Use `fiscal_year`, `fiscal_quarter` and `fiscal_period` on any fact table, or join `fiscal_calendar` for the budget phase, tourism season, snow season and holidays. See [Maps and fiscal calendars](/analyze/maps-and-fiscal). ## Excel and Power BI - **Star model.** Use `service_requests`, `cad_calls`, `tot_filings`, `utility_bills`, `permits` and `gl_journal_lines` as fact tables, with `neighborhoods`, `parcels`, `departments`, `fiscal_calendar` and `weather_daily` as shared dimensions. `utility_bills` and `tot_filings` are large, so load them with a filter or use the views. - **Measures to write.** TOT collected, Occupancy %, Requests, Percent within SLA, Average response minutes, Overtime pay, Budget variance, Net assessed value, Absentee owner %, Water billed and collected, Pavement condition. - **Map visuals.** Set latitude and longitude as such, size bubbles by a measure, and filter on fiscal year. For choropleths by neighborhood, use the neighborhood centers or bring your own boundary shapes. ## Realistic imperfections Government data is messy, and so is this: duplicate 311 requests (`duplicate_of_request_id`), requests with no coordinates or a parcel that never matched, mixed-case free-text addresses, requests that were never closed, request types that still point at a department that has merged away, vendors entered twice with different spellings, rentals that file late or never, audit adjustments, permits that expire or are withdrawn, inspections cancelled by snow, estimated water reads, written-off water bills and cases that stay open. --- # Retail data warehouse (retailwh) `retailwh` is a data warehouse for a fictional retailer with 43 locations in the United States, Canada, the United Kingdom and Germany, plus an online, phone and marketplace "store" for each country. It is not an application database: it is organized the way a warehouse is, with **fact tables** (what happened) around **dimension tables** (who, what, where, when), so it is the place to learn modeling patterns before you build them in Power BI or Excel. April-March fiscal year, FY2023 through 24 October 2026 (the start of FY2027), about 787,000 rows in 25 tables. > All customers, products, brands, suppliers and amounts are invented. Region boundaries are deliberately rough rectangles. ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `retailwh`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** open [the database page](/databases/retailwh) for the Playground, the SQL editor and the schema explorer. The schema explorer draws every declared relationship, which makes the star and snowflake shapes easy to see. - **Fiscal year:** April to March, named by the year it ends, so FY2026 is April 2025 to March 2026 and fiscal period 1 is April. `dim_date` has the fiscal columns; there are no fiscal columns on the facts, because a warehouse joins to the date dimension. ## The tables | Pattern | Tables | What they hold | |---|---|---| | Dates and money | `dim_date`, `dim_currency`, `fx_rates` | A daily calendar with fiscal columns and holidays; four currencies and a daily rate to USD | | Places | `dim_country`, `dim_region`, `dim_store` | 8 countries, 6 selling regions (with a rough boundary polygon), 43 locations with latitude and longitude | | Product snowflake | `dim_category`, `dim_subcategory`, `dim_product`, `dim_brand`, `dim_supplier`, `dim_product_price`, `dim_tag`, `bridge_product_tag` | 323 products in 17 subcategories and 5 categories, 24 brands, 14 suppliers, a price history and tags | | Customers (type 2) | `dim_customer` | 12,000 customers as about 19,000 rows: a new row when a customer moves, changes tier or changes segment | | Other dimensions | `dim_channel`, `dim_promotion` | In-store, online, phone and marketplace; 44 promotions | | Sales | `fact_sales`, `fact_returns` | About 379,000 order lines at the grain of one product on one order, and the lines that came back | | Inventory | `fact_inventory_snapshot` | Month-end stock by product and store | | Orders in flight | `fact_order_pipeline` | One row per online, phone or marketplace order, with a date for each stage | | Promotions | `fact_promo_coverage` | Which products were on offer in which stores for which promotion (a factless table) | | Plan and summary | `fact_budget`, `fact_sales_period_agg` | Budgets by fiscal period, category and region; and actual sales summarized to the same grain | Five views flatten the common joins: `vw_sales_detail` (the whole star in one row), `vw_customer_current`, `vw_inventory_latest`, `vw_order_cycle_time` and `vw_budget_vs_actual`. ## How the tables join ``` dim_date ───────────┐ (role-playing: order date, ship date, and six dates on the pipeline) dim_customer ───────┤ dim_channel ────────┤ dim_promotion ──────┼──< fact_sales >── dim_product >── dim_subcategory >── dim_category (snowflake) dim_currency ───────┤ │ └─────── dim_brand >── dim_supplier >── dim_country >── dim_currency dim_store >── dim_region >── dim_country >── dim_currency fact_returns >── fact_sales dim_product >──< dim_tag (via bridge_product_tag) fact_order_pipeline, fact_inventory_snapshot, fact_promo_coverage: other facts on the same dimensions fact_budget and fact_sales_period_agg: facts at a coarser grain, joined on category and region ``` ## Each pattern, with a query ### Star schema: one fact, many dimensions `vw_sales_detail` joins the star for you. Sales and margin by category for the 2026 fiscal year: ```sql SELECT fiscal_year, category_name, ROUND(SUM(net_amount_usd), 0) AS sales_usd, ROUND(SUM(margin_usd) * 100.0 / SUM(net_amount_usd), 1) AS margin_pct FROM vw_sales_detail WHERE fiscal_year = 2026 GROUP BY fiscal_year, category_name ORDER BY sales_usd DESC; ``` ### Snowflake: follow the chain Product reaches its category through the subcategory, and its supplier country through brand and supplier. Nothing is wrong with this, but each hop is another join and another relationship in Power BI: ```sql SELECT cat.category_name, sub.subcategory_name, b.brand_name, sup.supplier_name, co.country_name AS supplier_country, COUNT(DISTINCT p.product_key) AS products, ROUND(SUM(s.net_amount_usd), 0) AS sales_usd FROM fact_sales s JOIN dim_product p ON p.product_key = s.product_key JOIN dim_subcategory sub ON sub.subcategory_key = p.subcategory_key JOIN dim_category cat ON cat.category_key = sub.category_key JOIN dim_brand b ON b.brand_key = p.brand_key JOIN dim_supplier sup ON sup.supplier_key = b.supplier_key JOIN dim_country co ON co.country_key = sup.country_key GROUP BY cat.category_name, sub.subcategory_name, b.brand_name, sup.supplier_name, co.country_name ORDER BY sales_usd DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` ### Slowly changing dimension (type 2) When a customer changes, `dim_customer` gets a new row instead of changing the old one. Each version has `valid_from`, `valid_to` (9999-12-31 for the current one), `is_current` and a `change_reason`. Here are six customers' histories: ```sql SELECT customer_id, customer_name, loyalty_tier, segment, city, valid_from, valid_to, is_current, change_reason FROM dim_customer WHERE customer_id IN ('C000010', 'C000011', 'C000012', 'C000013', 'C000014', 'C000015') ORDER BY customer_id, valid_from; ``` `fact_sales.customer_key` points at the version that was true **on the order date**, so a sale keeps the tier and segment the customer had at the time. That is what `tier_at_sale` in the view means: ```sql SELECT tier_at_sale, COUNT(DISTINCT customer_id) AS customers, COUNT(DISTINCT order_id) AS orders, ROUND(SUM(net_amount_usd), 0) AS sales_usd, ROUND(SUM(net_amount_usd) / COUNT(DISTINCT order_id), 2) AS avg_order_usd FROM vw_sales_detail WHERE fiscal_year = 2026 GROUP BY tier_at_sale ORDER BY sales_usd DESC; ``` To count customers rather than versions, count `DISTINCT customer_id`. To report on today's attributes, join through `customer_id` to the row where `is_current = 1` (see `vw_customer_current`). ### Price history with validity dates `dim_product_price` is a type 2 table too: a list price with the dates it applies. Prices change most Aprils, and electronics get cheaper: ```sql SELECT p.sku, p.product_name, pp.valid_from, pp.valid_to, pp.list_price_usd FROM dim_product p JOIN dim_product_price pp ON pp.product_key = p.product_key WHERE p.product_key IN (SELECT product_key FROM dim_product_price GROUP BY product_key HAVING COUNT(*) >= 4) ORDER BY p.sku, pp.valid_from OFFSET 0 ROWS FETCH FIRST 8 ROWS ONLY; ``` ### Multi-currency Each sale line keeps the local amount, the exchange rate on the day, and the USD amount. The effective rate is not the average of the daily rates, because it is weighted by sales: ```sql SELECT cur.currency_code, COUNT(*) AS line_count, ROUND(SUM(s.net_amount_local), 0) AS sales_local, ROUND(SUM(s.net_amount_usd), 0) AS sales_usd, ROUND(SUM(s.net_amount_usd) / SUM(s.net_amount_local), 4) AS effective_rate, ROUND(MIN(s.fx_rate_to_usd), 4) AS lowest_rate, ROUND(MAX(s.fx_rate_to_usd), 4) AS highest_rate FROM fact_sales s JOIN dim_currency cur ON cur.currency_key = s.currency_key GROUP BY cur.currency_code ORDER BY sales_usd DESC; ``` ### Periodic snapshot: do not add a balance across time `fact_inventory_snapshot` records stock at each month-end. You can add it across products and stores, but not across months, because the same units appear in every month. Look at one month at a time, take the last month, or average: ```sql SELECT dd.fiscal_period_name, SUM(i.on_hand_units) AS on_hand_units_all_stores, SUM(i.units_sold_in_month) AS units_sold, SUM(CAST(i.is_stockout AS INT)) AS stockouts FROM fact_inventory_snapshot i JOIN dim_date dd ON dd.date_key = i.month_end_date_key WHERE dd.fiscal_year = 2026 GROUP BY dd.fiscal_period_name ORDER BY dd.fiscal_period_name; ``` `units_sold_in_month` is additive across months; `on_hand_units` is not. ### Accumulating snapshot: an order moving through stages `fact_order_pipeline` has one row per order and a date for each stage; a stage the order has not reached is null. `vw_order_cycle_time` turns the dates into days: ```sql SELECT ch.channel_name, od.fiscal_year, COUNT(*) AS orders, ROUND(AVG(o.days_to_ship * 1.0), 2) AS avg_days_to_ship, ROUND(AVG(o.days_to_deliver * 1.0), 2) AS avg_days_to_deliver FROM vw_order_cycle_time o JOIN dim_channel ch ON ch.channel_name = o.channel_name JOIN dim_date od ON od.calendar_date = o.order_date WHERE o.status IN ('Delivered', 'Returned') AND od.fiscal_year >= 2025 GROUP BY ch.channel_name, od.fiscal_year ORDER BY ch.channel_name, od.fiscal_year; ``` ### Factless fact: what was offered, and what did not sell `fact_promo_coverage` has no measures. It records that a product was on promotion in a store. Joining it to the sales shows the offers nobody took, which a sales table alone can never show: ```sql SELECT pr.promotion_name, COUNT(*) AS offered_combinations, SUM(CASE WHEN x.line_count IS NULL THEN 1 ELSE 0 END) AS never_sold, ROUND(100.0 * SUM(CASE WHEN x.line_count IS NULL THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_never_sold FROM fact_promo_coverage c JOIN dim_promotion pr ON pr.promotion_key = c.promotion_key LEFT JOIN (SELECT promotion_key, product_key, store_key, COUNT(*) AS line_count FROM fact_sales WHERE promotion_key IS NOT NULL GROUP BY promotion_key, product_key, store_key) x ON x.promotion_key = c.promotion_key AND x.product_key = c.product_key AND x.store_key = c.store_key WHERE pr.start_date >= '2025-01-01' AND pr.start_date < '2026-01-01' GROUP BY pr.promotion_name ORDER BY pct_never_sold DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` ### Facts at different grain: budget against actual > **Watch out.** The budget is by fiscal period, category and region; sales are by order line. Compare them at the budget's grain. The warehouse keeps a pre-summarized `fact_sales_period_agg` for this, which is also the standard fix when a view over a large fact table is slow: ```sql SELECT category_name, SUM(budget_usd) AS budget_usd, SUM(actual_usd) AS actual_usd, ROUND(SUM(actual_usd) * 100.0 / SUM(budget_usd), 1) AS pct_of_budget FROM vw_budget_vs_actual WHERE fiscal_year = 2026 GROUP BY category_name ORDER BY pct_of_budget DESC; ``` ### Many-to-many with a bridge table A product can have several tags and a tag many products. Summing sales by tag counts a product once for each of its tags, so the totals do not add up to total sales. That is correct for this question, but be careful when you use tags as a filter: ```sql SELECT t.tag_name, COUNT(DISTINCT p.product_key) AS products, ROUND(SUM(s.net_amount_usd), 0) AS sales_usd FROM fact_sales s JOIN dim_product p ON p.product_key = s.product_key JOIN bridge_product_tag b ON b.product_key = p.product_key JOIN dim_tag t ON t.tag_key = b.tag_key GROUP BY t.tag_name ORDER BY sales_usd DESC; ``` ### Role-playing dimension: one date table, several jobs `fact_sales` has an order date and a ship date, both pointing at `dim_date`. Join the date table twice with different aliases. This finds December 2025 orders that shipped in January: ```sql SELECT od.month_name AS order_month, sd.month_name AS ship_month, COUNT(*) AS line_count FROM fact_sales s JOIN dim_date od ON od.date_key = s.order_date_key JOIN dim_date sd ON sd.date_key = s.ship_date_key WHERE od.calendar_year = 2025 AND od.calendar_month = 12 AND sd.calendar_month <> od.calendar_month GROUP BY od.month_name, sd.month_name; ``` ### Geography: points and polygons Stores and customers have latitude and longitude, which is enough for a bubble map. `dim_region.boundary_geojson` holds a rough GeoJSON polygon for each region, to practice a shape map. Stores by region: ```sql SELECT r.region_name, co.country_name, COUNT(*) AS stores, SUM(st.square_feet) AS square_feet FROM dim_store st JOIN dim_region r ON r.region_key = st.region_key JOIN dim_country co ON co.country_key = r.country_key WHERE st.is_online = 0 GROUP BY r.region_name, co.country_name ORDER BY stores DESC; ``` ## Modeling it in Power BI and Excel - **Load the star.** Relate each fact key to its dimension, one-to-many with single-direction filtering, hide the keys, and write measures on the fact tables. Flatten the snowflake chains in Power Query unless you need them. - **Handle the customer versions.** Relate `fact_sales[customer_key]` to `dim_customer[customer_key]` to get attributes at the time of sale. Count customers with a distinct count on `customer_id`. - **One date table, several relationships.** Make `order_date_key` active and `ship_date_key` inactive, and use `USERELATIONSHIP` in a measure. See [time intelligence](/analyze/time-intelligence) and mark `dim_date` as the date table; the fiscal columns give a April-March year. - **Snapshots.** For on-hand units, write a measure that takes the last month in the filter (`LASTNONBLANK` or `CLOSINGBALANCEMONTH`), not a sum. - **Budget against actual.** Relate `fact_budget` and `fact_sales_period_agg` to shared category, region and fiscal-period dimensions, not to each other. - **Many-to-many tags.** Use a bridge table with a filter from the tag to the product; expect double counting across tags. - **Maps.** Put `latitude` and `longitude` in a map visual, and use `boundary_geojson` as a custom shape. See [Maps and fiscal calendars](/analyze/maps-and-fiscal). More structures, and what Power BI cannot do with them, are in [Data structures and Power BI limits](/analyze/data-structures). ## Things that happen in the data - Four stores open (Austin, Ottawa, Bristol, Cologne) and three close (Pittsburgh, Kansas City, Birmingham), so `opened_date` and `closed_date` matter. - Each store carries about a third of the products; the online locations carry all of them. - About 2 percent of non-store orders are cancelled and have a pipeline row but no sales lines. - Returns are most common for apparel and for orders that were shipped. - Exchange rates drift around their base values; the Canadian dollar, pound and euro each drift by several percent over the period. - Promotions apply only where `fact_promo_coverage` says the product was offered in that store. --- # Big data: manufacturer orders (mfgorders) `mfgorders` is a manufacturer's order history: **25,165,824 order lines** (6.3 million orders, about $6.3 billion) from January 2023 to 24 October 2026, sold to wholesalers, retail chains and independent retailers in eleven countries and made in eight plants. It is here to practice **querying large tables**, and it adds the one thing the other databases lack: **manufacturing**, with a three-level bill of materials and a customer hierarchy in which retailers belong to wholesalers. > All customers, products and amounts are invented. ## How 25 million rows take almost no disk Nothing big sits in the project folder. The rows are not stored in a data file you load: SQL Server builds them on the server, and MariaDB computes them when you ask. | Engine | What `order_lines` is | Disk | Speed | |---|---|---|---| | **SQL Server** | A real table with a **clustered columnstore index**: each column is stored and compressed separately | About 380 MB for all 25 million rows (an ordinary table would need several gigabytes) | Seconds. Every full-table query below runs in under four seconds | | **MariaDB** | A **view** over the virtual `seq_1_to_25165824` table: each row is calculated from its number when you query it | None | Roughly five microseconds a row, so a full scan takes minutes. Limit the rows with `order_line_id` | Both engines use the same integer arithmetic and no random numbers, so the data is identical. Because order dates grow with the order number, an `order_line_id` range is also a date range: the first two million lines are January through April 2023, and the last lines are October 2026. **On MariaDB, add a limit to the examples below.** Put `o.order_line_id <= 1000000 AND` at the start of the `WHERE` clause (or use `o.order_line_id > 25000000` for the latest lines). The queries are written for SQL Server and run unchanged there. ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `mfgorders`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** [the database page](/databases/mfgorders). The SQL editor works on both engines. - **Fiscal year:** November to October, named by the year it ends (`dim_date.fiscal_year`). ## The tables | Table | Rows | What it holds | |---|---:|---| | `order_lines` | 25,165,824 | One row per order line: `order_id`, `line_number`, `order_date`, `ship_date`, customer, product, plant, channel, `quantity`, `unit_price`, `discount_pct`, `net_amount`, `cost_amount` | | `dim_customer` | 20,000 | 500 wholesalers, 1,000 retail chains, 18,500 retailers; `parent_customer_id` links a retailer to its wholesaler; coordinates | | `dim_product` | 2,800 | 1,500 finished goods, 500 subassemblies and 800 components, with list price, standard cost, make or buy and lead time | | `bom` | about 8,500 | The bill of materials: parent, child, quantity per, scrap percent | | `dim_product_family`, `dim_plant`, `dim_order_channel` | 12, 8, 4 | Families, plants (with coordinates) and order channels | | `dim_date` | 1,461 | Calendar with fiscal columns | Views: `vw_bom_explosion` (every component of a finished good at every level, with cost) and `vw_customer_hierarchy`. The big table has **no enforced foreign keys**, which is normal for columnstore fact tables, so the schema explorer shows nothing connected to it. Join on the ids. ## Questions to try **How big is it?** The whole table in a few seconds: ```sql SELECT COUNT(*) AS order_lines, COUNT(DISTINCT o.order_id) AS orders, COUNT(DISTINCT o.customer_id) AS customers, MIN(o.order_date) AS first_order, MAX(o.order_date) AS last_order, ROUND(SUM(o.net_amount) / 1000000.0, 1) AS sales_millions FROM order_lines o WHERE o.order_date >= '2023-01-01'; ``` **Sales and margin by fiscal quarter and customer type.** A join to two dimensions across all 25 million lines: ```sql SELECT d.fiscal_year, d.fiscal_quarter, c.customer_type, COUNT(*) AS line_count, ROUND(SUM(o.net_amount) / 1000000.0, 1) AS sales_millions, ROUND(100.0 * (SUM(o.net_amount) - SUM(o.cost_amount)) / SUM(o.net_amount), 1) AS margin_pct FROM order_lines o JOIN dim_customer c ON c.customer_id = o.customer_id JOIN dim_date d ON d.calendar_date = o.order_date WHERE o.order_date >= '2023-01-01' GROUP BY d.fiscal_year, d.fiscal_quarter, c.customer_type ORDER BY d.fiscal_year, d.fiscal_quarter, c.customer_type; ``` Wholesalers get larger quantities at a 25 percent discount, so their margin is lower than retailers'. **Which products sell the most, and what share of the total is each?** A window function over the grouped result: ```sql SELECT p.sku, p.product_name, ROUND(SUM(o.net_amount), 0) AS sales, ROUND(100.0 * SUM(o.net_amount) / SUM(SUM(o.net_amount)) OVER (), 2) AS pct_of_total FROM order_lines o JOIN dim_product p ON p.product_id = o.product_id WHERE o.order_date >= '2023-01-01' GROUP BY p.sku, p.product_name ORDER BY sales DESC OFFSET 0 ROWS FETCH FIRST 10 ROWS ONLY; ``` **Is it a Pareto?** Rank the customers, then band them by rank: ```sql WITH c AS (SELECT o.customer_id, SUM(o.net_amount) AS s FROM order_lines o WHERE o.order_date >= '2023-01-01' GROUP BY o.customer_id), r AS (SELECT customer_id, s, ROW_NUMBER() OVER (ORDER BY s DESC) AS rn, SUM(s) OVER () AS total FROM c) SELECT CASE WHEN rn <= 200 THEN '1 top 1% (200 customers)' WHEN rn <= 1000 THEN '2 next 4%' ELSE '3 other 95%' END AS band, COUNT(*) AS customers, ROUND(SUM(s) / 1000000.0, 1) AS sales_millions, ROUND(100.0 * SUM(s) / MAX(total), 1) AS pct_of_sales FROM r GROUP BY CASE WHEN rn <= 200 THEN '1 top 1% (200 customers)' WHEN rn <= 1000 THEN '2 next 4%' ELSE '3 other 95%' END ORDER BY band; ``` The top 1 percent of customers (200 of about 20,000) place roughly a fifth of the sales, and the top 5 percent roughly 40 percent. **How do monthly sales trend?** A three-month moving average with a window frame: ```sql WITH m AS (SELECT d.calendar_year AS yr, d.calendar_month AS mo, SUM(o.net_amount) AS sales FROM order_lines o JOIN dim_date d ON d.calendar_date = o.order_date WHERE o.order_date >= '2023-01-01' GROUP BY d.calendar_year, d.calendar_month) SELECT yr, mo, ROUND(sales / 1000000.0, 1) AS sales_millions, ROUND(AVG(sales) OVER (ORDER BY yr, mo ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) / 1000000.0, 1) AS moving_avg_3_months FROM m ORDER BY yr, mo; ``` **Where is the margin, by plant and product family?** ```sql SELECT pl.plant_name, f.family_name, ROUND(SUM(o.net_amount) / 1000000.0, 1) AS sales_millions, ROUND(100.0 * (SUM(o.net_amount) - SUM(o.cost_amount)) / SUM(o.net_amount), 1) AS margin_pct FROM order_lines o JOIN dim_product p ON p.product_id = o.product_id JOIN dim_product_family f ON f.family_id = p.family_id JOIN dim_plant pl ON pl.plant_id = o.plant_id WHERE o.order_date >= '2023-01-01' GROUP BY pl.plant_name, f.family_name ORDER BY sales_millions DESC OFFSET 0 ROWS FETCH FIRST 8 ROWS ONLY; ``` Each product is made mostly in its home plant (85 percent of lines), so plants differ by product family. **How many lines have not shipped yet?** Open lines are in the latest weeks: ```sql SELECT ch.channel_name, COUNT(*) AS line_count, SUM(CASE WHEN o.ship_date IS NULL THEN 1 ELSE 0 END) AS not_yet_shipped, ROUND(100.0 * SUM(CASE WHEN o.ship_date IS NULL THEN 1 ELSE 0 END) / COUNT(*), 2) AS pct_not_shipped FROM order_lines o JOIN dim_order_channel ch ON ch.channel_id = o.channel_id WHERE o.order_date >= '2026-09-01' GROUP BY ch.channel_name ORDER BY ch.channel_name; ``` **Which wholesalers sell the most once you include their retailers?** The customer hierarchy is a self-join on `parent_customer_id`: ```sql SELECT w.customer_name AS wholesaler, w.country, COUNT(DISTINCT c.customer_id) AS accounts_in_hierarchy, ROUND(SUM(o.net_amount) / 1000000.0, 2) AS sales_millions FROM order_lines o JOIN dim_customer c ON c.customer_id = o.customer_id JOIN dim_customer w ON w.customer_id = COALESCE(c.parent_customer_id, c.customer_id) WHERE w.customer_type = 'Wholesaler' AND o.order_date >= '2023-01-01' GROUP BY w.customer_name, w.country ORDER BY sales_millions DESC OFFSET 0 ROWS FETCH FIRST 8 ROWS ONLY; ``` **Sales by region and customer type for one month:** ```sql SELECT c.region, c.customer_type, COUNT(*) AS line_count, ROUND(SUM(o.net_amount) / 1000000.0, 1) AS sales_millions FROM order_lines o JOIN dim_customer c ON c.customer_id = o.customer_id WHERE o.order_date >= '2026-01-01' AND o.order_date < '2026-02-01' GROUP BY c.region, c.customer_type ORDER BY sales_millions DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` Because `order_date` grows with the row number, SQL Server skips the compressed row groups that cannot contain January 2026 (segment elimination), so a one-month query reads a small part of the table. ### SQL Server only: approximate distinct counts On a very large table an exact `COUNT(DISTINCT ...)` is expensive. `APPROX_COUNT_DISTINCT` is much cheaper and is typically within a few percent: ```sql SELECT APPROX_COUNT_DISTINCT(order_id) AS approx_orders, COUNT(DISTINCT order_id) AS exact_orders FROM order_lines; ``` ### Manufacturing: the bill of materials `vw_bom_explosion` is a recursive query. It starts at each finished good and repeatedly follows `bom` down to subassemblies and components, multiplying the quantities. The full explosion of one finished good, with every level: ```sql SELECT finished_good_sku, bom_level, item_sku, item_name, item_type, make_or_buy, quantity_per_finished_good, ROUND(extended_cost_usd, 2) AS extended_cost_usd FROM vw_bom_explosion WHERE finished_good_sku = 'FG-00001' ORDER BY bom_level, item_sku; ``` Rolling the component cost up and comparing it with the product's standard cost shows where materials alone exceed the standard cost, which is a margin problem to chase: ```sql SELECT x.finished_good_sku, p.standard_cost_usd AS standard_cost, ROUND(x.component_cost, 2) AS rolled_up_component_cost, ROUND(x.component_cost - p.standard_cost_usd, 2) AS gap FROM (SELECT finished_good_sku, SUM(CASE WHEN item_type = 'Component' THEN extended_cost_usd ELSE 0 END) AS component_cost FROM vw_bom_explosion GROUP BY finished_good_sku) x JOIN dim_product p ON p.sku = x.finished_good_sku ORDER BY ABS(x.component_cost - p.standard_cost_usd) DESC OFFSET 0 ROWS FETCH FIRST 6 ROWS ONLY; ``` ## What makes a table like this fast, and what to do in Power BI - **Columnstore** compresses each column on its own and processes about 900 rows at a time (batch mode). Narrow, repetitive columns such as ids and codes compress best. - **Sorted loading.** The rows are inserted in date order, so each row group covers a narrow date range and a date filter skips most of them. - **Filter early, select few columns.** Avoid `SELECT *`: a columnstore only reads the columns you name. - **Pre-aggregate for reports.** A report rarely needs 25 million rows. Summarize to day, product and customer type (about a million rows) in SQL Server, or keep the detail in DirectQuery and the summary in Import, using Power BI aggregations. In `retailwh`, `fact_sales_period_agg` shows the same idea on a small scale. - **Import is limited by model size, not row count.** Order numbers and exact timestamps have a very high number of distinct values and compress poorly; drop what no report uses. - **DirectQuery** sends a query to SQL Server per visual, which columnstore answers quickly, but one query returns at most a million rows. Excel cannot use DirectQuery. - **Incremental refresh** loads only the recent partitions of a big table. See [Data structures and Power BI limits](/analyze/data-structures) for the limits that apply to models of this size. ## Things worth knowing - **Skew.** A few wholesalers and chains place a large share of the orders, and a few products outsell the rest, which makes top-N queries interesting. - **Open lines.** `ship_date` is null for lines that would ship after 24 October 2026. - **Hierarchy.** About 70 percent of retailers have a wholesaler as `parent_customer_id`; the rest buy directly. - **The build.** SQL Server builds the table in 24 batches of 1,048,576 rows (one compressed row group each), then compresses and shrinks the file. A rebuild takes about ten minutes; MariaDB takes seconds because it stores nothing. --- # Utility field operations (fieldops) `fieldops` is the operating data of a small water and electric utility: the equipment it maintains, the work orders crews complete, the vendors and purchase orders behind the parts, the customers it bills each month, and the outages it has to explain. It is built for **maintenance, procurement, billing and reliability reports** in Excel, Power BI and SQL, and every relationship is a declared key, so a modeling tool finds them on its own. > **These are invented records.** The people, vendors, customers and dollar amounts are made up. The shape of the work, such as emergency jobs finishing in days and low-priority jobs in weeks, is realistic on purpose. ## Connect to it - **SQL Server:** server `learnsql.cpetoday.com,1433`, database `fieldops`, tables in the `dbo` schema. See [Connect Excel and Power BI](/connect). - **In the browser:** open [the database page](/databases/fieldops) for the Playground, the SQL editor and the schema explorer. - It exists on both SQL Server and MariaDB with identical rows (about 161,000 in all). ## Tables at a glance | Area | Tables | What they hold | |---|---|---| | Organization | `Regions`, `Departments`, `Employees` | 6 regions, 10 departments (each name appears once), 400 employees with job title and hourly rate | | Equipment | `Assets` | 1,500 transformers, poles, meters, valves, pumps, hydrants, pipe segments and breakers, with install date, replacement cost and condition | | Maintenance | `WorkOrders`, `WorkOrderMaterials`, `Materials` | 25,000 work orders (type, priority, status, days to complete, labor hours and cost), 50,000 material lines, 300 materials | | Purchasing | `Vendors`, `PurchaseOrders`, `VendorInvoices` | 150 vendors with payment terms, 7,000 purchase orders, 9,000 invoices with days to pay and a disputed flag | | Customers | `CustomerAccounts`, `MonthlyBills` | 5,000 accounts in four rate classes and twelve monthly bills each for 2025 | | Reliability | `ServiceInterruptions` | 3,000 outages with duration, customers affected and cause | Views that save the joins: `vw_WorkOrderDetail`, `vw_WorkOrderMaterialCost`, `vw_BillingDetail`, `vw_VendorSpend` and `vw_Outages`. ## What the data does - A work order is in the **same region as its asset**, and its labor cost is the hours times the assigned employee's hourly rate. - **Urgent work finishes sooner:** emergency orders average about two days, low-priority orders more than a month. Open, in-progress and cancelled orders have no completion time (a blank, on purpose). - **Worn assets cost more:** poor and critical assets take more labor hours per order. - **Invoices follow orders:** each invoice belongs to an approved, received or closed purchase order, is dated after it, and is within 3% of its amount. Days to pay track the vendor's terms (Net 15 to Net 60), and disputed invoices take longer. - **Usage is seasonal:** monthly bills peak in summer, and a commercial or industrial account uses far more than a residential one. ## Questions to try ```sql -- Which asset types cost the most to maintain? SELECT AssetType, COUNT(*) AS orders, SUM(LaborCost) AS labor FROM vw_WorkOrderDetail WHERE Status = 'Completed' GROUP BY AssetType ORDER BY labor DESC; -- How long do work orders take, by priority? SELECT Priority, AVG(DaysToComplete) AS avg_days FROM vw_WorkOrderDetail WHERE Status = 'Completed' GROUP BY Priority; -- Which vendor categories get paid slowest, and how many invoices are disputed? SELECT Category, AVG(DaysToPay) AS avg_days_to_pay, SUM(CASE WHEN Disputed = 1 THEN 1 ELSE 0 END) AS disputed FROM vw_VendorSpend WHERE InvoiceID IS NOT NULL GROUP BY Category ORDER BY avg_days_to_pay DESC; -- Customer-minutes of outage by cause SELECT Cause, SUM(CustomerMinutes) AS customer_minutes FROM vw_Outages GROUP BY Cause ORDER BY customer_minutes DESC; ``` ## In Power BI and Excel - Load the tables (or the views) and let the relationships come in from the declared keys. `Regions`, `Departments` and `Employees` are shared dimensions for work orders, purchasing and accounts. - `WorkOrders` and `MonthlyBills` are the large facts. A date table related to `OpenedDate`, `PODate`, `InvoiceDate`, `BillDate` and `StartDate` (one active relationship, the others inactive) shows the **role-playing date** pattern. - Good measures: Work Orders, Completed %, Average Days to Complete, Labor Cost per Order, Backlog (open orders), Spend, Days Payable, Disputed %, Billed Revenue, Usage per Account, Customer-Minutes of Outage (the basis of reliability indexes such as SAIDI). - `WorkOrders.DaysToComplete` is blank for unfinished work, so averages ignore it; compare with a count of open orders. --- # Maps and fiscal calendars Seven of the sample databases share the same two building blocks: **addresses with coordinates** for mapping, and **a fiscal calendar** for charting by fiscal period. They behave the same way in `cpafirm`, `nonprofit`, `pqlab` (its tables are prefixed `core_`), `smarthome`, `bigbear`, `retailwh` and `mfgorders`, so a map or a fiscal-year chart you build for one works for the others. ## Addresses and coordinates Every table that holds an address has the same columns: | Column | Meaning | |---|---| | `address_line1`, `city`, `state`, `postal_code` | The address as entered | | `county`, `country` | Added from the city (county names, `US`) | | `latitude`, `longitude` | WGS84 decimal degrees, six decimals | | `geocode_quality` | How the point was found: `rooftop`, `street`, `postal_centroid` or `city_centroid` (`bigbear` adds `gps` for mobile-app reports, `address_match`, `neighborhood_centroid` and `failed`) | | `full_address` | The address in one line, for geocoding tools and map labels | > **Watch out.** Where the address is missing the coordinates are null. Where only the city is known you get the city centroid. | Database | Tables with coordinates | |---|---| | `cpafirm` | `clients`, `offices`, `client_visits` (origin and destination of each trip) | | `nonprofit` | `constituents` (about 18,500 donors, volunteers and prospects), `funders`, `vendors`, `program_sites`, `mobile_market_stops`, event venues | | `pqlab` | `core_customers`, `core_stores`, `core_suppliers` (HQ), `core_shipments` (destination and distance from the warehouse), plus the `lab_geo_*` practice tables | | `smarthome` | `homes`, `points_of_interest`, `geofences`, `trips`, `trip_points`, and an indoor floor plan in feet | | `bigbear` | `parcels`, `facilities`, `assets`, `streets` (segment endpoints), `service_requests`, `cad_calls`, `permits`, `inspections`, `code_cases`, `work_orders`, `capital_projects`, `special_events`, `businesses`, `vendors`, `property_owners` | | `retailwh` | `dim_store`, `dim_customer` (latitude and longitude), `dim_region` (a rough boundary polygon as GeoJSON) | | `mfgorders` | `dim_customer`, `dim_plant` | Postal codes in these databases are fictitious (a real city prefix with random digits). The point for a postal code is a synthetic centroid placed near the city, so use `postal_centroid` rows for rough maps only. ## Reference geography Each database has the same three reference tables (in `pqlab` they are `core_geo_states`, `core_geo_places` and `core_geo_postal_codes`): | Table | Holds | |---|---| | `geo_states` | The 50 states and DC: name, FIPS code, Census region and division, centroid, capital | | `geo_places` | Each city in use with county, centroid, population, time zone and metro area | | `geo_postal_codes` | A centroid for every postal code in the data | Join to them on `state_code`, on city and state, or on `postal_code`. Use `geo_states` for filled (shaded-area) maps and `geo_places` for bubble maps. ## Distance on a map The same great-circle (haversine) formula works on SQL Server and MariaDB. This finds the five clients farthest from the office that serves them in `cpafirm`: ```sql SELECT c.client_name, c.city, o.office_name, ROUND(3958.8 * 2 * ASIN(SQRT(POWER(SIN(RADIANS(c.latitude - o.latitude) / 2), 2) + COS(RADIANS(o.latitude)) * COS(RADIANS(c.latitude)) * POWER(SIN(RADIANS(c.longitude - o.longitude) / 2), 2))), 1) AS miles_from_office FROM clients c JOIN offices o ON o.office_id = c.office_id WHERE c.latitude IS NOT NULL ORDER BY miles_from_office DESC OFFSET 0 ROWS FETCH FIRST 5 ROWS ONLY; ``` Out-of-state clients (Arizona, for example) are served remotely, which is why they top the list. To build a catchment for the nearest store, use `lab_geo_distance_pairs` in `pqlab`, which already holds the miles and a flag for each customer's nearest store. ## Mapping in Excel and Power BI 1. **Set the data category.** In Power BI select `latitude` and `longitude` and set their data category to Latitude and Longitude. For a state map use `state` with the category State or Province. 2. **Bubble map.** Put `longitude` and `latitude` on the map visual, and size the bubbles by a measure such as donors, billed amount or miles. 3. **Filled map.** Join to `geo_states` and shade by state; use `census_region` for a region rollup. 4. **Routes.** `client_visits`, `trips` and `lab_geo_route_points` have an origin and a destination, or ordered points, which a line layer or a Shape map can draw. 5. **Big maps.** For thousands of points, round the coordinates (for example to one decimal) and group, so the map draws bins rather than every row. 6. **Excel.** Use 3D Maps (Insert, 3D Map) or a Filled Map chart; for a Geography data type, build it from `full_address` or `city` and `state`. ## Fiscal calendars Every database has a fiscal-calendar table (`fiscal_calendar`, or `core_fiscal_calendar` in `pqlab`) with one row per day, and its main fact tables carry three ready-made columns so you do not need a join to filter by fiscal period: | Column | Meaning | |---|---| | `fiscal_year` | The fiscal year (the naming rule is below) | | `fiscal_quarter` | 1 to 4 | | `fiscal_period` | 1 to 12, with 1 the first month of the fiscal year | The calendar tables add `fiscal_year_label` (such as FY2026), `fiscal_period_name` (such as FY2026-P03), the period and year start and end dates, `fiscal_week`, `fiscal_day_of_year`, and flags for the last day of a period and of a year. Each also has a few extras for its domain. | Database | Fiscal year | Named by | Extras in the calendar | |---|---|---|---| | `cpafirm` | July to June | year it ends | `tax_season` (Busy, Extension, Off) and `tax_deadline` | | `nonprofit` | July to June | year it ends | `giving_season`, holidays, Giving Tuesday | | `pqlab` | February to January | year it **starts** | a true 4-5-4 retail calendar (`retail_year`, `retail_period`, `retail_week_of_period`) and shopping seasons | | `smarthome` | October to September (water year) | year it ends | `utility_season` and the rate plan in force | | `bigbear` | July to June | year it ends | `budget_phase`, property tax installment dates, `tourism_season`, snow season and holidays | | `retailwh` | April to March | year it ends | `dim_date` with fiscal year, quarter, period and period name | | `mfgorders` | November to October | year it ends | `dim_date` with fiscal year, quarter, period and period name | "Named by the year it ends" means FY2026 is the year ending in 2026 (July 2025 to June 2026 for a July start). `pqlab` names its year by the year it starts, to match its older `core_calendar` table; its `retail_*` columns are the 4-5-4 weeks, which differ from the calendar-month fiscal periods by a few days at the edges. **Fiscal quarters of billing, with no join:** ```sql SELECT fiscal_year, fiscal_quarter, COUNT(*) AS invoices, SUM(total_amount) AS billed FROM invoices WHERE fiscal_year BETWEEN 2024 AND 2026 GROUP BY fiscal_year, fiscal_quarter ORDER BY fiscal_year, fiscal_quarter; ``` **Using the extras with a join** (revenue by tax season in `cpafirm`): ```sql SELECT f.tax_season, COUNT(*) AS invoices, SUM(i.total_amount) AS billed FROM invoices i JOIN fiscal_calendar f ON f.calendar_date = i.issue_date WHERE i.fiscal_year = 2025 GROUP BY f.tax_season ORDER BY billed DESC; ``` **Giving by season in `nonprofit`:** ```sql SELECT f.giving_season, COUNT(*) AS gifts, SUM(g.amount) AS raised FROM gifts g JOIN fiscal_calendar f ON f.calendar_date = g.gift_date WHERE g.fiscal_year = 2025 GROUP BY f.giving_season ORDER BY raised DESC; ``` **The retail calendar in `pqlab`:** ```sql SELECT f.retail_period, COUNT(*) AS orders, SUM(o.total) AS sales FROM core_orders o JOIN core_fiscal_calendar f ON f.calendar_date = o.order_date WHERE f.retail_year = 2025 GROUP BY f.retail_period ORDER BY f.retail_period; ``` ## Fiscal time in Power BI and Excel - **Use the calendar as your date table.** Load the fiscal calendar, mark `calendar_date` as the date table, relate it to the fact table's date, and put `fiscal_year_label` and `fiscal_period_name` on slicers and axes. - **Sort correctly.** Sort `fiscal_period_name` by itself (it sorts as text, FY2026-P01 to P12) or sort a month name by `fiscal_period`, so July comes first for a July year. - **Year to date.** Pass the year-end date to time intelligence, for example `TOTALYTD([Billed], 'fiscal_calendar'[calendar_date], "6/30")` for a June year end, or `"9/30"` for `smarthome`. See [time intelligence](/analyze/time-intelligence). - **Map by fiscal period.** Put `fiscal_year` or `fiscal_quarter` on a slicer next to a map, and the bubbles show that period's activity. - **Compare years.** `fiscal_period` lines up the same month of different fiscal years on one axis.