{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# Examples\n\nShort, self-contained recipes. Each one runs against a local EUROMOD model and its\nbundled training data; none of them needs a simulation.",
   "id": "cell-00"
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": "import os\n\nimport pandas as pd\n\n# Point this at your own EUROMOD model folder, or set EUROMOD_MODEL_PATH.\nMODEL_PATH = os.environ.get(\"EUROMOD_MODEL_PATH\", r\"C:\\EUROMOD_RELEASES\")\nDATA_DIR = os.path.join(MODEL_PATH, \"Input\")\n\n# Keep DataFrame previews compact in the rendered docs.\npd.set_option(\"display.max_columns\", 10)\npd.set_option(\"display.width\", 110)",
   "id": "cell-01"
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "7482"
     },
     "execution_count": 2
    }
   ],
   "source": "from euromod import Model\n\nmodel = Model(MODEL_PATH)\nsystem = model[\"BE\"][\"BE_2025\"]\ndata = pd.read_csv(os.path.join(DATA_DIR, \"BE_training_data.txt\"), sep=\"\\t\")\nlen(data)",
   "id": "cell-02"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Normalising shocks, and the id that follows\n\n`normalize_shocks` validates raw records and canonicalises them — group keys sorted,\nrows ordered by `(channel, metric, group, period)`. The canonical form hashes to a\ncontent id, so the same economic scenario always has the same id no matter which file\nor code path produced it.",
   "id": "cell-03"
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "id: shk_a9766c732912\n"
    },
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "  channel        metric          group period    op   value unit source\n0   align    employment  deh=3-4;dgn=1      1  grow  0.0655            \n1   align  unemployment  deh=3-4;dgn=1      1  grow -0.0200            ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>channel</th>\n      <th>metric</th>\n      <th>group</th>\n      <th>period</th>\n      <th>op</th>\n      <th>value</th>\n      <th>unit</th>\n      <th>source</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=3-4;dgn=1</td>\n      <td>1</td>\n      <td>grow</td>\n      <td>0.0655</td>\n      <td></td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>align</td>\n      <td>unemployment</td>\n      <td>deh=3-4;dgn=1</td>\n      <td>1</td>\n      <td>grow</td>\n      <td>-0.0200</td>\n      <td></td>\n      <td></td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 3
    }
   ],
   "source": "from euromod_linking import normalize_shocks\nfrom euromod_linking.shock_table import content_id\n\nrecords = [\n    {\"channel\": \"align\", \"metric\": \"employment\", \"group\": \"dgn=1;deh=3-4\",\n     \"period\": \"1\", \"op\": \"grow\", \"value\": 0.0655},\n    {\"channel\": \"align\", \"metric\": \"unemployment\", \"group\": \"deh=3-4;dgn=1\",\n     \"period\": \"1\", \"op\": \"grow\", \"value\": -0.02},\n]\ntable = normalize_shocks(records)\nprint(\"id:\", content_id(table))\ntable",
   "id": "cell-04"
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "id: shk_a9766c732912 - identical: True\n"
    }
   ],
   "source": "# Same shocks, written in a different order with the group keys swapped.\nshuffled = normalize_shocks(list(reversed([\n    {**records[0], \"group\": \"deh=3-4;dgn=1\"},\n    {**records[1], \"group\": \"dgn=1;deh=3-4\"},\n])))\nprint(\"id:\", content_id(shuffled), \"- identical:\", content_id(shuffled) == content_id(table))",
   "id": "cell-05"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Ingesting an external model's output file\n\nExternal model output is read through a declarative mapping spec rather than bespoke\nparsing code. Here is a workbook in a typical layout: one sheet per quantity, two\nunnamed leading columns holding a region code and an education class, and numbered\ncolumns holding a growth rate per projection period.",
   "id": "cell-06"
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "           c0    c1      c2      1      2      3\n0  employment  BE10     low  0.030  0.040  0.050\n1  employment  BE10  medium  0.031  0.041  0.051\n2  employment  BE10    high  0.032  0.042  0.052\n3  employment  BE21     low  0.031  0.041  0.051\n4  employment  BE21  medium  0.032  0.042  0.052",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>c0</th>\n      <th>c1</th>\n      <th>c2</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>employment</td>\n      <td>BE10</td>\n      <td>low</td>\n      <td>0.030</td>\n      <td>0.040</td>\n      <td>0.050</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>employment</td>\n      <td>BE10</td>\n      <td>medium</td>\n      <td>0.031</td>\n      <td>0.041</td>\n      <td>0.051</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>employment</td>\n      <td>BE10</td>\n      <td>high</td>\n      <td>0.032</td>\n      <td>0.042</td>\n      <td>0.052</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>employment</td>\n      <td>BE21</td>\n      <td>low</td>\n      <td>0.031</td>\n      <td>0.041</td>\n      <td>0.051</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>employment</td>\n      <td>BE21</td>\n      <td>medium</td>\n      <td>0.032</td>\n      <td>0.042</td>\n      <td>0.052</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 5
    }
   ],
   "source": "workbook = os.path.join(os.getcwd(), \"projections.xlsx\")\n\n\ndef sheet(status, base):\n    rows = []\n    for i, region in enumerate([\"BE10\", \"BE21\", \"BE32\", \"AT12\"]):\n        for j, education in enumerate([\"low\", \"medium\", \"high\"]):\n            row = {\"c0\": status, \"c1\": region, \"c2\": education}\n            for p in range(1, 4):\n                row[p] = round(base + 0.01 * p + 0.001 * (i + j), 4)\n            rows.append(row)\n    return pd.DataFrame(rows)\n\n\nwith pd.ExcelWriter(workbook, engine=\"openpyxl\") as xl:\n    sheet(\"employment\", 0.02).to_excel(xl, sheet_name=\"employment\", index=False)\n    sheet(\"unemployment\", -0.01).to_excel(xl, sheet_name=\"unemployment\", index=False)\n\npd.read_excel(workbook, sheet_name=\"employment\").head()",
   "id": "cell-07"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "A spec describes that layout. It can be a YAML file shipped in the package or,\nas here, a plain dict — which is the quickest way to work while a layout is still\nmoving.",
   "id": "cell-08"
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": "spec = {\n    \"mapping_version\": 1,\n    \"name\": \"regional_projections\",\n    \"description\": \"Regional employment and unemployment projections by education class.\",\n    \"reader\": {\"format\": \"excel\", \"sheets\": [\n        {\"sheet\": \"employment\", \"channel\": \"align\", \"metric\": \"employment\"},\n        {\"sheet\": \"unemployment\", \"channel\": \"align\", \"metric\": \"unemployment\"},\n    ]},\n    \"columns\": {\"region_code\": {\"position\": 1}, \"education\": {\"position\": 2}},\n    \"periods\": {\"mode\": \"numbered_columns\", \"range\": [1, 3]},\n    \"value_semantics\": {\"op\": \"grow\", \"unit\": \"rate\"},\n    \"group\": {\n        \"region\": {\"from\": \"region_code\", \"transform\": \"nuts_code\"},\n        \"deh\": {\"from\": \"education\",\n                \"allowed\": [\"low\", \"medium\", \"high\"],\n                \"values\": {\"low\": \"0-2\", \"medium\": \"3-4\", \"high\": \"5-99\"}},\n    },\n    \"filters\": [{\"column\": \"region_code\", \"not_null\": True}],\n}",
   "id": "cell-09"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "Ingesting translates the vocabulary at the model boundary: `medium` becomes\n`deh=3-4`, `BE21` becomes `region=21`, and rows for other countries are filtered out.\nEverything downstream speaks plain EUROMOD variables.",
   "id": "cell-10"
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "54 records\nwarning: Sheet 'employment': 3 rows for other countries skipped (kept BE)\nwarning: Sheet 'unemployment': 3 rows for other countries skipped (kept BE)\n"
    },
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "  channel      metric              group period    op  value  unit                       source\n0   align  employment  deh=0-2;region=10      1  grow  0.030  rate  projections.xlsx#employment\n1   align  employment  deh=0-2;region=10      2  grow  0.040  rate  projections.xlsx#employment\n2   align  employment  deh=0-2;region=10      3  grow  0.050  rate  projections.xlsx#employment\n3   align  employment  deh=0-2;region=21      1  grow  0.031  rate  projections.xlsx#employment\n4   align  employment  deh=0-2;region=21      2  grow  0.041  rate  projections.xlsx#employment\n5   align  employment  deh=0-2;region=21      3  grow  0.051  rate  projections.xlsx#employment\n6   align  employment  deh=0-2;region=32      1  grow  0.032  rate  projections.xlsx#employment\n7   align  employment  deh=0-2;region=32      2  grow  0.042  rate  projections.xlsx#employment",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>channel</th>\n      <th>metric</th>\n      <th>group</th>\n      <th>period</th>\n      <th>op</th>\n      <th>value</th>\n      <th>unit</th>\n      <th>source</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=10</td>\n      <td>1</td>\n      <td>grow</td>\n      <td>0.030</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=10</td>\n      <td>2</td>\n      <td>grow</td>\n      <td>0.040</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=10</td>\n      <td>3</td>\n      <td>grow</td>\n      <td>0.050</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=21</td>\n      <td>1</td>\n      <td>grow</td>\n      <td>0.031</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=21</td>\n      <td>2</td>\n      <td>grow</td>\n      <td>0.041</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=21</td>\n      <td>3</td>\n      <td>grow</td>\n      <td>0.051</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=32</td>\n      <td>1</td>\n      <td>grow</td>\n      <td>0.032</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>align</td>\n      <td>employment</td>\n      <td>deh=0-2;region=32</td>\n      <td>2</td>\n      <td>grow</td>\n      <td>0.042</td>\n      <td>rate</td>\n      <td>projections.xlsx#employment</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 7
    }
   ],
   "source": "from euromod_linking.ingest import ingest\n\ningested, warnings = ingest(workbook, spec, country=\"BE\")\nprint(f\"{len(ingested)} records\")\nfor w in warnings:\n    print(\"warning:\", w)\nnormalize_shocks(ingested).head(8)",
   "id": "cell-11"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Scaling an income list\n\nA `scale` shock can name an EUROMOD income list instead of a single variable. The\nlist is expanded *extension-aware* by walking the model's own `DefIl` definitions, so\nit stays correct when extensions change what the list contains.",
   "id": "cell-12"
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "{'ils_udb_yem': {'scaled': ['yem'], 'skipped_not_in_input': []}}"
     },
     "execution_count": 8
    }
   ],
   "source": "from euromod_linking import apply_scenario\n\nil_scenario = {\n    \"country_code\": \"BE\",\n    \"system_name\": \"BE_2025\",\n    \"shocks\": {\"inline\": [\n        {\"channel\": \"scale\", \"metric\": \"ils_udb_yem\", \"group\": \"\",\n         \"period\": \"1\", \"op\": \"grow\", \"value\": 0.05},\n    ]},\n    \"params\": {\"period\": \"1\"},\n}\n\nplan = apply_scenario(system, data, il_scenario, validate_only=True)\nplan[\"diagnostics\"][\"income_list_expansions\"]",
   "id": "cell-13"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Aligning the population to employment targets\n\n`lma_labour_alignment` moves people between labour-market states until each cell\nmatches an external target. Ask it what it *would* do before paying for the\nalignment itself:",
   "id": "cell-14"
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "methodology: lma_labour_alignment\nadd-ons    : [['LMA', 'LMA_BE']]\nextensions : [['LMA_trans', True]]\n"
    },
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "{'deh=3-4': {'n_rows': 5460, 'n_weighted_all_ages': 8813907.8}}"
     },
     "execution_count": 9
    }
   ],
   "source": "target_scenario = {\n    \"country_code\": \"BE\",\n    \"system_name\": \"BE_2025\",\n    \"shocks\": {\"inline\": [\n        {\"channel\": \"align\", \"metric\": \"employment\", \"group\": \"deh=3-4\",\n         \"period\": \"1\", \"op\": \"grow\", \"value\": 0.0655},\n    ]},\n    \"params\": {\"period\": \"1\"},\n}\n\nplan = apply_scenario(system, data, target_scenario, validate_only=True)\nprint(\"methodology:\", plan[\"methodology\"])\nprint(\"add-ons    :\", plan[\"addons\"])\nprint(\"extensions :\", plan[\"extensions\"])\nplan[\"diagnostics\"][\"cell_population\"]",
   "id": "cell-15"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## When a model cannot run a methodology\n\nThe same scenario against an older system is rejected during validation — before\neither simulation runs, and before the alignment is computed. The LMA add-on does\nnot cover every system in a country.",
   "id": "cell-16"
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "- lma_labour_alignment needs add-on system LMA_BE, but no system of the LMA add-on applies to BE_2001.\n"
    }
   ],
   "source": "from euromod_linking import ScenarioError\n\nold_system = model[\"BE\"][\"BE_2001\"]\nold_scenario = dict(target_scenario, system_name=\"BE_2001\")\n\ntry:\n    apply_scenario(old_system, data, old_scenario, validate_only=True)\nexcept ScenarioError as e:\n    for problem in e.problems:\n        print(\"-\", problem)",
   "id": "cell-17"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "The report behind that message says which requirement was met and which was\nnot:",
   "id": "cell-18"
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "addon      LMA          satisfied=False  (no add-on system LMA_BE applies to BE_2001)\nextension  LMA_trans    satisfied=True  (accepted by this system)\n\nnote: This model looks like J2.54 (from folder-name), at or above the J2.54 floor for lma_labour_alignment — so the release is not what is missing.\n"
    }
   ],
   "source": "from euromod_linking import check_compatibility\n\nreport = check_compatibility(old_system, \"lma_labour_alignment\")\nfor r in report.requirements:\n    print(f\"{r.kind:10} {r.name:12} satisfied={r.satisfied}  ({r.detail})\")\nprint()\nfor note in report.notes:\n    print(\"note:\", note)",
   "id": "cell-19"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "Note what the report separates. The `LMA_trans` extension *is* accepted —\nthis model is new enough — but no LMA add-on system applies to `BE_2001`. Reading\nthose two lines tells you the fix is a different system, not a different model.\n\n## Checking a whole model at once\n\n`compatibility_matrix` answers \"what can I run here\" without writing a scenario.\nPass a country code unless you mean it — walking every country of a full model means\nloading every country.",
   "id": "cell-20"
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "    system           methodology     ok model_release min_model_release\n0  BE_2001  lma_labour_alignment  False         J2.54             J2.54\n1  BE_2001       scale_variables   True         J2.54                  \n2  BE_2005  lma_labour_alignment  False         J2.54             J2.54\n3  BE_2005       scale_variables   True         J2.54                  \n4  BE_2006  lma_labour_alignment  False         J2.54             J2.54\n5  BE_2006       scale_variables   True         J2.54                  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>system</th>\n      <th>methodology</th>\n      <th>ok</th>\n      <th>model_release</th>\n      <th>min_model_release</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>BE_2001</td>\n      <td>lma_labour_alignment</td>\n      <td>False</td>\n      <td>J2.54</td>\n      <td>J2.54</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>BE_2001</td>\n      <td>scale_variables</td>\n      <td>True</td>\n      <td>J2.54</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>BE_2005</td>\n      <td>lma_labour_alignment</td>\n      <td>False</td>\n      <td>J2.54</td>\n      <td>J2.54</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>BE_2005</td>\n      <td>scale_variables</td>\n      <td>True</td>\n      <td>J2.54</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>BE_2006</td>\n      <td>lma_labour_alignment</td>\n      <td>False</td>\n      <td>J2.54</td>\n      <td>J2.54</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>BE_2006</td>\n      <td>scale_variables</td>\n      <td>True</td>\n      <td>J2.54</td>\n      <td></td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 12
    }
   ],
   "source": "from euromod_linking import compatibility_matrix\n\nmatrix = compatibility_matrix(model, \"BE\")\nmatrix[[\"system\", \"methodology\", \"ok\", \"model_release\", \"min_model_release\"]].head(6)",
   "id": "cell-21"
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": "os.remove(workbook)",
   "id": "cell-22"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.3",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}