{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": "# Getting started\n\nThis notebook takes one shock from an external model and turns it into transformed\nEUROMOD input, using nothing but a local model and its own bundled training data.\n\nIt runs top to bottom against any EUROMOD model. Which *methodologies* it can\ndemonstrate depends on the model's release — the first thing we do is ask.", "id": "cell-00" }, { "cell_type": "markdown", "metadata": {}, "source": "## Install\n\n```bash\npip install euromod-linking # or: pip install -e .\npip install euromod-linking[excel] # to ingest .xlsx/.xlsm/.xls model output\n```\n\nPython 3.11+, plus a local EUROMOD model and installation. The model is distributed\nby the JRC and can be requested at\n; point `MODEL_PATH` at the\nfolder you unpack.\n\nThis targets the JRC's EUROMOD model for the 27 EU member states. Models built on\nthe same engine — SOUTHMOD, SWISSMOD, and national models outside the EU — use\ndifferent variable and income-list conventions and will not work here.", "id": "cell-01" }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": "import os\n\nimport pandas as pd\n\n# Point this at your own EUROMOD model folder, or set EUROMOD_MODEL_PATH.\nMODEL_PATH = os.environ.get(\"EUROMOD_MODEL_PATH\", r\"C:\\EUROMOD_RELEASES\")\nDATA_DIR = os.path.join(MODEL_PATH, \"Input\")\n\n# Keep DataFrame previews compact in the rendered docs.\npd.set_option(\"display.max_columns\", 10)\npd.set_option(\"display.width\", 110)", "id": "cell-02" }, { "cell_type": "markdown", "metadata": {}, "source": "## 1. What can this model run?\n\nA methodology needs things from the model — `lma_labour_alignment` needs the LMA\nadd-on and the `LMA_trans` extension switch. The engine does not complain when they\nare missing; it just quietly does nothing. So ask first.", "id": "cell-03" }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "engine : 3.8.8\nrelease: ('J2.54', 'folder-name')\n" } ], "source": "from euromod import Model\n\nfrom euromod_linking import compatibility_matrix\nfrom euromod_linking.compat import model_release\n\nmodel = Model(MODEL_PATH)\nprint(\"engine :\", model.software_version)\nprint(\"release:\", model_release(MODEL_PATH))", "id": "cell-04" }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "output_type": "execute_result", "metadata": {}, "data": { "text/plain": " methodology ok model_release min_model_release\n42 lma_labour_alignment True J2.54 J2.54\n43 scale_variables True J2.54 ", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
methodologyokmodel_releasemin_model_release
42lma_labour_alignmentTrueJ2.54J2.54
43scale_variablesTrueJ2.54
\n
" }, "execution_count": 3 } ], "source": "matrix = compatibility_matrix(model, \"BE\")\nmatrix[matrix[\"system\"] == \"BE_2025\"][\n [\"methodology\", \"ok\", \"model_release\", \"min_model_release\"]\n]", "id": "cell-05" }, { "cell_type": "markdown", "metadata": {}, "source": "The release is read from the folder name, which is a best-effort guess — see\n[Model compatibility](../concepts/compatibility.md) for the full detection ladder and\nwhy a missing release is never treated as a failure.\n\nWhat actually decides is the capability check: whether the model exposes the add-ons\nand extension switches a methodology declares. Ask it about one methodology and it\nlists them:", "id": "cell-06" }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "lma_labour_alignment on BE/BE_2025: ok\n note: Could not verify the LMA_trans extension: this model's extension names could not be read.\n addon LMA satisfied=True\n extension LMA_trans satisfied=None\n" } ], "source": "from euromod_linking import check_compatibility\n\nsystem = model[\"BE\"][\"BE_2025\"]\nreport = check_compatibility(system, \"lma_labour_alignment\")\nprint(report)\nfor r in report.requirements:\n print(f\" {r.kind:10} {r.name:12} satisfied={r.satisfied}\")", "id": "cell-07" }, { "cell_type": "markdown", "metadata": {}, "source": "## 2. Load the microdata\n\nThe model ships training data for every country, which is enough to demonstrate the\ntransform. A real study uses the survey microdata for the country.", "id": "cell-08" }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "7,482 rows, 11,322,391 weighted people\n" }, { "output_type": "execute_result", "metadata": {}, "data": { "text/plain": " idhh idperson dwt dag dgn deh yem\n0 1 101 42302.102 40 0 3 0.00000\n1 2 201 50581.000 40 0 3 0.00000\n2 3 301 23908.301 40 0 3 178.41209\n3 4 401 23908.301 40 0 3 713.64838\n4 5 501 14644.400 40 0 3 1248.88500", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
idhhidpersondwtdagdgndehyem
0110142302.10240030.00000
1220150581.00040030.00000
2330123908.3014003178.41209
3440123908.3014003713.64838
4550114644.40040031248.88500
\n
" }, "execution_count": 5 } ], "source": "data = pd.read_csv(os.path.join(DATA_DIR, \"BE_training_data.txt\"), sep=\"\\t\")\nprint(f\"{len(data):,} rows, {data['dwt'].sum():,.0f} weighted people\")\ndata[[\"idhh\", \"idperson\", \"dwt\", \"dag\", \"dgn\", \"deh\", \"yem\"]].head()", "id": "cell-09" }, { "cell_type": "markdown", "metadata": {}, "source": "## 3. Write a scenario\n\nA scenario binds shocks to a country and system. This one raises employment income\nby 3% for people with medium education — a `scale` shock, which selects the\n`scale_variables` methodology.\n\nNote what is *not* here: the methodology. It is resolved from the shock's channel and\nechoed back, so a scenario cannot silently be handled by a different method than the\none that produced an earlier result.", "id": "cell-10" }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": "scenario = {\n \"country_code\": \"BE\",\n \"system_name\": \"BE_2025\",\n \"shocks\": {\"inline\": [\n {\"channel\": \"scale\", \"metric\": \"yem\", \"group\": \"deh=3-4\",\n \"period\": \"1\", \"op\": \"grow\", \"value\": 0.03},\n ]},\n \"params\": {\"period\": \"1\"},\n}", "id": "cell-11" }, { "cell_type": "markdown", "metadata": {}, "source": "## 4. Look before you leap\n\n`validate_only=True` stops before the expensive work and reports what the scenario\n*would* do: which people each cell resolves to, and what the methodology would target.\nThis is where you find out a shock is the wrong size, while it is still cheap to fix.", "id": "cell-12" }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": "methodology: scale_variables\nadd-ons : [] | extensions: []\n" }, { "output_type": "execute_result", "metadata": {}, "data": { "text/plain": "{'deh=3-4': {'n_rows': 5460, 'n_weighted_all_ages': 8813907.8}}" }, "execution_count": 7 } ], "source": "from euromod_linking import apply_scenario\n\nplan = apply_scenario(system, data, scenario, validate_only=True)\nprint(\"methodology:\", plan[\"methodology\"])\nprint(\"add-ons :\", plan[\"addons\"], \"| extensions:\", plan[\"extensions\"])\nplan[\"diagnostics\"][\"cell_population\"]", "id": "cell-13" }, { "cell_type": "markdown", "metadata": {}, "source": "## 5. Apply it\n\nNow the real transform. It returns the counterfactual input and, when the methodology\nrestructures rows, a matching baseline built on the same rows. `scale_variables`\nadds and removes no rows, so the baseline here is the untouched `data`.", "id": "cell-14" }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "output_type": "execute_result", "metadata": {}, "data": { "text/plain": " rows rows whose yem changed mean yem before mean yem after\ndeh=3-4 (shocked) 5460 2744 1838.102132 1893.245196\noutside the cell 2022 0 0.000000 0.000000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
rowsrows whose yem changedmean yem beforemean yem after
deh=3-4 (shocked)546027441838.1021321893.245196
outside the cell202200.0000000.000000
\n
" }, "execution_count": 8 } ], "source": "plan = apply_scenario(system, data, scenario)\ncounterfactual = plan[\"counterfactual\"]\n\ncell = data[\"deh\"].between(3, 4)\nmoved = counterfactual[\"yem\"] != data[\"yem\"]\n\npd.DataFrame({\n \"rows\": [cell.sum(), (~cell).sum()],\n \"rows whose yem changed\": [(moved & cell).sum(), (moved & ~cell).sum()],\n \"mean yem before\": [data.loc[cell, \"yem\"].mean(), data.loc[~cell, \"yem\"].mean()],\n \"mean yem after\": [counterfactual.loc[cell, \"yem\"].mean(),\n counterfactual.loc[~cell, \"yem\"].mean()],\n}, index=[\"deh=3-4 (shocked)\", \"outside the cell\"])", "id": "cell-15" }, { "cell_type": "markdown", "metadata": {}, "source": "Inside the cell, mean employment income is up 3%. Outside it, not a single\nrow changed. That is the whole contract of a `scale` shock.\n\n(The model's bundled training data is a simplified extract — every earner in it\nhappens to sit at `deh=3` — so the second row has no earnings to move. Real survey\nmicrodata spreads across the education range.)\n\n## 6. Run both halves\n\n`run_scenario` does everything above and then runs both simulations through the\nengine, so the difference between them isolates the shock:\n\n```python\nfrom euromod_linking import run_scenario\n\nout = run_scenario(system, scenario, input_path=DATA_DIR)\nout[\"baseline_output\"], out[\"counterfactual_output\"]\n```\n\nIt is not executed here because a full simulation takes minutes and its output is\nlarge. Everything up to this point is pure pandas.\n\n## Where to go next\n\n- [The shock table](../concepts/shock-table.md) — the one format every external model\n is normalised into\n- [Population cells](../concepts/population-cells.md) — how `group` selects people\n- [Scenario documents](../concepts/scenarios.md) — the constants distinction that is\n easy to get wrong\n- [Adapters](../methods/index.md) — what each methodology does and how to use it\n- [Examples](examples.ipynb) — ingesting a model output file, income lists, and what\n an unsupported methodology looks like", "id": "cell-16" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.12.3", "mimetype": "text/x-python", "codemirror_mode": { "name": "ipython", "version": 3 }, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py" } }, "nbformat": 4, "nbformat_minor": 5 }