including multiple files which contain ```jupyter: python3``` in the frontmatter block AND some actual code blocks, will print the content of subsequent the frontmatter block

[kompre]

Bug description

I bumped into an error upgrading to version 1.5.54 that regards including multiple qmd files with the {{< include >}} shortcode.

When I include multiple qmd files which contains in the raw block / frontamatter block at least the jupyter keyword, AND also a code block, then the raw block will be printed when rendered.

Here the link at the repo with a mwe

In summary the gist is this:

• a couple of files are included into the main.qmd

# Preface {.unnumbered}

This is a Quarto book.

To learn more about Quarto books visit <https://quarto.org/docs/books>.


{{< include included_file.qmd >}}

{{< include included_file_2.qmd >}}

• both included file have frontmatter/raw block

---
jupyter: python3
---


## Included File

a code block will trigger the error.

commenting the code block will not trigger the error

```{python}
x=1
```

---
jupyter: python3
---

## Included File 2

some text

The rendered result look like this:

I find out this working with the book type, where I may split a chapter into multiple section/files to be more manageable. Those files are usually qmd files converted from a ipynb so the frontmatter is written by default by the quarto convert cli.

In previous version (1.4.554) this was not happening, and I rely pretty much on the quarto convert tool to be able to work with ipynb, but actually render the converted qmd file. I've set up a pre-render command that will automatically convert each ipynb files in the folder into the corresponding qmd file, so I would appreciate if I don't need to edit again the converted qmd file, because conversion could happen several time during the writing phase.

Steps to reproduce

1. render the provided document at the link
2. comment out the code block in included_file.qmd to trigger the correct behavior (no raw matter printed)

Expected behavior

the content of the raw block should not be rendered, as it was the case with previous version of quarto.

Actual behavior

the content of the raw block get rendered into the final document

Your environment

• win11
• vscode

Quarto check output

Quarto 1.5.55
[>] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[>] Checking versions of quarto dependencies......OK
[>] Checking Quarto installation......OK
      Version: 1.5.55
      Path: C:\Program Files\Quarto\bin
      CodePage: 1252

[>] Checking tools....................OK
      TinyTeX: v2024.07.03
      Chromium: (not installed)

[>] Checking LaTeX....................OK
      Using: TinyTex
      Path: C:\Users\s.follador\AppData\Roaming\TinyTeX\bin\windows\
      Version: 2024

[>] Checking basic markdown render....OK

[>] Checking Python 3 installation....OK
      Version: 3.12.4
      Path: C:/Users/s.follador/AppData/Local/pypoetry/Cache/virtualenvs/quarto-test-hTEtzXvo-py3.12/Scripts/python.exe
      Jupyter: 5.7.2
      Kernels: python3

[>] Checking Jupyter engine render....OK

[>] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

[kompre]

problem present also with version 1.6.4

downgrading to 1.4.554 in the meanwhile

[mcanouil]

That's not a bug with include.
This is clearly stated in the documentation in a red callout: https://quarto.org/docs/authoring/includes.html

Include shortcodes are equivalent to copying and pasting the text from the included file into the main file. This means that relative references (links, images, etc.) inside the included file resolve based on the directory of the main file not the included file.

It also means that if the included file has a metadata block, that block will take effect in all included files. In most cases, having metadata blocks in an included file will cause unexpected behaviour.

The difference with previous Quarto versions, seems to be that the included metadata block is not merged but used as is as plain text.

A simpler and smaller reprex:

Input	Output
--- title: "Quarto Playground" format: html --- {{< include _demo.qmd >}}	--- whatever: "Hello, World!" --- ```{python} print("Hello, World!") ```

And since include is a copy/paste:

Input	Output
--- title: "Quarto Playground" format: html --- --- whatever: "Hello, World!" --- ```{python} print("Hello, World!") ```

Now the real question/issue is what changed which lead to metadata block to no longer be read as such when not at the top of a document? I can reproduce the change of behaviour between 1.4 and 1.5 (I'll make a bisect "later").

Side note: It makes not much sense to set an engine in a file used in the include shortcode as in the end and as documented, there can be only one engine in a document.
I believe you want embed shortcode here or if you do want include then don't set the engine.

[mcanouil]

That's "funny". The issue only appears with Jupyter as the engine.

[cscheid]

It looks from here like it's a problem with the way we/Pandoc attempts to distinguish between horizontal rules (---) and metadata block delimiters (also ---... but with some rules about the context in which it can appear)

[mcanouil]

The faulty commit/regression: 86122d8

FYI, slide delimiter and horizontal lines are supposed to be surrounded by empty lines (source: Pandoc).

[kompre]

It makes not much sense to set an engine in a file used in the include shortcode

I don't actually add the jupyter: python3 myself, but it is the result of using quarto convert mydoc.ipynb --output mydoc.qmd

Let me better explain my workflow:

1. I drafts on ipynb (easier for me since I have lots of code cells)
2. convert ipynb to qmd
3. render main qmd file with the included converted qmd files
4. repeat from step one as necessary

In my workflow I don't actually directly edit the converted qmd files, as it would become too much burdensome doing it each time. (I set up a script that will convert any matching ipynb file to qmd, to be invoked at the pre-ender step when rendering the quarto document).

Why not use the ipynb document directly?

Well I actually want the document to be re-run at rendering time, starting from a qmd file, because I need to work around some vscode idiosyncrasies. For example vscode uses the katex engine to render latex expressions in the ipynb, but the katex engine sometimes has problems with some valid latex expressions and will fail to render, while the quarto latex engine will render them fine (e.g. the \label{} command).

In order to fix this very specific unwanted behavior, I usually insert an include shortcode in the ipynb, which contains some sensible config options that will run only at rendering time if the file called upon is a qmd, meanwhile editing an ipynb file, include shortcode have not effect.

I believe you want embed shortcode

I noticed that the difference between an embed and an include is that the embed will run the file in its own namespace, separate from the "main" file's own, while the include will run it into the "main" file namespace, thus the results being accessible for further evaluations after the include being executed. Sometimes I rely on the latter behavior, the embed is not always a viable option in my use case.

[kompre]

Hi,

I tested with the latest stable release 1.6.39 and the issue is still present.