building from source task

[pxp9]

the purpose from this task is to make easy to test Elixir AtomVM with main branch code from the AtomVM repository for ESP32.

this task requires idf.py installed or docker for compiling the AtomVM from source.

feel free to comment any issue in the implementation as well.

Thank you for your attention.

Workflow with custom build

mix atomvm.esp32.build --atomvm-path ./_build/atomvm_source/AtomVM/ --use-docker --chip esp32s3

mix atomvm.esp32.install --image ./_build/atomvm_source/AtomVM/src/platforms/esp32/build/atomvm-esp32s3.img

Using ESP-IDF Docker container

[UncleGrumpy]

This is a very cool feature! One that has been on my todo list to add to the atomvm_rebar3_plugin for a while. I work much more in Erlang, so my Elixir is not that great, but I did spot a few problems, and have some suggestions on improving usability.

[UncleGrumpy]

It might also be good to consider allowing a ESP-IDF docker image. I suspect many more Elixir developers will already have docker installed (or at least be more familiar with it than ESP-IDF), so it would be extra cool to pull in a docker image of ESP-IDF if they don't already have it installed.

[pxp9]

Hey @UncleGrumpy , when i try to build AtomVM , the build fails because we are using the the stat struct but the header <sys/stat.h> is missing, is there something i need to add in the args of cmake to build it correctly or is this a bug in AtomVM main branch build ?

[913/1054] Building C object esp-idf/avm_sys/CMakeFiles/__idf_avm_sys.dir/sys.c.obj
  FAILED: [code=1] esp-idf/avm_sys/CMakeFiles/__idf_avm_sys.dir/sys.c.obj

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c: In function 'sys_open_avm_from_file':

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c:413:21: error: storage size of 'file_stats' isn't known
    413 |         struct stat file_stats;
        |                     ^~~~~~~~~~

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c:414:22: error: implicit declaration of function 'stat' [-Wimplicit-function-declaration]
    414 |         if (UNLIKELY(stat(path, &file_stats) < 0)) {
        |                      ^~~~

  /home/pxp9/Programming/try_nerves/atom_vm_firmware/esp32_s3/_build/atomvm_source/AtomVM/src/platforms/esp32/components/avm_sys/sys.c:413:21: warning: unused variable 'file_stats' [-Wunused-variable]
    413 |         struct stat file_stats;
        |                     ^~~~~~~~~~

  Build stopped:
  ninja: build stopped: subcommand failed.

[pxp9]

Added docker feat based on this

https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/tools/idf-docker-image.html#building-a-project-with-cmake

[UncleGrumpy]

Hey @UncleGrumpy , when i try to build AtomVM , the build fails because we are using the the stat struct but the header <sys/stat.h> is missing, is there something i need to add in the args of cmake to build it correctly or is this a bug in AtomVM main branch build ?

This is not something I have seen before, and I have been working almost exclusively with main branch for at least the last few months.

Was this testing a docker build, or did this happen with an installed ESP-IDF? It has been a few ESP-IDF releases since I last used an esp-idf docker image, although we do use them in the AtomVM CI, so I don’t think that would be the problem.

[pxp9]

@UncleGrumpy , this issue from the missing header happened with both toolchains. Esp idf installed and docker

[UncleGrumpy]

Interesting. Which ESP-IDF version? I think I am a point release behind locally (still using 5.5.1). I will try updating my toolchain this evening and see if I can reproduce the problem, I believe the AtomVM CI may also need to be bumped to 5.5.2, I will look into that too. This may be a new issue that you were the first to discover. I think you should open an issue in the AtomVM repo, and be sure to include Host OS, and ESP-IDF version in the report, along with the error and any other details that might be relevant.

[pxp9]

I tried with the last ESP version 6.X and then I tried with 5.4.1 which afaik is the last supported by AtomVM but not sure. It happens the same with both versions.

[UncleGrumpy]

I tried with the last ESP version 6.X and then I tried with 5.4.1 which afaik is the last supported by AtomVM but not sure.

I have not tried 6.0 yet, but I believe a PR was recently merged to allow ESP-IDF 6.0 builds. ESP-IDF 5.5.1 is definitely working for main branch, we might not have updated the main branch docs to reflect this, that is something we will need to make sure is up to date before a 0.7.0 release of AtomVM. Since you are hitting this error with ESP-IDF 5.4.1 and 6.0 then I don’t think just switching to 5.5.x will be any different.

[pxp9]

Okay @UncleGrumpy , today I will post an issue in AtomVM with this error and how to reproduce it.

Meanwhile could you review the code and see if everything is fine ?

[pxp9]

Hey @UncleGrumpy , I tried 4 times to compile the AtomVM cloning each time just in case and it seems to be fixed.

this command but with each version

mix atomvm.esp32.build --use-docker --chip esp32s3 --idf-version latest

1. I tried with v5.4.1 ESP-IDF container and it worked
2. I tried with v5.5.1 ESP-IDF container and it worked
3. I tried with latest (v6.1) ESP-IDF container and it did not work
4. I tried with (the tag release-v6.0) ESP-IDF container and it did not work

it seems i had something weird in the cache or it has been fixed.

by default --use-docker uses the version v5.4.1

[UncleGrumpy]

Hey @UncleGrumpy , I tried 4 times to compile the AtomVM cloning each time just in case and it seems to be fixed.

this command but with each version

mix atomvm.esp32.build --use-docker --chip esp32s3 --idf-version latest

1. I tried with v5.4.1 ESP-IDF container and it worked
2. I tried with v5.5.1 ESP-IDF container and it worked
3. I tried with latest (v6.1) ESP-IDF container and it did not work
4. I tried with (the tag release-v6.0) ESP-IDF container and it did not work

it seems i had something weird in the cache or it has been fixed.

by default --use-docker uses the version v5.4.1

That is great! I would not necessarily expect any of the 6.x IDF versions to work at the moment. I think AtomVM itself will need some updating. If it works with stable releases from 5.1 (or 5.2) to 5.5 that is all that would be expected at the moment. The AtomVM main branch uses 5.5.x for release builds so that should probably be the default here too. I believe 5.5.2 is the latest, we still need to bump that in the AtomVM CI, which I believe is still using 5.5.1. There shouldn’t be any breakage between these versions since the point releases are usually just bug fixes.

[UncleGrumpy]

Meanwhile could you review the code and see if everything is fine ?

I will take another look this evening, but I think you will get some much more useful feedback when some of devs that work more in Elixir have a chance to review this, especially @bettio and @petermm.

[UncleGrumpy]

i was thinking we can add a guide of how to install the ESP-IDF in nix flake since it is a reproduceable environment is cool. (for telling the users who want to build old releases how to make them).

This would be great, but this repo would not be the best place for that. I would suggest either submitting “Tutorial” here:
https://atomvm.org/tutorials/

This can be done from the atomvm/atomvm.org repo. I would add a new page and link to it from the Tutorials page.

Or you could add a new section to the end of the esp32 build instructions here:
https://doc.atomvm.org/main/build-instructions.html#building-for-esp32

Which is generated from atomvm/AtomVM/doc/src/build-instructions.md.

[pxp9]

i was thinking we can add a guide of how to install the ESP-IDF in nix flake since it is a reproduceable environment is cool. (for telling the users who want to build old releases how to make them).

This would be great, but this repo would not be the best place for that. I would suggest either submitting “Tutorial” here: https://atomvm.org/tutorials/

This can be done from the atomvm/atomvm.org repo. I would add a new page and link to it from the Tutorials page.

Or you could add a new section to the end of the esp32 build instructions here: https://doc.atomvm.org/main/build-instructions.html#building-for-esp32

Which is generated from atomvm/AtomVM/doc/src/build-instructions.md.

@UncleGrumpy

i believe tutorials section is better, do you think this is ready to be merged ?

[pxp9]

@bettio , could you review please ?

[petermm]

This is so great @pxp9 we really want to land it - will you have the bandwidth soon, or do you want me to land it for you?

I'm thinking something like https://github.com/petermm/ExAtomVM/tree/complete-docker-build

[pxp9]

This is so great @pxp9 we really want to land it - will you have the bandwidth soon, or do you want me to land it for you?

I'm thinking something like https://github.com/petermm/ExAtomVM/tree/complete-docker-build

hey , thanks for reaching, i totally forgot about this, i work on so many things , i will work on this , i like your commits so i will take them and put them in this PR

you can reach me in telegram as well , I am in the AtomVM community group , i am also in the AtomVm discord as well.

[pxp9]

i feel we should generate like a default idf_components.yml, user might miss this feature of personalizing components in the AtomVM.

what do you think @petermm ?

[pxp9]

Also i noticed that PR flag , collides with ref flag , the purpose of the ref flag is to take AtomVM from whatever commit or branch and it should work , maybe we should do the git fetch when ref is a remote branch ?

what do you think @petermm ?

[petermm]

I've reverted the --pr option and made the --ref handle PRs instead.

I've added idf_component.yml.example - if user has local path setup with idf_component etc. we don't want to erase too easily

I added support for --chip esp32s2,esp32c3 so it builds multiple artifacts for the chips - nice for CI etc.

Believe we are more than good to go, pick the last commits, and then we can do a final review - there will always be edge cases (like if local path ask confirmation before we overwrite the idf_component.yml) etc.

[pxp9]

@petermm i will check and merge your changes into my PR

[pxp9]

@petermm added your changes here