diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs index c5039f58b54dc..195e1e06eea46 100644 --- a/.git-blame-ignore-revs +++ b/.git-blame-ignore-revs @@ -1,3 +1,36 @@ +# all: Prune trailing whitespace. +dda9b9c6da5d3c31fa8769e581a753e95a270803 + +# all: Remove the "STATIC" macro and just use "static" instead. +decf8e6a8bb940d5829ca3296790631fcece7b21 + +# renesas-ra: Fix spelling mistakes found by codespell. +b3f2f18f927fa2fad10daf63d8c391331f5edf58 + +# all: Update Python formatting to ruff-format. +bbd8760bd9a2302e5abee29db279102bb11d7732 + +# all: Fix various spelling mistakes found by codespell 2.2.6. +cf490a70917a1b2d38ba9b58e763e0837d0f7ca7 + +# all: Fix spelling mistakes based on codespell check. +b1229efbd1509654dec6053865ab828d769e29db + +# top: Update Python formatting to black "2023 stable style". +8b2748269244304854b3462cb8902952b4dcb892 + +# all: Reformat remaining C code that doesn't have a space after a comma. +5b700b0af90591d6b1a2c087bb8de6b7f1bfdd2d + +# ports: Reformat more C and Python source code. +5c32111fa0e31e451b0f1666bdf926be2fdfd82c + +# all: Update Python formatting to latest Black version 22.1.0. +ab2923dfa1174dc177f0a90cb00a7e4ff87958d2 + +# all: Update Python formatting to latest Black version 21.12b0. +3770fab33449a5dadf8eb06edfae0767e75320a6 + # tools/gen-cpydiff.py: Fix formatting of doc strings for new Black. 0f78c36c5aa458a954eed39a46942209107a553e diff --git a/.gitattributes b/.gitattributes index e6d31d6aa31fc..2d8496db50488 100644 --- a/.gitattributes +++ b/.gitattributes @@ -8,10 +8,12 @@ # These are binary so should never be modified by git. *.a binary +*.ico binary *.png binary *.jpg binary *.dxf binary *.mpy binary +*.der binary # These should also not be modified by git. tests/basics/string_cr_conversion.py -text diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000000000..1ec6c7067f9cd --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,109 @@ +name: Bug report +description: Report a bug or unexpected behaviour +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Please provide as much detail as you can, it really helps us find and fix bugs faster. + + #### Not a bug report? + + * If you have a question \"How Do I ...?\", please post it on [GitHub Discussions](https://github.com/orgs/micropython/discussions/) or [Discord](https://discord.gg/RB8HZSAExQ) instead of here. + * For missing or incorrect documentation, or feature requests, then please [choose a different issue type](https://github.com/micropython/micropython/issues/new/choose). + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) matching this bug before reporting. + - type: input + id: port-board-hw + attributes: + label: Port, board and/or hardware + description: | + Which MicroPython port(s) and board(s) are you using? + placeholder: | + esp32 port, ESP32-Fantastic board. + validations: + required: true + - type: textarea + id: version + attributes: + label: MicroPython version + description: | + To find the version: + + 1. Open a serial REPL. + 2. Type Ctrl-B to see the startup message. + 3. Copy-paste that output here. + + If the issue is about building MicroPython, please provide output of `git describe --dirty` and as much information as possible about the build environment. + + If the version or configuration is modified from the official MicroPython releases or the master branch, please tell us the details of this as well. + placeholder: | + MicroPython v6.28.3 on 2029-01-23; PyBoard 9 with STM32F9 + validations: + required: true + - type: textarea + id: steps-reproduce + attributes: + label: Reproduction + description: | + What steps will reproduce the problem? Please include all details that could be relevant about the environment, configuration, etc. + + If there is Python code to reproduce this issue then please either: + a. Type it into a code block below ([code block guide](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks)), or + b. Post longer code to a [GitHub gist](https://gist.github.com/), or + c. Create a sample project on GitHub. + + For build issues, please provide the exact build commands that you ran. + placeholder: | + 1. Copy paste the code provided below into a new file + 2. Use `mpremote run` to execute it on the board. + validations: + required: true + - type: textarea + id: expected + attributes: + label: Expected behaviour + description: | + What did you expect MicroPython to do? If comparing output with CPython or a different MicroPython port/version then please provide that output here. + placeholder: | + Expected to print "Hello World". + + Here is the correct output, seen with previous MicroPython version v3.14.159: + + > [...] + - type: textarea + id: what-happened + attributes: + label: Observed behaviour + description: | + What actually happened? Where possible please paste exact output, or the complete build log, etc. Very long output can be linked in a [GitHub gist](https://gist.github.com/). + placeholder: | + This unexpected exception appears: + + > [...] + validations: + required: true + - type: textarea + id: additional + attributes: + label: Additional Information + description: | + Is there anything else that might help to resolve this issue? + value: No, I've provided everything above. + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true + - type: markdown + attributes: + value: | + Thanks for taking the time to help improve MicroPython. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000000000..f3662464da337 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,11 @@ +blank_issues_enabled: false +contact_links: + - name: MicroPython GitHub Discussions + url: https://github.com/orgs/micropython/discussions + about: Community discussion about all things MicroPython. This is the best place to start if you have questions about using MicroPython or getting started with MicroPython development. + - name: MicroPython Documentation + url: https://docs.micropython.org/ + about: Documentation for using and working with MicroPython and libraries. + - name: MicroPython Downloads + url: https://micropython.org/download/ + about: Pre-built firmware and information for most supported boards. diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml new file mode 100644 index 0000000000000..93051e51c80c7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -0,0 +1,46 @@ +name: Documentation issue +description: Report areas of the documentation or examples that need improvement +title: "docs: " +labels: ["documentation"] +body: + - type: markdown + attributes: + value: | + This form is for reporting issues with the documentation or examples provided with MicroPython. + + If you have a general question \"How Do I ...?\", please post it on [GitHub Discussions](https://github.com/orgs/micropython/discussions/) or [Discord](https://discord.gg/RB8HZSAExQ) instead of here. + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) before reporting a new one. + - type: input + id: page + attributes: + label: Documentation URL + description: | + Does this issue relate to a particular page in the [online documentation](https://docs.micropython.org/en/latest/)? If yes, please paste the URL of the page: + placeholder: | + https://docs.micropython.org/en/latest/ + - type: textarea + id: version + attributes: + label: Description + description: | + Please describe what was missing from the documentation and/or what was incorrect/incomplete. + validations: + required: true + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true + - type: markdown + attributes: + value: | + Thanks for taking the time to help improve MicroPython. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000000000..7d5162a32a4af --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,74 @@ +name: Feature request +description: Request a feature or improvement +labels: ['enhancement'] +body: + - type: markdown + attributes: + value: | + This form is for requesting features or improvements in MicroPython. + + #### Get feedback first + + Before submitting a new feature idea here, suggest starting a discussion on [Discord](https://discord.gg/RB8HZSAExQ) or [GitHub Discussions](https://github.com/orgs/micropython/discussions/) to get early feedback from the community and maintainers. + + #### Not a MicroPython core feature? + + * If you have a question \"How Do I ...?\", please post it on GitHub Discussions or Discord instead of here. + * Could this feature be implemented as a pure Python library? If so, please open the request on the [micropython-lib repository](https://github.com/micropython/micropython-lib/issues) instead. + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) before opening a new one. + - type: textarea + id: feature + attributes: + label: Description + description: | + Describe the feature you'd like to see added to MicroPython. What does this feature enable and why is it useful? + + * For core Python features, where possible please include a link to the relevant PEP or CPython documentation. + * For new architectures / ports / boards, please provide links to relevant documentation, specifications, and toolchains. Any information about the popularity and unique features about this hardware would also be useful. + * For features for existing ports (e.g. new peripherals or microcontroller features), please describe which port(s) it applies to, and whether this is could be an extension to the machine API or a port-specific module? + * For drivers (e.g. for external hardware), please link to datasheets and/or existing drivers from other sources. + + If there is an existing discussion somewhere about this feature, please add a link to it as well. + validations: + required: true + - type: textarea + id: size + attributes: + label: Code Size + description: | + MicroPython aims to strike a balance between functionality and code size. Can this feature be optionally enabled? + + If you believe the usefulness of this feature would outweigh the additional code size, please explain. (It's OK to say you're unsure here, we're happy to discuss this with you.) + - type: dropdown + id: implementation + attributes: + label: Implementation + description: | + What is your suggestion for implementing this feature? + + (See also: [How to sponsor](https://github.com/sponsors/micropython#sponsors), [How to submit a Pull Request](https://github.com/micropython/micropython/wiki/ContributorGuidelines).) + options: + - I hope the MicroPython maintainers or community will implement this feature + - I intend to implement this feature and would submit a Pull Request if desirable + - I would like to sponsor development of this feature + multiple: true + validations: + required: true + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true + - type: markdown + attributes: + value: | + Thanks for taking the time to suggest improvements for MicroPython. diff --git a/.github/ISSUE_TEMPLATE/security.yml b/.github/ISSUE_TEMPLATE/security.yml new file mode 100644 index 0000000000000..57c2a5885ed85 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/security.yml @@ -0,0 +1,60 @@ +name: Security report +description: Report a security issue or vulnerability in MicroPython +labels: ["security"] +body: + - type: markdown + attributes: + value: | + This form is for reporting security issues in MicroPython that are not readily exploitable. + + 1. For issues that are readily exploitable or have high impact, please email contact@micropython.org instead. + 1. If this is a question about security, please ask it in [Discussions](https://github.com/orgs/micropython/discussions/) or [Discord](https://discord.gg/RB8HZSAExQ) instead. + + #### Existing issue? + + * Please search for [existing issues](https://github.com/micropython/micropython/issues) before reporting a new one. + + - type: input + id: port-board-hw + attributes: + label: Port, board and/or hardware + description: | + Which MicroPython port(s) and board(s) are you using? + placeholder: | + esp32 port, ESP32-Duper board. + - type: textarea + id: version + attributes: + label: MicroPython version + description: | + To find the version: + + 1. Open a serial REPL. + 2. Type Ctrl-B to see the startup message. + 3. Copy-paste that output here. + + If the version or configuration is modified from the official MicroPython releases or the master branch, please tell us the details of this as well. + placeholder: | + MicroPython v6.28.3 on 2029-01-23; PyBoard 9 with STM32F9 + - type: textarea + id: report + attributes: + label: Issue Report + description: | + Please provide a clear and concise description of the security issue. + + * What does this issue allow an attacker to do? + * How does the attacker exploit this issue? + validations: + required: true + - type: dropdown + id: code-of-conduct + attributes: + label: Code of Conduct + description: | + Do you agree to follow the MicroPython [Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md) to ensure a safe and respectful space for everyone? + options: + - "Yes, I agree" + multiple: true + validations: + required: true diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000000000..2c7d1708395e2 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,7 @@ +version: 2 +updates: + # Maintain dependencies for GitHub Actions + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "daily" diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000000..e11cebddb3710 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,33 @@ + + +### Summary + + + + +### Testing + + + + +### Trade-offs and Alternatives + + + diff --git a/.github/workflows/biome.yml b/.github/workflows/biome.yml new file mode 100644 index 0000000000000..88744f16ca7d6 --- /dev/null +++ b/.github/workflows/biome.yml @@ -0,0 +1,16 @@ +name: JavaScript code lint and formatting with Biome + +on: [push, pull_request] + +jobs: + eslint: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Biome + uses: biomejs/setup-biome@v2 + with: + version: 1.5.3 + - name: Run Biome + run: biome ci --indent-style=space --indent-width=4 tests/ ports/webassembly diff --git a/.github/workflows/code_formatting.yml b/.github/workflows/code_formatting.yml index aab347d78e97d..9f30f048cfdbe 100644 --- a/.github/workflows/code_formatting.yml +++ b/.github/workflows/code_formatting.yml @@ -2,15 +2,19 @@ name: Check code formatting on: [push, pull_request] +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: - build: - runs-on: ubuntu-latest + code-formatting: + runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v2 - - uses: actions/setup-python@v1 + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 - name: Install packages - run: source tools/ci.sh && ci_code_formatting_setup + run: source tools/ci.sh && ci_c_code_formatting_setup - name: Run code formatting - run: source tools/ci.sh && ci_code_formatting_run + run: source tools/ci.sh && ci_c_code_formatting_run - name: Check code formatting run: git diff --exit-code diff --git a/.github/workflows/code_size.yml b/.github/workflows/code_size.yml index 7570261e7d23b..67261933798cb 100644 --- a/.github/workflows/code_size.yml +++ b/.github/workflows/code_size.yml @@ -1,22 +1,31 @@ name: Check code size on: - push: pull_request: paths: - '.github/workflows/*.yml' - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'ports/bare-arm/**' + - 'ports/mimxrt/**' - 'ports/minimal/**' + - 'ports/rp2/**' + - 'ports/samd/**' + - 'ports/stm32/**' + - 'ports/unix/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true jobs: build: - runs-on: ubuntu-20.04 + runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 with: fetch-depth: 100 - name: Install packages @@ -24,4 +33,18 @@ jobs: - name: Build run: source tools/ci.sh && ci_code_size_build - name: Compute code size difference - run: tools/metrics.py diff --error-threshold 0 ~/size0 ~/size1 + run: tools/metrics.py diff ~/size0 ~/size1 | tee diff + - name: Save PR number + if: github.event_name == 'pull_request' + env: + PR_NUMBER: ${{ github.event.number }} + run: echo $PR_NUMBER > pr_number + - name: Upload diff + if: github.event_name == 'pull_request' + uses: actions/upload-artifact@v4 + with: + name: code-size-report + path: | + diff + pr_number + retention-days: 1 diff --git a/.github/workflows/code_size_comment.yml b/.github/workflows/code_size_comment.yml new file mode 100644 index 0000000000000..521afad709d17 --- /dev/null +++ b/.github/workflows/code_size_comment.yml @@ -0,0 +1,105 @@ +name: Code size comment + +on: + workflow_run: + workflows: [Check code size] + types: [completed] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + comment: + runs-on: ubuntu-22.04 + steps: + - name: 'Download artifact' + id: download-artifact + uses: actions/github-script@v7 + with: + result-encoding: string + script: | + const fs = require('fs'); + + const allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({ + owner: context.repo.owner, + repo: context.repo.repo, + run_id: context.payload.workflow_run.id, + }); + + const matchArtifact = allArtifacts.data.artifacts.filter((artifact) => { + return artifact.name == "code-size-report" + }); + + if (matchArtifact.length === 0) { + console.log('no matching artifact found'); + console.log('result: "skip"'); + + return 'skip'; + } + + const download = await github.rest.actions.downloadArtifact({ + owner: context.repo.owner, + repo: context.repo.repo, + artifact_id: matchArtifact[0].id, + archive_format: 'zip', + }); + + fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/code-size-report.zip`, Buffer.from(download.data)); + + console.log('artifact downloaded to `code-size-report.zip`'); + console.log('result: "ok"'); + + return 'ok'; + - name: 'Unzip artifact' + if: steps.download-artifact.outputs.result == 'ok' + run: unzip code-size-report.zip + - name: Post comment to pull request + if: steps.download-artifact.outputs.result == 'ok' + uses: actions/github-script@v7 + with: + github-token: ${{secrets.GITHUB_TOKEN}} + script: | + const fs = require('fs'); + + const prNumber = Number(fs.readFileSync('pr_number')); + const codeSizeReport = `Code size report: + + \`\`\` + ${fs.readFileSync('diff')} + \`\`\` + `; + + const comments = await github.paginate( + github.rest.issues.listComments, + { + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + } + ); + + comments.reverse(); + + const previousComment = comments.find(comment => + comment.user.login === 'github-actions[bot]' + ) + + // if github-actions[bot] already made a comment, update it, + // otherwise create a new comment. + + if (previousComment) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: previousComment.id, + body: codeSizeReport, + }); + } else { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + body: codeSizeReport, + }); + } diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml new file mode 100644 index 0000000000000..1d6b1dc9d2b27 --- /dev/null +++ b/.github/workflows/codespell.yml @@ -0,0 +1,13 @@ +name: Check spelling with codespell + +on: [push, pull_request] + +jobs: + codespell: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + # codespell version should be kept in sync with .pre-commit-config.yml + - run: pip install --user codespell==2.4.1 tomli + - run: codespell + diff --git a/.github/workflows/commit_formatting.yml b/.github/workflows/commit_formatting.yml index 5f96fbb934426..fcbcaa7092ee2 100644 --- a/.github/workflows/commit_formatting.yml +++ b/.github/workflows/commit_formatting.yml @@ -1,14 +1,18 @@ name: Check commit message formatting -on: [push, pull_request] +on: [pull_request] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 with: - fetch-depth: '100' - - uses: actions/setup-python@v1 + fetch-depth: 100 + - uses: actions/setup-python@v5 - name: Check commit message formatting run: source tools/ci.sh && ci_commit_formatting_run diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index ec6b4c7f1962a..d01a4b50c9810 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -1,18 +1,23 @@ name: Build docs on: + push: pull_request: paths: - docs/** +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 - - uses: actions/setup-python@v1 + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 - name: Install Python packages - run: pip install Sphinx + run: pip install -r docs/requirements.txt - name: Build docs run: make -C docs/ html diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml new file mode 100644 index 0000000000000..6613f106625a2 --- /dev/null +++ b/.github/workflows/examples.yml @@ -0,0 +1,25 @@ +name: Check examples + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'examples/**' + - 'ports/unix/**' + - 'py/**' + - 'shared/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + embedding: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: make -C examples/embedding -f micropython_embed.mk && make -C examples/embedding + - name: Run + run: ./examples/embedding/embed | grep "hello world" diff --git a/.github/workflows/mpremote.yml b/.github/workflows/mpremote.yml new file mode 100644 index 0000000000000..ee91b6360b9b4 --- /dev/null +++ b/.github/workflows/mpremote.yml @@ -0,0 +1,29 @@ +name: Package mpremote + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + # Setting this to zero means fetch all history and tags, + # which hatch-vcs can use to discover the version tag. + fetch-depth: 0 + - uses: actions/setup-python@v5 + - name: Install build tools + run: pip install build + - name: Build mpremote wheel + run: cd tools/mpremote && python -m build --wheel + - name: Archive mpremote wheel + uses: actions/upload-artifact@v4 + with: + name: mpremote + path: | + tools/mpremote/dist/mpremote*.whl diff --git a/.github/workflows/mpy_format.yml b/.github/workflows/mpy_format.yml new file mode 100644 index 0000000000000..b6768a46c3cdc --- /dev/null +++ b/.github/workflows/mpy_format.yml @@ -0,0 +1,24 @@ +name: .mpy file format and tools + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'examples/**' + - 'tests/**' + - 'tools/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-22.04 # use 22.04 to get python2 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_mpy_format_setup + - name: Test mpy-tool.py + run: source tools/ci.sh && ci_mpy_format_test diff --git a/.github/workflows/ports.yml b/.github/workflows/ports.yml new file mode 100644 index 0000000000000..1f262b0ba4bee --- /dev/null +++ b/.github/workflows/ports.yml @@ -0,0 +1,22 @@ +name: Build ports metadata + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - ports/** + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - name: Build ports download metadata + run: mkdir boards && ./tools/autobuild/build-downloads.py . ./boards diff --git a/.github/workflows/ports_alif.yml b/.github/workflows/ports_alif.yml new file mode 100644 index 0000000000000..0e96e7d816e50 --- /dev/null +++ b/.github/workflows/ports_alif.yml @@ -0,0 +1,33 @@ +name: alif port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/alif/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_alif: + strategy: + fail-fast: false + matrix: + ci_func: # names are functions in ci.sh + - alif_ae3_build + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_alif_setup + - name: Build ci_${{matrix.ci_func }} + run: source tools/ci.sh && ci_${{ matrix.ci_func }} diff --git a/.github/workflows/ports_cc3200.yml b/.github/workflows/ports_cc3200.yml index 0eaa36da3796e..f178a140587db 100644 --- a/.github/workflows/ports_cc3200.yml +++ b/.github/workflows/ports_cc3200.yml @@ -8,15 +8,20 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/cc3200/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_cc3200_setup - name: Build diff --git a/.github/workflows/ports_esp32.yml b/.github/workflows/ports_esp32.yml index 09817ee126546..4c07f89437757 100644 --- a/.github/workflows/ports_esp32.yml +++ b/.github/workflows/ports_esp32.yml @@ -8,25 +8,50 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/esp32/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: - build_idf402: - runs-on: ubuntu-20.04 - steps: - - uses: actions/checkout@v2 - - name: Install packages - run: source tools/ci.sh && ci_esp32_idf402_setup - - name: Build - run: source tools/ci.sh && ci_esp32_build - - build_idf43: - runs-on: ubuntu-20.04 + build_idf: + strategy: + fail-fast: false + matrix: + ci_func: # names are functions in ci.sh + - esp32_build_cmod_spiram_s2 + - esp32_build_s3_c3 + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 - - name: Install packages - run: source tools/ci.sh && ci_esp32_idf43_setup - - name: Build - run: source tools/ci.sh && ci_esp32_build + - uses: actions/checkout@v4 + + - id: idf_ver + name: Read the ESP-IDF version (including Python version) + run: source tools/ci.sh && echo "IDF_VER=${IDF_VER}-py${PYTHON_VER}" | tee "$GITHUB_OUTPUT" + + - name: Cached ESP-IDF install + id: cache_esp_idf + uses: actions/cache@v4 + with: + path: | + ./esp-idf/ + ~/.espressif/ + !~/.espressif/dist/ + ~/.cache/pip/ + key: esp-idf-${{ steps.idf_ver.outputs.IDF_VER }} + + - name: Install ESP-IDF packages + if: steps.cache_esp_idf.outputs.cache-hit != 'true' + run: source tools/ci.sh && ci_esp32_idf_setup + + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2 + with: + key: esp32-${{ matrix.ci_func }} + + - name: Build ci_${{matrix.ci_func }} + run: source tools/ci.sh && ci_${{ matrix.ci_func }} diff --git a/.github/workflows/ports_esp8266.yml b/.github/workflows/ports_esp8266.yml index f4ce1f8212d87..5236edf40b959 100644 --- a/.github/workflows/ports_esp8266.yml +++ b/.github/workflows/ports_esp8266.yml @@ -8,15 +8,20 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/esp8266/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_esp8266_setup && ci_esp8266_path >> $GITHUB_PATH - name: Build diff --git a/.github/workflows/ports_mimxrt.yml b/.github/workflows/ports_mimxrt.yml new file mode 100644 index 0000000000000..7743e036ab377 --- /dev/null +++ b/.github/workflows/ports_mimxrt.yml @@ -0,0 +1,33 @@ +name: mimxrt port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/mimxrt/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + defaults: + run: + working-directory: 'micropython repo' # test build with space in path + steps: + - uses: actions/checkout@v4 + with: + path: 'micropython repo' + - name: Install packages + run: source tools/ci.sh && ci_mimxrt_setup + - name: Build + run: source tools/ci.sh && ci_mimxrt_build diff --git a/.github/workflows/ports_nrf.yml b/.github/workflows/ports_nrf.yml index 1ba3b0ce61e96..76727c9d1f6bd 100644 --- a/.github/workflows/ports_nrf.yml +++ b/.github/workflows/ports_nrf.yml @@ -8,15 +8,20 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/nrf/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: - runs-on: ubuntu-20.04 + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_nrf_setup - name: Build diff --git a/.github/workflows/ports_powerpc.yml b/.github/workflows/ports_powerpc.yml index 88fa59767b756..c41b13e5ddffe 100644 --- a/.github/workflows/ports_powerpc.yml +++ b/.github/workflows/ports_powerpc.yml @@ -8,15 +8,20 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/powerpc/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_powerpc_setup - name: Build diff --git a/.github/workflows/ports_qemu-arm.yml b/.github/workflows/ports_qemu-arm.yml deleted file mode 100644 index 8d144ca3cb7aa..0000000000000 --- a/.github/workflows/ports_qemu-arm.yml +++ /dev/null @@ -1,27 +0,0 @@ -name: qemu-arm port - -on: - push: - pull_request: - paths: - - '.github/workflows/*.yml' - - 'tools/**' - - 'py/**' - - 'extmod/**' - - 'lib/**' - - 'drivers/**' - - 'ports/qemu-arm/**' - - 'tests/**' - -jobs: - build_and_test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - name: Install packages - run: source tools/ci.sh && ci_qemu_arm_setup - - name: Build and run test suite - run: source tools/ci.sh && ci_qemu_arm_build - - name: Print failures - if: failure() - run: grep --text "FAIL" ports/qemu-arm/build/console.out diff --git a/.github/workflows/ports_qemu.yml b/.github/workflows/ports_qemu.yml new file mode 100644 index 0000000000000..57192c43936e5 --- /dev/null +++ b/.github/workflows/ports_qemu.yml @@ -0,0 +1,44 @@ +name: qemu port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/qemu/**' + - 'tests/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_and_test_arm: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_qemu_setup_arm + - name: Build and run test suite + run: source tools/ci.sh && ci_qemu_build_arm + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + build_and_test_rv32: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_qemu_setup_rv32 + - name: Build and run test suite + run: source tools/ci.sh && ci_qemu_build_rv32 + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ports_renesas-ra.yml b/.github/workflows/ports_renesas-ra.yml new file mode 100644 index 0000000000000..b9fa74331dc0b --- /dev/null +++ b/.github/workflows/ports_renesas-ra.yml @@ -0,0 +1,29 @@ +name: renesas-ra port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'drivers/**' + - 'ports/renesas-ra/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build_renesas_ra_board: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_renesas_ra_setup + - name: Build + run: source tools/ci.sh && ci_renesas_ra_board_build + diff --git a/.github/workflows/ports_rp2.yml b/.github/workflows/ports_rp2.yml index 668b79cae256c..748f38e143893 100644 --- a/.github/workflows/ports_rp2.yml +++ b/.github/workflows/ports_rp2.yml @@ -8,15 +8,25 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/rp2/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest + defaults: + run: + working-directory: 'micropython repo' # test build with space in path steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 + with: + path: 'micropython repo' - name: Install packages run: source tools/ci.sh && ci_rp2_setup - name: Build diff --git a/.github/workflows/ports_samd.yml b/.github/workflows/ports_samd.yml index bde4ed965faa0..5bf1826cd17bb 100644 --- a/.github/workflows/ports_samd.yml +++ b/.github/workflows/ports_samd.yml @@ -8,15 +8,20 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/samd/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_samd_setup - name: Build diff --git a/.github/workflows/ports_stm32.yml b/.github/workflows/ports_stm32.yml index 4634339c96a6a..8800f145189b8 100644 --- a/.github/workflows/ports_stm32.yml +++ b/.github/workflows/ports_stm32.yml @@ -8,25 +8,29 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'drivers/**' - 'ports/stm32/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: - build_pyb: - runs-on: ubuntu-20.04 + build_stm32: + strategy: + fail-fast: false + matrix: + ci_func: # names are functions in ci.sh + - stm32_pyb_build + - stm32_nucleo_build + - stm32_misc_build + runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_stm32_setup - - name: Build - run: source tools/ci.sh && ci_stm32_pyb_build + - name: Build ci_${{matrix.ci_func }} + run: source tools/ci.sh && ci_${{ matrix.ci_func }} - build_nucleo: - runs-on: ubuntu-20.04 - steps: - - uses: actions/checkout@v2 - - name: Install packages - run: source tools/ci.sh && ci_stm32_setup - - name: Build - run: source tools/ci.sh && ci_stm32_nucleo_build diff --git a/.github/workflows/ports_teensy.yml b/.github/workflows/ports_teensy.yml deleted file mode 100644 index 7ae77c80af604..0000000000000 --- a/.github/workflows/ports_teensy.yml +++ /dev/null @@ -1,23 +0,0 @@ -name: teensy port - -on: - push: - pull_request: - paths: - - '.github/workflows/*.yml' - - 'tools/**' - - 'py/**' - - 'extmod/**' - - 'lib/**' - - 'drivers/**' - - 'ports/teensy/**' - -jobs: - build: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - name: Install packages - run: source tools/ci.sh && ci_teensy_setup - - name: Build - run: source tools/ci.sh && ci_teensy_build diff --git a/.github/workflows/ports_unix.yml b/.github/workflows/ports_unix.yml index aa4b8abdc9893..2547015038e4d 100644 --- a/.github/workflows/ports_unix.yml +++ b/.github/workflows/ports_unix.yml @@ -8,16 +8,22 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'examples/**' + - 'mpy-cross/**' - 'ports/unix/**' - 'tests/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: minimal: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Build run: source tools/ci.sh && ci_unix_minimal_build - name: Run main test suite @@ -29,24 +35,34 @@ jobs: reproducible: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Build with reproducible date run: source tools/ci.sh && ci_unix_minimal_build env: SOURCE_DATE_EPOCH: 1234567890 - name: Check reproducible build date - run: echo | ports/unix/micropython-minimal -i | grep 'on 2009-02-13;' + run: echo | ports/unix/build-minimal/micropython -i | grep 'on 2009-02-13;' standard: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Build run: source tools/ci.sh && ci_unix_standard_build - name: Run main test suite run: source tools/ci.sh && ci_unix_standard_run_tests - - name: Run performance benchmarks - run: source tools/ci.sh && ci_unix_standard_run_perfbench + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + standard_v2: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: source tools/ci.sh && ci_unix_standard_v2_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_standard_v2_run_tests - name: Print failures if: failure() run: tests/run-tests.py --print-failures @@ -54,34 +70,37 @@ jobs: coverage: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_unix_coverage_setup - name: Build run: source tools/ci.sh && ci_unix_coverage_build - name: Run main test suite run: source tools/ci.sh && ci_unix_coverage_run_tests + - name: Test merging .mpy files + run: source tools/ci.sh && ci_unix_coverage_run_mpy_merge_tests - name: Build native mpy modules run: source tools/ci.sh && ci_native_mpy_modules_build - name: Test importing .mpy generated by mpy_ld.py run: source tools/ci.sh && ci_unix_coverage_run_native_mpy_tests - - name: Run lcov coverage analysis + - name: Run gcov coverage analysis run: | - mkdir -p coverage - lcov --rc lcov_branch_coverage=1 --directory ports/unix/build-coverage --capture --output-file coverage/lcov.info.all - lcov --remove coverage/lcov.info.all '*/lib/*' '*/ports/unix/*' '*/utils/*' --output-file coverage/lcov.info - - name: Send to coveralls - uses: coverallsapp/github-action@master + (cd ports/unix && gcov -o build-coverage/py ../../py/*.c || true) + (cd ports/unix && gcov -o build-coverage/extmod ../../extmod/*.c || true) + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v5 with: - github-token: ${{ secrets.GITHUB_TOKEN }} + fail_ci_if_error: true + verbose: true + token: ${{ secrets.CODECOV_TOKEN }} - name: Print failures if: failure() run: tests/run-tests.py --print-failures coverage_32bit: - runs-on: ubuntu-latest + runs-on: ubuntu-22.04 # use 22.04 to get libffi-dev:i386 steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_unix_32bit_setup - name: Build @@ -97,9 +116,9 @@ jobs: run: tests/run-tests.py --print-failures nanbox: - runs-on: ubuntu-latest + runs-on: ubuntu-22.04 # use 22.04 to get python2, and libffi-dev:i386 steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_unix_32bit_setup - name: Build @@ -113,7 +132,7 @@ jobs: float: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Build run: source tools/ci.sh && ci_unix_float_build - name: Run main test suite @@ -123,9 +142,9 @@ jobs: run: tests/run-tests.py --print-failures stackless_clang: - runs-on: ubuntu-20.04 + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_unix_clang_setup - name: Build @@ -137,9 +156,9 @@ jobs: run: tests/run-tests.py --print-failures float_clang: - runs-on: ubuntu-20.04 + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_unix_clang_setup - name: Build @@ -153,7 +172,12 @@ jobs: settrace: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + # Python 3.12 is the default for ubuntu-24.04, but that has compatibility issues with settrace tests. + # Can remove this step when ubuntu-latest uses a more recent Python 3.x as the default. + with: + python-version: '3.11' - name: Build run: source tools/ci.sh && ci_unix_settrace_build - name: Run main test suite @@ -165,7 +189,12 @@ jobs: settrace_stackless: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + # Python 3.12 is the default for ubuntu-24.04, but that has compatibility issues with settrace tests. + # Can remove this step when ubuntu-latest uses a more recent Python 3.x as the default. + with: + python-version: '3.11' - name: Build run: source tools/ci.sh && ci_unix_settrace_stackless_build - name: Run main test suite @@ -175,10 +204,12 @@ jobs: run: tests/run-tests.py --print-failures macos: - runs-on: macos-11.0 + runs-on: macos-latest steps: - - uses: actions/checkout@v2 - - uses: actions/setup-python@v1 + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.8' - name: Build run: source tools/ci.sh && ci_unix_macos_build - name: Run tests @@ -186,3 +217,48 @@ jobs: - name: Print failures if: failure() run: tests/run-tests.py --print-failures + + qemu_mips: + # ubuntu-22.04 is needed for older libffi. + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_qemu_mips_setup + - name: Build + run: source tools/ci.sh && ci_unix_qemu_mips_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_qemu_mips_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + qemu_arm: + # ubuntu-22.04 is needed for older libffi. + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_qemu_arm_setup + - name: Build + run: source tools/ci.sh && ci_unix_qemu_arm_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_qemu_arm_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures + + qemu_riscv64: + # ubuntu-22.04 is needed for older libffi. + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_unix_qemu_riscv64_setup + - name: Build + run: source tools/ci.sh && ci_unix_qemu_riscv64_build + - name: Run main test suite + run: source tools/ci.sh && ci_unix_qemu_riscv64_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ports_webassembly.yml b/.github/workflows/ports_webassembly.yml new file mode 100644 index 0000000000000..880f15ab34469 --- /dev/null +++ b/.github/workflows/ports_webassembly.yml @@ -0,0 +1,32 @@ +name: webassembly port + +on: + push: + pull_request: + paths: + - '.github/workflows/*.yml' + - 'tools/**' + - 'py/**' + - 'extmod/**' + - 'shared/**' + - 'lib/**' + - 'ports/webassembly/**' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install packages + run: source tools/ci.sh && ci_webassembly_setup + - name: Build + run: source tools/ci.sh && ci_webassembly_build + - name: Run tests + run: source tools/ci.sh && ci_webassembly_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ports_windows.yml b/.github/workflows/ports_windows.yml index 1bfe40c7fded2..84e018ba15d16 100644 --- a/.github/workflows/ports_windows.yml +++ b/.github/workflows/ports_windows.yml @@ -8,15 +8,143 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'ports/unix/**' - 'ports/windows/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: - build: + build-vs: + strategy: + fail-fast: false + matrix: + platform: [x86, x64] + configuration: [Debug, Release] + variant: [dev, standard] + visualstudio: ['2017', '2019', '2022'] + include: + - visualstudio: '2017' + runner: windows-latest + vs_version: '[15, 16)' + - visualstudio: '2019' + runner: windows-2019 + vs_version: '[16, 17)' + - visualstudio: '2022' + runner: windows-2022 + vs_version: '[17, 18)' + # trim down the number of jobs in the matrix + exclude: + - variant: standard + configuration: Debug + - visualstudio: '2019' + configuration: Debug + env: + CI_BUILD_CONFIGURATION: ${{ matrix.configuration }} + runs-on: ${{ matrix.runner }} + steps: + - name: Install Visual Studio 2017 + if: matrix.visualstudio == '2017' + run: | + choco install visualstudio2017buildtools + choco install visualstudio2017-workload-vctools + choco install windows-sdk-8.1 + - uses: microsoft/setup-msbuild@v2 + with: + vs-version: ${{ matrix.vs_version }} + - uses: actions/setup-python@v5 + if: matrix.runner == 'windows-2019' + with: + python-version: '3.9' + - uses: actions/checkout@v4 + - name: Build mpy-cross.exe + run: msbuild mpy-cross\mpy-cross.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} + - name: Update submodules + run: git submodule update --init lib/micropython-lib + - name: Build micropython.exe + run: msbuild ports\windows\micropython.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} -property:PyVariant=${{ matrix.variant }} + - name: Get micropython.exe path + id: get_path + run: | + $exePath="$(msbuild ports\windows\micropython.vcxproj -nologo -v:m -t:ShowTargetPath -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} -property:PyVariant=${{ matrix.variant }})" + echo ("micropython=" + $exePath.Trim()) >> $env:GITHUB_OUTPUT + - name: Run tests + id: test + env: + MICROPY_MICROPYTHON: ${{ steps.get_path.outputs.micropython }} + working-directory: tests + run: python run-tests.py + - name: Print failures + if: failure() && steps.test.conclusion == 'failure' + working-directory: tests + run: python run-tests.py --print-failures + - name: Run mpy tests + id: test_mpy + env: + MICROPY_MICROPYTHON: ${{ steps.get_path.outputs.micropython }} + working-directory: tests + run: python run-tests.py --via-mpy -d basics float micropython + - name: Print mpy failures + if: failure() && steps.test_mpy.conclusion == 'failure' + working-directory: tests + run: python run-tests.py --print-failures + + build-mingw: + strategy: + fail-fast: false + matrix: + variant: [dev, standard] + sys: [mingw32, mingw64] + include: + - sys: mingw32 + env: i686 + - sys: mingw64 + env: x86_64 + runs-on: windows-2022 + env: + CHERE_INVOKING: enabled_from_arguments + defaults: + run: + shell: msys2 {0} + steps: + - uses: actions/setup-python@v5 + # note: can go back to installing mingw-w64-${{ matrix.env }}-python after + # MSYS2 updates to Python >3.12 (due to settrace compatibility issue) + with: + python-version: '3.11' + - uses: msys2/setup-msys2@v2 + with: + msystem: ${{ matrix.sys }} + update: true + install: >- + make + mingw-w64-${{ matrix.env }}-gcc + pkg-config + git + diffutils + path-type: inherit # Remove when setup-python is removed + - uses: actions/checkout@v4 + - name: Build mpy-cross.exe + run: make -C mpy-cross -j2 + - name: Update submodules + run: make -C ports/windows VARIANT=${{ matrix.variant }} submodules + - name: Build micropython.exe + run: make -C ports/windows -j2 VARIANT=${{ matrix.variant }} + - name: Run tests + id: test + run: make -C ports/windows test_full VARIANT=${{ matrix.variant }} + - name: Print failures + if: failure() && steps.test.conclusion == 'failure' + working-directory: tests + run: python run-tests.py --print-failures + + cross-build-on-linux: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Install packages run: source tools/ci.sh && ci_windows_setup - name: Build diff --git a/.github/workflows/ports_zephyr.yml b/.github/workflows/ports_zephyr.yml index d9ae2b8c55a2a..eb85af6a36154 100644 --- a/.github/workflows/ports_zephyr.yml +++ b/.github/workflows/ports_zephyr.yml @@ -8,17 +8,53 @@ on: - 'tools/**' - 'py/**' - 'extmod/**' + - 'shared/**' - 'lib/**' - 'ports/zephyr/**' +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: jlumbroso/free-disk-space@main + with: + # Only free up a few things so this step runs quickly. + android: false + dotnet: true + haskell: true + large-packages: false + docker-images: false + swap-storage: false + - uses: actions/checkout@v4 + - id: versions + name: Read Zephyr version + run: source tools/ci.sh && echo "ZEPHYR=$ZEPHYR_VERSION" | tee "$GITHUB_OUTPUT" + - name: Cached Zephyr Workspace + id: cache_workspace + uses: actions/cache@v4 + with: + # note that the Zephyr CI docker image is 15GB. At time of writing + # GitHub caches are limited to 10GB total for a project. So we only + # cache the "workspace" + path: ./zephyrproject + key: zephyr-workspace-${{ steps.versions.outputs.ZEPHYR }} + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2 + with: + key: zephyr - name: Install packages run: source tools/ci.sh && ci_zephyr_setup - name: Install Zephyr + if: steps.cache_workspace.outputs.cache-hit != 'true' run: source tools/ci.sh && ci_zephyr_install - name: Build run: source tools/ci.sh && ci_zephyr_build + - name: Run main test suite + run: source tools/ci.sh && ci_zephyr_run_tests + - name: Print failures + if: failure() + run: tests/run-tests.py --print-failures diff --git a/.github/workflows/ruff.yml b/.github/workflows/ruff.yml new file mode 100644 index 0000000000000..4c4a2a3162ed6 --- /dev/null +++ b/.github/workflows/ruff.yml @@ -0,0 +1,13 @@ +name: Python code lint and formatting with ruff + +on: [push, pull_request] + +jobs: + ruff: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + # ruff version should be kept in sync with .pre-commit-config.yaml & also micropython-lib + - run: pipx install ruff==0.11.6 + - run: ruff check --output-format=github . + - run: ruff format --diff . diff --git a/.gitignore b/.gitignore index 7342489927888..b5010dfd14db6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,48 +1,26 @@ -# Compiled Sources -################### -*.o -*.a -*.elf -*.bin -*.map -*.hex -*.dis -*.exe - -# Packages -############ - -# Logs and Databases -###################### -*.log - -# VIM Swap Files -###################### -*.swp +# This .gitignore file is intended to be minimal. +# +# If you find that you need additional rules, such as IDE temporary +# files, please do so either via a global .gitignore file (registered +# with core.excludesFile), or by adding private repository-specific +# rules to .git/info/exclude. See https://git-scm.com/docs/gitignore +# for more information. # Build directories -###################### build/ build-*/ +docs/genrst/ -# Test failure outputs -###################### +# Test failure outputs and intermediate artefacts tests/results/* +tests/ports/unix/ffi_lib.so # Python cache files -###################### __pycache__/ -*.pyc # Customized Makefile/project overrides -###################### GNUmakefile user.props -# Generated rst files -###################### -genrst/ - # MacOS desktop metadata files -###################### .DS_Store diff --git a/.gitmodules b/.gitmodules index faa89c4f5cea1..02849ec9bdd11 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,20 +1,19 @@ [submodule "lib/axtls"] path = lib/axtls - url = https://github.com/pfalcon/axtls - branch = micropython + url = https://github.com/micropython/axtls.git [submodule "lib/libffi"] path = lib/libffi - url = https://github.com/atgreen/libffi + url = https://github.com/libffi/libffi [submodule "lib/lwip"] path = lib/lwip url = https://github.com/lwip-tcpip/lwip.git [submodule "lib/berkeley-db-1.xx"] path = lib/berkeley-db-1.xx - url = https://github.com/pfalcon/berkeley-db-1.xx + url = https://github.com/micropython/berkeley-db-1.xx [submodule "lib/stm32lib"] path = lib/stm32lib url = https://github.com/micropython/stm32lib - branch = work-F4-1.13.1+F7-1.5.0+L4-1.3.0 + branch = work-F0-1.9.0+F4-1.16.0+F7-1.7.0+G0-1.5.1+G4-1.3.0+H7-1.6.0+L0-1.11.2+L1-1.10.3+L4-1.17.0+WB-1.10.0+WL-1.1.0 [submodule "lib/nrfx"] path = lib/nrfx url = https://github.com/NordicSemiconductor/nrfx.git @@ -24,12 +23,13 @@ [submodule "lib/asf4"] path = lib/asf4 url = https://github.com/adafruit/asf4 + branch = circuitpython [submodule "lib/tinyusb"] path = lib/tinyusb url = https://github.com/hathach/tinyusb [submodule "lib/mynewt-nimble"] path = lib/mynewt-nimble - url = https://github.com/apache/mynewt-nimble.git + url = https://github.com/micropython/mynewt-nimble.git [submodule "lib/btstack"] path = lib/btstack url = https://github.com/bluekitchen/btstack.git @@ -42,3 +42,35 @@ [submodule "lib/pico-sdk"] path = lib/pico-sdk url = https://github.com/raspberrypi/pico-sdk.git +[submodule "lib/fsp"] + path = lib/fsp + url = https://github.com/renesas/fsp.git +[submodule "lib/wiznet"] + path = lib/wiznet5k + url = https://github.com/andrewleech/wiznet_ioLibrary_Driver.git + # Requires https://github.com/Wiznet/ioLibrary_Driver/pull/120 + # url = https://github.com/Wiznet/ioLibrary_Driver.git +[submodule "lib/cyw43-driver"] + path = lib/cyw43-driver + url = https://github.com/georgerobotics/cyw43-driver.git +[submodule "lib/micropython-lib"] + path = lib/micropython-lib + url = https://github.com/micropython/micropython-lib.git +[submodule "lib/protobuf-c"] + path = lib/protobuf-c + url = https://github.com/protobuf-c/protobuf-c.git +[submodule "lib/open-amp"] + path = lib/open-amp + url = https://github.com/OpenAMP/open-amp.git +[submodule "lib/libmetal"] + path = lib/libmetal + url = https://github.com/OpenAMP/libmetal.git +[submodule "lib/arduino-lib"] + path = lib/arduino-lib + url = https://github.com/arduino/arduino-lib-mpy.git +[submodule "lib/alif_ensemble-cmsis-dfp"] + path = lib/alif_ensemble-cmsis-dfp + url = https://github.com/alifsemi/alif_ensemble-cmsis-dfp.git +[submodule "lib/alif-security-toolkit"] + path = lib/alif-security-toolkit + url = https://github.com/micropython/alif-security-toolkit.git diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000000000..ac9785bb59232 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,27 @@ +repos: + - repo: local + hooks: + - id: codeformat + name: MicroPython codeformat.py for changed C files + entry: tools/codeformat.py -v -c -f + language: python + - id: verifygitlog + name: MicroPython git commit message format checker + entry: tools/verifygitlog.py --check-file --ignore-rebase + language: python + verbose: true + stages: [commit-msg] + - repo: https://github.com/charliermarsh/ruff-pre-commit + # Version should be kept in sync with .github/workflows/ruff.yml & also micropython-lib + rev: v0.11.6 + hooks: + - id: ruff + - id: ruff-format + - repo: https://github.com/codespell-project/codespell + # Version should be kept in sync with .github/workflows/codespell.yml + rev: v2.4.1 + hooks: + - id: codespell + name: Spellcheck for changed files (codespell) + additional_dependencies: + - tomli diff --git a/CODECONVENTIONS.md b/CODECONVENTIONS.md index 78fb912a6ab75..d3f71cb083deb 100644 --- a/CODECONVENTIONS.md +++ b/CODECONVENTIONS.md @@ -11,7 +11,7 @@ It's also ok to drop file extensions. Besides prefix, first line of a commit message should describe a change clearly and to the point, and be a grammatical sentence with -final full stop. First line should fit within 72 characters. Examples +final full stop. First line must fit within 72 characters. Examples of good first line of commit messages: py/objstr: Add splitlines() method. @@ -27,12 +27,9 @@ change beyond 5 lines would likely require such detailed description. To get good practical examples of good commits and their messages, browse the `git log` of the project. -When committing you are encouraged to sign-off your commit by adding -"Signed-off-by" lines and similar, eg using "git commit -s". If you don't -explicitly sign-off in this way then the commit message, which includes your -name and email address in the "Author" line, implies your sign-off. In either -case, of explicit or implicit sign-off, you are certifying and signing off -against the following: +When committing you must sign-off your commit by adding "Signed-off-by:" +line(s) at the end of the commit message, e.g. using `git commit -s`. You +are then certifying and signing off against the following: * That you wrote the change yourself, or took it from a project with a compatible license (in the latter case the commit message, and possibly @@ -49,28 +46,146 @@ against the following: * Your contribution including commit message will be publicly and indefinitely available for anyone to access, including redistribution under the terms of the project's license. -* Your signature for all of the above, which is the "Signed-off-by" line - or the "Author" line in the commit message, includes your full real name and - a valid and active email address by which you can be contacted in the - foreseeable future. +* Your signature for all of the above, which is the "Signed-off-by" line, + includes your full real name and a valid and active email address by + which you can be contacted in the foreseeable future. Code auto-formatting ==================== -Both C and Python code are auto-formatted using the `tools/codeformat.py` -script. This uses [uncrustify](https://github.com/uncrustify/uncrustify) to -format C code and [black](https://github.com/psf/black) to format Python code. -After making changes, and before committing, run this tool to reformat your -changes to the correct style. Without arguments this tool will reformat all -source code (and may take some time to run). Otherwise pass as arguments to -the tool the files that changed and it will only reformat those. +Both C and Python code formatting are controlled for consistency across the +MicroPython codebase. C code is formatted using the `tools/codeformat.py` +script which uses [uncrustify](https://github.com/uncrustify/uncrustify). +Python code is linted and formatted using +[ruff & ruff format](https://github.com/astral-sh/ruff). +After making changes, and before committing, run `tools/codeformat.py` to +reformat your C code and `ruff format` for any Python code. Without +arguments this tool will reformat all source code (and may take some time +to run). Otherwise pass as arguments to the tool the files that changed, +and it will only reformat those. + +uncrustify +========== + +Only [uncrustify](https://github.com/uncrustify/uncrustify) v0.71 or v0.72 can +be used for MicroPython. Different uncrustify versions produce slightly +different formatting, and the configuration file formats are often +incompatible. v0.73 or newer *will not work*. + +Depending on your operating system version, it may be possible to install a pre-compiled +uncrustify version: + +Ubuntu, Debian +-------------- + +Ubuntu versions 21.10 or 22.04LTS, and Debian versions bullseye or bookworm all +include v0.72 so can be installed directly: + +``` +$ apt install uncrustify +``` + +Arch Linux +---------- + +The current Arch uncrustify version is too new. There is an [old Arch package +for v0.72](https://archive.archlinux.org/packages/u/uncrustify/) that can be +installed from the Arch Linux archive ([more +information](https://wiki.archlinux.org/title/Downgrading_packages#Arch_Linux_Archive)). Use +the [IgnorePkg feature](https://wiki.archlinux.org/title/Pacman#Skip_package_from_being_upgraded) +to prevent it re-updating. + +Brew +---- + +This command may work, please raise a new Issue if it doesn't: + +``` +curl -L https://github.com/Homebrew/homebrew-core/raw/2b07d8192623365078a8b855a164ebcdf81494a6/Formula/uncrustify.rb > uncrustify.rb && brew install uncrustify.rb && rm uncrustify.rb +``` + +Code spell checking +=================== + +Code spell checking is done using [codespell](https://github.com/codespell-project/codespell#codespell) +and runs in a GitHub action in CI. Codespell is configured via `pyproject.toml` +to avoid false positives. It is recommended run codespell before submitting a +PR. To simplify this, codespell is configured as a pre-commit hook and will be +installed if you run `pre-commit install` (see below). + +If you want to install and run codespell manually, you can do so by running: + +``` +$ pip install codespell tomli +$ codespell +``` + +Automatic Pre-Commit Hooks +========================== + +To have code formatting and commit message conventions automatically checked, +a configuration file is provided for the [pre-commit](https://pre-commit.com/) +tool. + +First install `pre-commit`, either from your system package manager or via +`pip`. When installing `pre-commit` via pip, it is recommended to use a +virtual environment. Other sources, such as Brew are also available, see +[the docs](https://pre-commit.com/index.html#install) for details. + +``` +$ apt install pre-commit # Ubuntu, Debian +$ pacman -Sy python-precommit # Arch Linux +$ brew install pre-commit # Brew +$ pip install pre-commit # PyPI +``` + +Next, install [uncrustify (see above)](#uncrustify). Other dependencies are managed by +pre-commit automatically, but uncrustify needs to be installed and available on +the PATH. + +Then, inside the MicroPython repository, register the git hooks for pre-commit +by running: + +``` +$ pre-commit install --hook-type pre-commit --hook-type commit-msg +``` + +pre-commit will now automatically run during `git commit` for both code and +commit message formatting. + +The same formatting checks will be run by CI for any Pull Request submitted to +MicroPython. Pre-commit allows you to see any failure more quickly, and in many +cases will automatically correct it in your local working copy. + +To unregister `pre-commit` from your MicroPython repository, run: + +``` +$ pre-commit uninstall --hook-type pre-commit --hook-type commit-msg +``` + +Tips: + +* To skip pre-commit checks on a single commit, use `git commit -n` (for + `--no-verify`). +* To ignore the pre-commit message format check temporarily, start the commit + message subject line with "WIP" (for "Work In Progress"). + +Running pre-commit manually +=========================== + +Once pre-commit is installed as per the previous section it can be manually +run against the MicroPython python codebase to update file formatting on +demand, with either: +* `pre-commit run --all-files` to fix all files in the MicroPython codebase +* `pre-commit run --file ./path/to/my/file` to fix just one file +* `pre-commit run --file ./path/to/my/folder/*` to fix just one folder Python code conventions ======================= Python code follows [PEP 8](https://legacy.python.org/dev/peps/pep-0008/) and -is auto-formatted using [black](https://github.com/psf/black) with a line-length -of 99 characters. +is auto-formatted using [ruff format](https://docs.astral.sh/ruff/formatter) +with a line-length of 99 characters. Naming conventions: - Module names are short and all lowercase; eg pyb, stm. @@ -91,14 +206,21 @@ adhere to the existing style and use `tools/codeformat.py` to check any changes. The main conventions, and things not enforceable via the auto-formatter, are described below. -White space: +As the MicroPython code base is over ten years old, not every source file +conforms fully to these conventions. If making small changes to existing code, +then it's usually acceptable to follow the existing code's style. New code or +major changes should follow the conventions described here. + +## White space + - Expand tabs to 4 spaces. - Don't leave trailing whitespace at the end of a line. - For control blocks (if, for, while), put 1 space between the keyword and the opening parenthesis. - Put 1 space after a comma, and 1 space around operators. -Braces: +## Braces + - Use braces for all blocks, even no-line and single-line pieces of code. - Put opening braces on the end of the line it belongs to, not on @@ -106,18 +228,43 @@ Braces: - For else-statements, put the else on the same line as the previous closing brace. -Header files: +## Header files + - Header files should be protected from multiple inclusion with #if directives. See an existing header for naming convention. -Names: +## Names + - Use underscore_case, not camelCase for all names. - Use CAPS_WITH_UNDERSCORE for enums and macros. - When defining a type use underscore_case and put '_t' after it. -Integer types: MicroPython runs on 16, 32, and 64 bit machines, so it's -important to use the correctly-sized (and signed) integer types. The -general guidelines are: +### Public names (declared in headers) + +- MicroPython-specific names (especially any declared in `py/` and `extmod/` + directories) should generally start with `mp_` or `MP_`. +- Functions and variables declared in a header should generally share a longer + common prefix. Usually the prefix matches the file name (i.e. items defined in + `py/obj.c` are declared in `py/obj.h` and should be prefixed `mp_obj_`). There + are exceptions, for example where one header file contains declarations + implemented in multiple source files for expediency. + +### Private names (specific to a single .c file) + +- For static functions and variables exposed to Python (i.e. a static C function + that is wrapped in `MP_DEFINE_CONST_FUN_...` and attached to a module), use + the file-level shared common prefix, i.e. name them as if the function or + variable was not static. +- Other static definitions in source files (i.e. functions or variables defined + in a .c file that are only used within that .c file) don't need any prefix + (specifically: no `s_` or `_` prefix, and generally avoid adding the + file-level common prefix). + +## Integer types + +MicroPython runs on 16, 32, and 64 bit machines, so it's important to use the +correctly-sized (and signed) integer types. The general guidelines are: + - For most cases use mp_int_t for signed and mp_uint_t for unsigned integer values. These are guaranteed to be machine-word sized and therefore big enough to hold the value from a MicroPython small-int @@ -126,11 +273,13 @@ general guidelines are: - You can use int/uint, but remember that they may be 16-bits wide. - If in doubt, use mp_int_t/mp_uint_t. -Comments: +## Comments + - Be concise and only write comments for things that are not obvious. - Use `// ` prefix, NOT `/* ... */`. No extra fluff. -Memory allocation: +## Memory allocation + - Use m_new, m_renew, m_del (and friends) to allocate and free heap memory. These macros are defined in py/misc.h. @@ -165,7 +314,7 @@ Documentation conventions ========================= MicroPython generally follows CPython in documentation process and -conventions. reStructuredText syntax is used for the documention. +conventions. reStructuredText syntax is used for the documentation. Specific conventions/suggestions: diff --git a/LICENSE b/LICENSE index 3193eb8cec189..929a2e97de7bf 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2013-2021 Damien P. George +Copyright (c) 2013-2025 Damien P. George Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -19,3 +19,70 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +-------------------------------------------------------------------------------- + +Unless specified otherwise (see below), the above license and copyright applies +to all files in this repository. + +Individual files may include additional copyright holders. + +The various ports of MicroPython may include third-party software that is +licensed under different terms. These licenses are summarised in the tree +below, please refer to these files and directories for further license and +copyright information. Note that (L)GPL-licensed code listed below is only +used during the build process and is not part of the compiled source code. + +/ (MIT) + /drivers + /cc3100 (BSD-3-clause) + /lib + /asf4 (Apache-2.0) + /axtls (BSD-3-clause) + /config + /scripts + /config (GPL-2.0-or-later) + /Rules.mak (GPL-2.0) + /berkeley-db-1xx (BSD-4-clause) + /btstack (See btstack/LICENSE) + /cmsis (BSD-3-clause) + /crypto-algorithms (NONE) + /libhydrogen (ISC) + /libmetal (BSD-3-clause) + /littlefs (BSD-3-clause) + /lwip (BSD-3-clause) + /mynewt-nimble (Apache-2.0) + /nrfx (BSD-3-clause) + /nxp_driver (BSD-3-Clause) + /oofatfs (BSD-1-clause) + /open-amp (BSD-3-clause) + /pico-sdk (BSD-3-clause) + /re15 (BSD-3-clause) + /stm32lib (BSD-3-clause) + /tinyusb (MIT) + /uzlib (Zlib) + /wiznet5k (MIT) + /logo (uses OFL-1.1) + /ports + /cc3200 + /hal (BSD-3-clause) + /simplelink (BSD-3-clause) + /FreeRTOS (GPL-2.0 with FreeRTOS exception) + /esp32 + /ppp_set_auth.* (Apache-2.0) + /rp2 + /mutex_extra.c (BSD-3-clause) + /clocks_extra.c (BSD-3-clause) + /stm32 + /usbd*.c (MCD-ST Liberty SW License Agreement V2) + /stm32_it.* (MIT + BSD-3-clause) + /system_stm32*.c (MIT + BSD-3-clause) + /boards + /startup_stm32*.s (BSD-3-clause) + /*/stm32*.h (BSD-3-clause) + /usbdev (MCD-ST Liberty SW License Agreement V2) + /usbhost (MCD-ST Liberty SW License Agreement V2) + /zephyr + /src (Apache-2.0) + /tools + /dfu.py (LGPL-3.0-only) diff --git a/README.md b/README.md index f75590a5ba7d0..c78a2384604fe 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -[![CI badge](https://github.com/micropython/micropython/workflows/unix%20port/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![Coverage badge](https://coveralls.io/repos/micropython/micropython/badge.png?branch=master)](https://coveralls.io/r/micropython/micropython?branch=master) +[![Unix CI badge](https://github.com/micropython/micropython/actions/workflows/ports_unix.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![STM32 CI badge](https://github.com/micropython/micropython/actions/workflows/ports_stm32.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![Docs CI badge](https://github.com/micropython/micropython/actions/workflows/docs.yml/badge.svg)](https://docs.micropython.org/) [![codecov](https://codecov.io/gh/micropython/micropython/branch/master/graph/badge.svg?token=I92PfD05sD)](https://codecov.io/gh/micropython/micropython) The MicroPython project ======================= @@ -15,174 +15,133 @@ code-base, including project-wide name changes and API changes. MicroPython implements the entire Python 3.4 syntax (including exceptions, `with`, `yield from`, etc., and additionally `async`/`await` keywords from -Python 3.5). The following core datatypes are provided: `str` (including -basic Unicode support), `bytes`, `bytearray`, `tuple`, `list`, `dict`, `set`, -`frozenset`, `array.array`, `collections.namedtuple`, classes and instances. -Builtin modules include `sys`, `time`, and `struct`, etc. Select ports have -support for `_thread` module (multithreading). Note that only a subset of -Python 3 functionality is implemented for the data types and modules. - -MicroPython can execute scripts in textual source form or from precompiled -bytecode, in both cases either from an on-device filesystem or "frozen" into -the MicroPython executable. - -See the repository http://github.com/micropython/pyboard for the MicroPython -board (PyBoard), the officially supported reference electronic circuit board. - -Major components in this repository: -- py/ -- the core Python implementation, including compiler, runtime, and +Python 3.5 and some select features from later versions). The following core +datatypes are provided: `str`(including basic Unicode support), `bytes`, +`bytearray`, `tuple`, `list`, `dict`, `set`, `frozenset`, `array.array`, +`collections.namedtuple`, classes and instances. Builtin modules include +`os`, `sys`, `time`, `re`, and `struct`, etc. Some ports have support for +`_thread` module (multithreading), `socket` and `ssl` for networking, and +`asyncio`. Note that only a subset of Python 3 functionality is implemented +for the data types and modules. + +MicroPython can execute scripts in textual source form (.py files) or from +precompiled bytecode (.mpy files), in both cases either from an on-device +filesystem or "frozen" into the MicroPython executable. + +MicroPython also provides a set of MicroPython-specific modules to access +hardware-specific functionality and peripherals such as GPIO, Timers, ADC, +DAC, PWM, SPI, I2C, CAN, Bluetooth, and USB. + +Getting started +--------------- + +See the [online documentation](https://docs.micropython.org/) for the API +reference and information about using MicroPython and information about how +it is implemented. + +We use [GitHub Discussions](https://github.com/micropython/micropython/discussions) +as our forum, and [Discord](https://discord.gg/RB8HZSAExQ) for chat. These +are great places to ask questions and advice from the community or to discuss your +MicroPython-based projects. + +For bugs and feature requests, please [raise an issue](https://github.com/micropython/micropython/issues/new/choose) +and follow the templates there. + +For information about the [MicroPython pyboard](https://store.micropython.org/pyb-features), +the officially supported board from the +[original Kickstarter campaign](https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers), +see the [schematics and pinouts](http://github.com/micropython/pyboard) and +[documentation](https://docs.micropython.org/en/latest/pyboard/quickref.html). + +Contributing +------------ + +MicroPython is an open-source project and welcomes contributions. To be +productive, please be sure to follow the +[Contributors' Guidelines](https://github.com/micropython/micropython/wiki/ContributorGuidelines) +and the [Code Conventions](https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md). +Note that MicroPython is licenced under the MIT license, and all contributions +should follow this license. + +About this repository +--------------------- + +This repository contains the following components: +- [py/](py/) -- the core Python implementation, including compiler, runtime, and core library. -- mpy-cross/ -- the MicroPython cross-compiler which is used to turn scripts +- [mpy-cross/](mpy-cross/) -- the MicroPython cross-compiler which is used to turn scripts into precompiled bytecode. -- ports/unix/ -- a version of MicroPython that runs on Unix. -- ports/stm32/ -- a version of MicroPython that runs on the PyBoard and similar - STM32 boards (using ST's Cube HAL drivers). -- ports/minimal/ -- a minimal MicroPython port. Start with this if you want - to port MicroPython to another microcontroller. -- tests/ -- test framework and test scripts. -- docs/ -- user documentation in Sphinx reStructuredText format. Rendered - HTML documentation is available at http://docs.micropython.org. - -Additional components: -- ports/bare-arm/ -- a bare minimum version of MicroPython for ARM MCUs. Used - mostly to control code size. -- ports/teensy/ -- a version of MicroPython that runs on the Teensy 3.1 - (preliminary but functional). -- ports/pic16bit/ -- a version of MicroPython for 16-bit PIC microcontrollers. -- ports/cc3200/ -- a version of MicroPython that runs on the CC3200 from TI. -- ports/esp8266/ -- a version of MicroPython that runs on Espressif's ESP8266 SoC. -- ports/esp32/ -- a version of MicroPython that runs on Espressif's ESP32 SoC. -- ports/nrf/ -- a version of MicroPython that runs on Nordic's nRF51 and nRF52 MCUs. -- extmod/ -- additional (non-core) modules implemented in C. -- tools/ -- various tools, including the pyboard.py module. -- examples/ -- a few example Python scripts. - -The subdirectories above may include READMEs with additional info. +- [ports/](ports/) -- platform-specific code for the various ports and architectures that MicroPython runs on. +- [lib/](lib/) -- submodules for external dependencies. +- [tests/](tests/) -- test framework and test scripts. +- [docs/](docs/) -- user documentation in Sphinx reStructuredText format. This is used to generate the [online documentation](http://docs.micropython.org). +- [extmod/](extmod/) -- additional (non-core) modules implemented in C. +- [tools/](tools/) -- various tools, including the pyboard.py module. +- [examples/](examples/) -- a few example Python scripts. "make" is used to build the components, or "gmake" on BSD-based systems. You will also need bash, gcc, and Python 3.3+ available as the command `python3` (if your system only has Python 2.7 then invoke make with the additional option -`PYTHON=python2`). +`PYTHON=python2`). Some ports (rp2 and esp32) additionally use CMake. + +Supported platforms & architectures +----------------------------------- + +MicroPython runs on a wide range of microcontrollers, as well as on Unix-like +(including Linux, BSD, macOS, WSL) and Windows systems. + +Microcontroller targets can be as small as 256kiB flash + 16kiB RAM, although +devices with at least 512kiB flash + 128kiB RAM allow a much more +full-featured experience. + +The [Unix](ports/unix) and [Windows](ports/windows) ports allow both +development and testing of MicroPython itself, as well as providing +lightweight alternative to CPython on these platforms (in particular on +embedded Linux systems). + +The ["minimal"](ports/minimal) port provides an example of a very basic +MicroPython port and can be compiled as both a standalone Linux binary as +well as for ARM Cortex M4. Start with this if you want to port MicroPython to +another microcontroller. Additionally the ["bare-arm"](ports/bare-arm) port +is an example of the absolute minimum configuration, and is used to keep +track of the code size of the core runtime and VM. + +In addition, the following ports are provided in this repository: + - [cc3200](ports/cc3200) -- Texas Instruments CC3200 (including PyCom WiPy). + - [esp32](ports/esp32) -- Espressif ESP32 SoC (including ESP32S2, ESP32S3, ESP32C3, ESP32C6). + - [esp8266](ports/esp8266) -- Espressif ESP8266 SoC. + - [mimxrt](ports/mimxrt) -- NXP m.iMX RT (including Teensy 4.x). + - [nrf](ports/nrf) -- Nordic Semiconductor nRF51 and nRF52. + - [pic16bit](ports/pic16bit) -- Microchip PIC 16-bit. + - [powerpc](ports/powerpc) -- IBM PowerPC (including Microwatt) + - [qemu](ports/qemu) -- QEMU-based emulated target (for testing) + - [renesas-ra](ports/renesas-ra) -- Renesas RA family. + - [rp2](ports/rp2) -- Raspberry Pi RP2040 (including Pico and Pico W). + - [samd](ports/samd) -- Microchip (formerly Atmel) SAMD21 and SAMD51. + - [stm32](ports/stm32) -- STMicroelectronics STM32 family (including F0, F4, F7, G0, G4, H7, L0, L4, WB) + - [webassembly](ports/webassembly) -- Emscripten port targeting browsers and NodeJS. + - [zephyr](ports/zephyr) -- Zephyr RTOS. The MicroPython cross-compiler, mpy-cross ----------------------------------------- -Most ports require the MicroPython cross-compiler to be built first. This -program, called mpy-cross, is used to pre-compile Python scripts to .mpy -files which can then be included (frozen) into the firmware/executable for -a port. To build mpy-cross use: +Most ports require the [MicroPython cross-compiler](mpy-cross) to be built +first. This program, called mpy-cross, is used to pre-compile Python scripts +to .mpy files which can then be included (frozen) into the +firmware/executable for a port. To build mpy-cross use: $ cd mpy-cross $ make -The Unix version ----------------- - -The "unix" port requires a standard Unix environment with gcc and GNU make. -x86 and x64 architectures are supported (i.e. x86 32- and 64-bit), as well -as ARM and MIPS. Making full-featured port to another architecture requires -writing some assembly code for the exception handling and garbage collection. -Alternatively, fallback implementation based on setjmp/longjmp can be used. - -To build (see section below for required dependencies): - - $ cd ports/unix - $ make submodules - $ make - -Then to give it a try: - - $ ./micropython - >>> list(5 * x + y for x in range(10) for y in [4, 2, 1]) - -Use `CTRL-D` (i.e. EOF) to exit the shell. -Learn about command-line options (in particular, how to increase heap size -which may be needed for larger applications): - - $ ./micropython -h - -Run complete testsuite: - - $ make test - -Unix version comes with a builtin package manager called upip, e.g.: - - $ ./micropython -m upip install micropython-pystone - $ ./micropython -m pystone - -Browse available modules on -[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython). -Standard library modules come from -[micropython-lib](https://github.com/micropython/micropython-lib) project. - External dependencies --------------------- -Building MicroPython ports may require some dependencies installed. - -For Unix port, `libffi` library and `pkg-config` tool are required. On -Debian/Ubuntu/Mint derivative Linux distros, install `build-essential` -(includes toolchain and make), `libffi-dev`, and `pkg-config` packages. - -Other dependencies can be built together with MicroPython. This may -be required to enable extra features or capabilities, and in recent -versions of MicroPython, these may be enabled by default. To build -these additional dependencies, in the port directory you're -interested in (e.g. `ports/unix/`) first execute: +The core MicroPython VM and runtime has no external dependencies, but a given +port might depend on third-party drivers or vendor HALs. This repository +includes [several submodules](lib/) linking to these external dependencies. +Before compiling a given port, use + $ cd ports/name $ make submodules -This will fetch all the relevant git submodules (sub repositories) that -the port needs. Use the same command to get the latest versions of -submodules as they are updated from time to time. After that execute: - - $ make deplibs - -This will build all available dependencies (regardless whether they -are used or not). If you intend to build MicroPython with additional -options (like cross-compiling), the same set of options should be passed -to `make deplibs`. To actually enable/disable use of dependencies, edit -`ports/unix/mpconfigport.mk` file, which has inline descriptions of the options. -For example, to build SSL module (required for `upip` tool described above, -and so enabled by default), `MICROPY_PY_USSL` should be set to 1. - -For some ports, building required dependences is transparent, and happens -automatically. But they still need to be fetched with the `make submodules` -command. - -The STM32 version ------------------ - -The "stm32" port requires an ARM compiler, arm-none-eabi-gcc, and associated -bin-utils. For those using Arch Linux, you need arm-none-eabi-binutils, -arm-none-eabi-gcc and arm-none-eabi-newlib packages. Otherwise, try here: -https://launchpad.net/gcc-arm-embedded - -To build: - - $ cd ports/stm32 - $ make submodules - $ make - -You then need to get your board into DFU mode. On the pyboard, connect the -3V3 pin to the P1/DFU pin with a wire (on PYBv1.0 they are next to each other -on the bottom left of the board, second row from the bottom). - -Then to flash the code via USB DFU to your device: - - $ make deploy - -This will use the included `tools/pydfu.py` script. If flashing the firmware -does not work it may be because you don't have the correct permissions, and -need to use `sudo make deploy`. -See the README.md file in the ports/stm32/ directory for further details. - -Contributing ------------- - -MicroPython is an open-source project and welcomes contributions. To be -productive, please be sure to follow the -[Contributors' Guidelines](https://github.com/micropython/micropython/wiki/ContributorGuidelines) -and the [Code Conventions](https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md). -Note that MicroPython is licenced under the MIT license, and all contributions -should follow this license. +to ensure that all required submodules are initialised. diff --git a/docs/Makefile b/docs/Makefile index 05709617c35b9..766a669a500dc 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -3,7 +3,7 @@ # You can set these variables from the command line. PYTHON = python3 -SPHINXOPTS = -W --keep-going +SPHINXOPTS = -W --keep-going -j auto SPHINXBUILD = sphinx-build PAPER = BUILDDIR = build/$(MICROPY_PORT) diff --git a/docs/README.md b/docs/README.md index 1591911c34560..892726ba17ffe 100644 --- a/docs/README.md +++ b/docs/README.md @@ -47,7 +47,7 @@ This can be achieved with: make latexpdf -but require rather complete install of LaTeX with various extensions. On -Debian/Ubuntu, try (500MB+ download): +but requires a rather complete install of LaTeX with various extensions. On +Debian/Ubuntu, try (1GB+ download): - apt-get install texlive-latex-recommended texlive-latex-extra + apt install texlive-latex-recommended texlive-latex-extra texlive-xetex texlive-fonts-extra cm-super xindy diff --git a/docs/conf.py b/docs/conf.py index 53cbf9baeddc0..eb61487582024 100755 --- a/docs/conf.py +++ b/docs/conf.py @@ -19,54 +19,59 @@ # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath('.')) +sys.path.insert(0, os.path.abspath(".")) + +# The MICROPY_VERSION env var should be "vX.Y.Z" (or unset). +micropy_version = os.getenv("MICROPY_VERSION") or "latest" +micropy_all_versions = (os.getenv("MICROPY_ALL_VERSIONS") or "latest").split(",") +url_pattern = "%s/en/%%s" % (os.getenv("MICROPY_URL_PREFIX") or "/",) # The members of the html_context dict are available inside topindex.html -micropy_version = os.getenv('MICROPY_VERSION') or 'latest' -micropy_all_versions = (os.getenv('MICROPY_ALL_VERSIONS') or 'latest').split(',') -url_pattern = '%s/en/%%s' % (os.getenv('MICROPY_URL_PREFIX') or '/',) html_context = { - 'cur_version':micropy_version, - 'all_versions':[ - (ver, url_pattern % ver) for ver in micropy_all_versions - ], - 'downloads':[ - ('PDF', url_pattern % micropy_version + '/micropython-docs.pdf'), + "cur_version": micropy_version, + "all_versions": [(ver, url_pattern % ver) for ver in micropy_all_versions], + "downloads": [ + ("PDF", url_pattern % micropy_version + "/micropython-docs.pdf"), ], + "is_release": micropy_version != "latest", } +# Authors used in various parts of the documentation. +micropy_authors = "MicroPython authors and contributors" + # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' +# needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.doctest', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', + "sphinx.ext.autodoc", + "sphinx.ext.doctest", + "sphinx.ext.intersphinx", + "sphinx.ext.todo", + "sphinx.ext.coverage", + "sphinxcontrib.jquery", ] # Add any paths that contain templates here, relative to this directory. -templates_path = ['templates'] +templates_path = ["templates"] # The suffix of source filenames. -source_suffix = '.rst' +source_suffix = ".rst" # The encoding of source files. -#source_encoding = 'utf-8-sig' +# source_encoding = 'utf-8-sig' # The master toctree document. -master_doc = 'index' +master_doc = "index" # General information about the project. -project = 'MicroPython' -copyright = '2014-2021, Damien P. George, Paul Sokolovsky, and contributors' +project = "MicroPython" +copyright = "- The MicroPython Documentation is Copyright © 2014-2025, " + micropy_authors # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -74,45 +79,45 @@ # # We don't follow "The short X.Y version" vs "The full version, including alpha/beta/rc tags" # breakdown, so use the same version identifier for both to avoid confusion. -version = release = '1.15' +version = release = micropy_version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. -#language = None +# language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: -#today = '' +# today = '' # Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' +# today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = ['build', '.venv'] +exclude_patterns = ["build", ".venv"] # The reST default role (used for this markup: `text`) to use for all # documents. -default_role = 'any' +default_role = "any" # If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True +# add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). -#add_module_names = True +# add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. -#show_authors = False +# show_authors = False # The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' +pygments_style = "sphinx" # A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] +# modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False +# keep_warnings = False # Global include files. Sphinx docs suggest using rst_epilog in preference # of rst_prolog, so we follow. Absolute paths below mean "from the base @@ -124,153 +129,168 @@ # -- Options for HTML output ---------------------------------------------- # on_rtd is whether we are on readthedocs.org -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' +on_rtd = os.environ.get("READTHEDOCS", None) == "True" if not on_rtd: # only import and set the theme if we're building docs locally try: import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path(), '.'] + + html_theme = "sphinx_rtd_theme" + html_theme_path = [sphinx_rtd_theme.get_html_theme_path(), "."] except: - html_theme = 'default' - html_theme_path = ['.'] + html_theme = "default" + html_theme_path = ["."] else: - html_theme_path = ['.'] + html_theme_path = ["."] # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. -#html_theme_options = {} +# html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. # html_theme_path = ['.'] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". -#html_title = None +# html_title = None # A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None +# html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. -#html_logo = '../../logo/trans-logo.png' +# html_logo = '../../logo/trans-logo.png' # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. -html_favicon = 'static/favicon.ico' +html_favicon = "static/favicon.ico" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['static'] +html_static_path = ["static"] +# Add a custom CSS file for HTML generation +html_css_files = [ + "custom.css", +] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. -#html_extra_path = [] +# html_extra_path = [] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. -html_last_updated_fmt = '%d %b %Y' +html_last_updated_fmt = "%d %b %Y" # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. -#html_use_smartypants = True +# html_use_smartypants = True # Custom sidebar templates, maps document names to template names. -#html_sidebars = {} +# html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. html_additional_pages = {"index": "topindex.html"} # If false, no module index is generated. -#html_domain_indices = True +# html_domain_indices = True # If false, no index is generated. -#html_use_index = True +# html_use_index = True # If true, the index is split into individual pages for each letter. -#html_split_index = False +# html_split_index = False # If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True +# html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True +# html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True +# html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. -#html_use_opensearch = '' +# html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None +# html_file_suffix = None # Output file base name for HTML help builder. -htmlhelp_basename = 'MicroPythondoc' +htmlhelp_basename = "MicroPythondoc" # -- Options for LaTeX output --------------------------------------------- latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -#'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -#'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -#'preamble': '', -# Include 3 levels of headers in PDF ToC -'preamble': '\setcounter{tocdepth}{2}', + # The paper size ('letterpaper' or 'a4paper'). + #'papersize': 'letterpaper', + # The font size ('10pt', '11pt' or '12pt'). + #'pointsize': '10pt', + # Additional stuff for the LaTeX preamble. + #'preamble': '', + # Include 3 levels of headers in PDF ToC + "preamble": r"\setcounter{tocdepth}{2}", } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'MicroPython.tex', 'MicroPython Documentation', - 'Damien P. George, Paul Sokolovsky, and contributors', 'manual'), + ( + master_doc, + "MicroPython.tex", + "MicroPython Documentation", + micropy_authors, + "manual", + ), ] # The name of an image file (relative to this directory) to place at the top of # the title page. -#latex_logo = None +# latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. -#latex_use_parts = False +# latex_use_parts = False # If true, show page references after internal links. -#latex_show_pagerefs = False +# latex_show_pagerefs = False # If true, show URL addresses after external links. -#latex_show_urls = False +# latex_show_urls = False # Documents to append as an appendix to all manuals. -#latex_appendices = [] +# latex_appendices = [] # If false, no module index is generated. -#latex_domain_indices = True +# latex_domain_indices = True +# Enable better Unicode support so that `make latexpdf` doesn't fail +latex_engine = "xelatex" # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - ('index', 'micropython', 'MicroPython Documentation', - ['Damien P. George, Paul Sokolovsky, and contributors'], 1), + ( + "index", + "micropython", + "MicroPython Documentation", + [micropy_authors], + 1, + ), ] # If true, show URL addresses after external links. -#man_show_urls = False +# man_show_urls = False # -- Options for Texinfo output ------------------------------------------- @@ -279,23 +299,29 @@ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'MicroPython', 'MicroPython Documentation', - 'Damien P. George, Paul Sokolovsky, and contributors', 'MicroPython', 'One line description of project.', - 'Miscellaneous'), + ( + master_doc, + "MicroPython", + "MicroPython Documentation", + micropy_authors, + "MicroPython", + "One line description of project.", + "Miscellaneous", + ), ] # Documents to append as an appendix to all manuals. -#texinfo_appendices = [] +# texinfo_appendices = [] # If false, no module index is generated. -#texinfo_domain_indices = True +# texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' +# texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False +# texinfo_no_detailmenu = False # Example configuration for intersphinx: refer to the Python standard library. -intersphinx_mapping = {'python': ('https://docs.python.org/3.5', None)} +intersphinx_mapping = {"python": ("https://docs.python.org/3.5", None)} diff --git a/docs/develop/cmodules.rst b/docs/develop/cmodules.rst index 38225e8686a7b..c5aa919b7645a 100644 --- a/docs/develop/cmodules.rst +++ b/docs/develop/cmodules.rst @@ -49,9 +49,17 @@ A MicroPython user C module is a directory with the following files: in your ``micropython.mk`` to a local make variable, eg ``EXAMPLE_MOD_DIR := $(USERMOD_DIR)`` - Your ``micropython.mk`` must add your modules source files relative to your - expanded copy of ``$(USERMOD_DIR)`` to ``SRC_USERMOD``, eg - ``SRC_USERMOD += $(EXAMPLE_MOD_DIR)/example.c`` + Your ``micropython.mk`` must add your modules source files to the + ``SRC_USERMOD_C`` or ``SRC_USERMOD_LIB_C`` variables. The former will be + processed for ``MP_QSTR_`` and ``MP_REGISTER_MODULE`` definitions, the latter + will not (e.g. helpers and library code that isn't MicroPython-specific). + These paths should include your expanded copy of ``$(USERMOD_DIR)``, e.g.:: + + SRC_USERMOD_C += $(EXAMPLE_MOD_DIR)/modexample.c + SRC_USERMOD_LIB_C += $(EXAMPLE_MOD_DIR)/utils/algorithm.c + + Similarly, use ``SRC_USERMOD_CXX`` and ``SRC_USERMOD_LIB_CXX`` for C++ + source files. If you want to include assembly files use ``SRC_USERMOD_LIB_ASM``. If you have custom compiler options (like ``-I`` to add directories to search for header files), these should be added to ``CFLAGS_USERMOD`` for C code @@ -87,9 +95,12 @@ A MicroPython user C module is a directory with the following files: Basic example ------------- -This simple module named ``cexample`` provides a single function -``cexample.add_ints(a, b)`` which adds the two integer args together and returns -the result. It can be found in the MicroPython source tree +The ``cexample`` module provides examples for a function and a class. The +``cexample.add_ints(a, b)`` function adds two integer args together and returns +the result. The ``cexample.Timer()`` type creates timers that can be used to +measure the elapsed time since the object is instantiated. + +The module can be found in the MicroPython source tree `in the examples directory `_ and has a source file and a Makefile fragment with content as described above:: @@ -221,23 +232,25 @@ as described above. If a module is not enabled by default then the corresponding C preprocessor macro must be enabled. This macro name can be found by searching for the ``MP_REGISTER_MODULE`` line in the module's source code (it usually appears at the end of the main source file). -The third argument to ``MP_REGISTER_MODULE`` is the macro name, and this must be set -to 1 using ``CFLAGS_EXTRA`` to make the module available. If the third argument is just -the number 1 then the module is enabled by default. +This macro should be surrounded by a ``#if X`` / ``#endif`` pair, and the configuration +option ``X`` must be set to 1 using ``CFLAGS_EXTRA`` to make the module available. If +there is no ``#if X`` / ``#endif`` pair then the module is enabled by default. For example, the ``examples/usercmodule/cexample`` module is enabled by default so has the following line in its source code: .. code-block:: c - MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule, 1); + MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule); Alternatively, to make this module disabled by default but selectable through a preprocessor configuration option, it would be: .. code-block:: c - MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule, MODULE_CEXAMPLE_ENABLED); + #if MODULE_CEXAMPLE_ENABLED + MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule); + #endif In this case the module is enabled by adding ``CFLAGS_EXTRA=-DMODULE_CEXAMPLE_ENABLED=1`` to the ``make`` command, or editing ``mpconfigport.h`` or ``mpconfigboard.h`` to add @@ -262,3 +275,13 @@ can now be accessed in Python just like any other builtin module, e.g. import cexample print(cexample.add_ints(1, 3)) # should display 4 + +.. code-block:: python + + from cexample import Timer + from time import sleep_ms + + watch = Timer() + sleep_ms(1000) + print(watch.time()) + # should display approximately 1000 diff --git a/docs/develop/compiler.rst b/docs/develop/compiler.rst index 2007657490f55..0c25ad3a01ef1 100644 --- a/docs/develop/compiler.rst +++ b/docs/develop/compiler.rst @@ -98,7 +98,7 @@ Then also edit ``py/lexer.c`` to add the new keyword literal text: .. code-block:: c :emphasize-lines: 12 - STATIC const char *const tok_kw[] = { + static const char *const tok_kw[] = { ... "or", "pass", @@ -147,10 +147,17 @@ The most relevant method you should know about is this: .. code-block:: c mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, bool is_repl) { + // Create a context for this module, and set its globals dict. + mp_module_context_t *context = m_new_obj(mp_module_context_t); + context->module.globals = mp_globals_get(); + // Compile the input parse_tree to a raw-code structure. - mp_raw_code_t *rc = mp_compile_to_raw_code(parse_tree, source_file, is_repl); + mp_compiled_module_t cm; + cm.context = context; + mp_compile_to_raw_code(parse_tree, source_file, is_repl, &cm); + // Create and return a function object that executes the outer module. - return mp_make_function_from_raw_code(rc, MP_OBJ_NULL, MP_OBJ_NULL); + return mp_make_function_from_proto_fun(cm.rc, cm.context, NULL); } The compiler compiles the code in four passes: scope, stack size, code size and emit. @@ -294,7 +301,7 @@ code statement: .. code-block:: c - STATIC void emit_native_unary_op(emit_t *emit, mp_unary_op_t op) { + static void emit_native_unary_op(emit_t *emit, mp_unary_op_t op) { vtype_kind_t vtype; emit_pre_pop_reg(emit, &vtype, REG_ARG_2); if (vtype == VTYPE_PYOBJ) { diff --git a/docs/develop/extendingmicropython.rst b/docs/develop/extendingmicropython.rst index 7fb1ae47a0e4e..b4029c421c89a 100644 --- a/docs/develop/extendingmicropython.rst +++ b/docs/develop/extendingmicropython.rst @@ -16,4 +16,3 @@ live in the main MicroPython repository. cmodules.rst natmod.rst - \ No newline at end of file diff --git a/docs/develop/gettingstarted.rst b/docs/develop/gettingstarted.rst index 30b26071ea012..fed632ea1ac15 100644 --- a/docs/develop/gettingstarted.rst +++ b/docs/develop/gettingstarted.rst @@ -4,7 +4,7 @@ Getting Started =============== This guide covers a step-by-step process on setting up version control, obtaining and building -a copy of the source code for a port, building the documentation, running tests, and a description of the +a copy of the source code for a port, building the documentation, running tests, and a description of the directory structure of the MicroPython code base. Source control with git @@ -16,7 +16,7 @@ code is pulled and pushed to and from the main repository. Install the respectiv of Git for your operating system to follow through the rest of the steps. .. note:: - For a reference on the installation instructions, please refer to + For a reference on the installation instructions, please refer to the `Git installation instructions `_. Learn about the basic git commands in this `Git Handbook `_ or any other sources on the internet. @@ -58,12 +58,12 @@ MicroPython repository. After the above configuration, your setup should be similar to this: .. code-block:: bash - + $ git remote -v - origin https://github.com//micropython (fetch) - origin https://github.com//micropython (push) - upstream https://github.com/micropython/micropython (fetch) - upstream https://github.com/micropython/micropython (push) + origin https://github.com//micropython (fetch) + origin https://github.com//micropython (push) + upstream https://github.com/micropython/micropython (fetch) + upstream https://github.com/micropython/micropython (push) You should now have a copy of the source code. By default, you are pointing to the master branch. To prepare for further development, it is recommended @@ -73,7 +73,7 @@ to work on a development branch. $ git checkout -b dev-branch -You can give it any name. You will have to compile MicroPython whenever you change +You can give it any name. You will have to compile MicroPython whenever you change to a different branch. Compile and build the code @@ -100,10 +100,10 @@ For the stm32 port, the ARM cross-compiler is required: .. code-block:: bash - $ sudo apt-get install arm-none-eabi-gcc arm-none-eabi-binutils arm-none-eabi-newlib + $ sudo apt-get install gcc-arm-none-eabi libnewlib-arm-none-eabi See the `ARM GCC -toolchain `_ +toolchain `_ for the latest details. Python is also required. Python 2 is supported for now, but we recommend using Python 3. @@ -112,7 +112,7 @@ Check that you have Python available on your system: .. code-block:: bash $ python3 - Python 3.5.0 (default, Jul 17 2020, 14:04:10) + Python 3.5.0 (default, Jul 17 2020, 14:04:10) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> @@ -140,8 +140,8 @@ If it built successfully, you should see a message similar to this: .. code-block:: bash LINK mpy-cross - text data bss dec hex filename - 279328 776 880 280984 44998 mpy-cross + text data bss dec hex filename + 279328 776 880 280984 44998 mpy-cross .. note:: @@ -181,8 +181,8 @@ If MicroPython built correctly, you should see the following: .. code-block:: bash LINK micropython - text data bss dec hex filename - 412033 5680 2496 420209 66971 micropython + text data bss dec hex filename + 412033 5680 2496 420209 66971 micropython Now run it: @@ -215,7 +215,7 @@ as detailed in the :ref:`required_dependencies` section, then build: $ make submodules $ make -Please refer to the `stm32 documentation `_ +Please refer to the `stm32 documentation `_ for more details on flashing the firmware. .. note:: @@ -228,7 +228,7 @@ You can also specify which board to use: .. code-block:: bash $ cd ports/stm32 - $ make submodules + $ make BOARD= submodules $ make BOARD= See `ports/stm32/boards `_ @@ -245,7 +245,7 @@ that you use a virtual environment: $ python3 -m venv env $ source env/bin/activate - $ pip install sphinx + $ pip install -r docs/requirements.txt Navigate to the ``docs`` directory: @@ -259,7 +259,7 @@ Build the docs: $ make html -Open ``docs/build/html/index.html`` in your browser to view the docs locally. Refer to the +Open ``docs/build/html/index.html`` in your browser to view the docs locally. Refer to the documentation on `importing your documentation `_ to use Read the Docs. @@ -278,7 +278,7 @@ To run a selection of tests on a board/device connected over USB use: .. code-block:: bash $ cd tests - $ ./run-tests.py --target minimal --device /dev/ttyACM0 + $ ./run-tests.py -t /dev/ttyACM0 See also :ref:`writingtests`. @@ -322,7 +322,8 @@ tests tools - Contains helper tools including the ``upip`` and the ``pyboard.py`` module. + Contains scripts used by the build and CI process, as well as user tools such + as ``pyboard.py`` and ``mpremote``. examples diff --git a/docs/develop/index.rst b/docs/develop/index.rst index 7a6a6be67c5b9..327038f1978bd 100644 --- a/docs/develop/index.rst +++ b/docs/develop/index.rst @@ -5,7 +5,7 @@ This chapter covers a tour of MicroPython from the perspective of a developer, c to MicroPython. It acts as a comprehensive resource on the implementation details of MicroPython for both novice and expert contributors. -Development around MicroPython usually involves modifying the core runtime, porting or +Development around MicroPython usually involves modifying the core runtime, porting or maintaining a new library. This guide describes at great depth, the implementation details of MicroPython including a getting started guide, compiler internals, porting MicroPython to a new platform and implementing a core MicroPython library. @@ -24,4 +24,3 @@ MicroPython to a new platform and implementing a core MicroPython library. publiccapi.rst extendingmicropython.rst porting.rst - \ No newline at end of file diff --git a/docs/develop/library.rst b/docs/develop/library.rst index bebddcc8a3605..830211d81c862 100644 --- a/docs/develop/library.rst +++ b/docs/develop/library.rst @@ -34,7 +34,7 @@ An example is the ``gc`` module discussed in :ref:`memorymanagement`. >>> gc.enable() >>> -MicroPython has several other builtin standard/core modules like ``io``, ``uarray`` etc. +MicroPython has several other builtin standard/core modules like ``io``, ``array`` etc. Adding a new core module involves several modifications. First, create the ``C`` file in the ``py/`` directory. In this example we are adding a @@ -48,23 +48,23 @@ hypothetical new module ``subsystem`` in the file ``modsubsystem.c``: #if MICROPY_PY_SUBSYSTEM // info() - STATIC mp_obj_t py_subsystem_info(void) { + static mp_obj_t py_subsystem_info(void) { return MP_OBJ_NEW_SMALL_INT(42); } MP_DEFINE_CONST_FUN_OBJ_0(subsystem_info_obj, py_subsystem_info); - STATIC const mp_rom_map_elem_t mp_module_subsystem_globals_table[] = { + static const mp_rom_map_elem_t mp_module_subsystem_globals_table[] = { { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_subsystem) }, { MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&subsystem_info_obj) }, }; - STATIC MP_DEFINE_CONST_DICT(mp_module_subsystem_globals, mp_module_subsystem_globals_table); + static MP_DEFINE_CONST_DICT(mp_module_subsystem_globals, mp_module_subsystem_globals_table); const mp_obj_module_t mp_module_subsystem = { .base = { &mp_type_module }, .globals = (mp_obj_dict_t *)&mp_module_subsystem_globals, }; - MP_REGISTER_MODULE(MP_QSTR_subsystem, mp_module_subsystem, MICROPY_PY_SUBSYSTEM); + MP_REGISTER_MODULE(MP_QSTR_subsystem, mp_module_subsystem); #endif diff --git a/docs/develop/natmod.rst b/docs/develop/natmod.rst index 8ffe49591cc69..18678eaefbf11 100644 --- a/docs/develop/natmod.rst +++ b/docs/develop/natmod.rst @@ -34,11 +34,13 @@ options for the ``ARCH`` variable, see below): * ``x86`` (32 bit) * ``x64`` (64 bit x86) +* ``armv6m`` (ARM Thumb, eg Cortex-M0) * ``armv7m`` (ARM Thumb 2, eg Cortex-M3) * ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7) * ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7) * ``xtensa`` (non-windowed, eg ESP8266) -* ``xtensawin`` (windowed with window size 8, eg ESP32) +* ``xtensawin`` (windowed with window size 8, eg ESP32, ESP32S3) +* ``rv32imc`` (RISC-V 32 bits with compressed instructions, eg ESP32C3, ESP32C6) When compiling and linking the native .mpy file the architecture must be chosen and the corresponding file can only be imported on that architecture. For more @@ -68,6 +70,13 @@ The known limitations are: So, if your C code has writable data, make sure the data is defined globally, without an initialiser, and only written to within functions. +The native module is not automatically linked against the standard static libraries +like ``libm.a`` and ``libgcc.a``, which can lead to ``undefined symbol`` errors. +You can link the runtime libraries by setting ``LINK_RUNTIME = 1`` +in your Makefile. Custom static libraries can also be linked by adding +``MPY_LD_FLAGS += -l path/to/library.a``. Note that these are linked into +the native module and will not be shared with other modules or the system. + Linker limitation: the native module is not linked against the symbol table of the full MicroPython firmware. Rather, it is linked against an explicit table of exported symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware @@ -127,7 +136,7 @@ The file ``factorial.c`` contains: #include "py/dynruntime.h" // Helper function to compute factorial - STATIC mp_int_t factorial_helper(mp_int_t x) { + static mp_int_t factorial_helper(mp_int_t x) { if (x == 0) { return 1; } @@ -135,7 +144,7 @@ The file ``factorial.c`` contains: } // This is the function which will be called from Python, as factorial(x) - STATIC mp_obj_t factorial(mp_obj_t x_obj) { + static mp_obj_t factorial(mp_obj_t x_obj) { // Extract the integer from the MicroPython input object mp_int_t x = mp_obj_get_int(x_obj); // Calculate the factorial @@ -144,7 +153,7 @@ The file ``factorial.c`` contains: return mp_obj_new_int(result); } // Define a Python reference to the function above - STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial); + static MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial); // This is the entry point and is called when the module is imported mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) { @@ -171,7 +180,7 @@ The file ``Makefile`` contains: # Source files (.c or .py) SRC = factorial.c - # Architecture to build for (x86, x64, armv7m, xtensa, xtensawin) + # Architecture to build for (x86, x64, armv6m, armv7m, xtensa, xtensawin, rv32imc) ARCH = x64 # Include to get the rules for compiling and linking the module diff --git a/docs/develop/optimizations.rst b/docs/develop/optimizations.rst index d972cde66616a..533dd64116f03 100644 --- a/docs/develop/optimizations.rst +++ b/docs/develop/optimizations.rst @@ -25,13 +25,15 @@ into the firmware image as part of the main firmware compilation process, which the bytecode will be executed from ROM. This can lead to a significant memory saving, and reduce heap fragmentation. +See :ref:`manifest` for more information. + Variables --------- MicroPython processes local and global variables differently. Global variables are stored and looked up from a global dictionary that is allocated on the heap (note that each module has its own separate dict, so separate namespace). -Local variables on the other hand are are stored on the Python value stack, which may +Local variables on the other hand are stored on the Python value stack, which may live on the C stack or on the heap. They are accessed directly by their offset within the Python stack, which is more efficient than a global lookup in a dict. diff --git a/docs/develop/porting.rst b/docs/develop/porting.rst index 59dd570008bcd..99725e1000bef 100644 --- a/docs/develop/porting.rst +++ b/docs/develop/porting.rst @@ -38,12 +38,13 @@ The basic MicroPython firmware is implemented in the main port file, e.g ``main. .. code-block:: c + #include "py/builtin.h" #include "py/compile.h" #include "py/gc.h" #include "py/mperrno.h" #include "py/stackctrl.h" - #include "lib/utils/gchelper.h" - #include "lib/utils/pyexec.h" + #include "shared/runtime/gchelper.h" + #include "shared/runtime/pyexec.h" // Allocate memory for the MicroPython GC heap. static char heap[4096]; @@ -53,8 +54,6 @@ The basic MicroPython firmware is implemented in the main port file, e.g ``main. mp_stack_ctrl_init(); gc_init(heap, heap + sizeof(heap)); mp_init(); - mp_obj_list_init(MP_OBJ_TO_PTR(mp_sys_path), 0); - mp_obj_list_init(MP_OBJ_TO_PTR(mp_sys_argv), 0); // Start a normal REPL; will exit when ctrl-D is entered on a blank line. pyexec_friendly_repl(); @@ -84,7 +83,7 @@ The basic MicroPython firmware is implemented in the main port file, e.g ``main. } // There is no filesystem so opening a file raises an exception. - mp_lexer_t *mp_lexer_new_from_file(const char *filename) { + mp_lexer_t *mp_lexer_new_from_file(qstr filename) { mp_raise_OSError(MP_ENOENT); } @@ -97,19 +96,23 @@ We also need a Makefile at this point for the port: # Include py core make definitions. include $(TOP)/py/py.mk + include $(TOP)/extmod/extmod.mk # Set CFLAGS and libraries. - CFLAGS = -I. -I$(BUILD) -I$(TOP) - LIBS = -lm + CFLAGS += -I. -I$(BUILD) -I$(TOP) + LIBS += -lm # Define the required source files. SRC_C = \ main.c \ mphalport.c \ - lib/mp-readline/readline.c \ - lib/utils/gchelper_generic.c \ - lib/utils/pyexec.c \ - lib/utils/stdout_helpers.c \ + shared/readline/readline.c \ + shared/runtime/gchelper_generic.c \ + shared/runtime/pyexec.c \ + shared/runtime/stdout_helpers.c \ + + # Define source files containung qstrs. + SRC_QSTR += shared/readline/readline.c shared/runtime/pyexec.c # Define the required object files. OBJ = $(PY_CORE_O) $(addprefix $(BUILD)/, $(SRC_C:.c=.o)) @@ -172,9 +175,6 @@ The following is an example of an ``mpconfigport.h`` file: #define MP_STATE_PORT MP_STATE_VM - #define MICROPY_PORT_ROOT_POINTERS \ - const char *readline_hist[8]; - This configuration file contains machine-specific configurations including aspects like if different MicroPython features should be enabled e.g. ``#define MICROPY_ENABLE_GC (1)``. Making this Setting ``(0)`` disables the feature. @@ -245,8 +245,8 @@ That should give a MicroPython REPL. You can then run commands like: .. code-block:: bash MicroPython v1.13 on 2021-01-01; example-board with unknown-cpu - >>> import usys - >>> usys.implementation + >>> import sys + >>> sys.implementation ('micropython', (1, 13, 0)) >>> @@ -262,29 +262,24 @@ To add a custom module like ``myport``, first add the module definition in a fil #include "py/runtime.h" - STATIC mp_obj_t myport_info(void) { + static mp_obj_t myport_info(void) { mp_printf(&mp_plat_print, "info about my port\n"); return mp_const_none; } - STATIC MP_DEFINE_CONST_FUN_OBJ_0(myport_info_obj, myport_info); + static MP_DEFINE_CONST_FUN_OBJ_0(myport_info_obj, myport_info); - STATIC const mp_rom_map_elem_t myport_module_globals_table[] = { + static const mp_rom_map_elem_t myport_module_globals_table[] = { { MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_myport) }, { MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&myport_info_obj) }, }; - STATIC MP_DEFINE_CONST_DICT(myport_module_globals, myport_module_globals_table); + static MP_DEFINE_CONST_DICT(myport_module_globals, myport_module_globals_table); const mp_obj_module_t myport_module = { .base = { &mp_type_module }, .globals = (mp_obj_dict_t *)&myport_module_globals, }; - MP_REGISTER_MODULE(MP_QSTR_myport, myport_module, 1); - -Note: the "1" as the third argument in ``MP_REGISTER_MODULE`` enables this new module -unconditionally. To allow it to be conditionally enabled, replace the "1" by -``MICROPY_PY_MYPORT`` and then add ``#define MICROPY_PY_MYPORT (1)`` in ``mpconfigport.h`` -accordingly. + MP_REGISTER_MODULE(MP_QSTR_myport, myport_module); You will also need to edit the Makefile to add ``modmyport.c`` to the ``SRC_C`` list, and a new line adding the same file to ``SRC_QSTR`` (so qstrs are searched for in this new file), @@ -298,7 +293,7 @@ like this: mphalport.c \ ... - SRC_QSTR += modport.c + SRC_QSTR += modmyport.c If all went correctly then, after rebuilding, you should be able to import the new module: diff --git a/docs/develop/qstr.rst b/docs/develop/qstr.rst index cd1fc47862cf0..5d1ac5bb4a49f 100644 --- a/docs/develop/qstr.rst +++ b/docs/develop/qstr.rst @@ -59,7 +59,7 @@ Processing happens in the following stages: information. Note that this step only uses files that have changed, which means that ``qstr.i.last`` will only contain data from files that have changed since the last compile. - + 2. ``qstr.split`` is an empty file created after running ``makeqstrdefs.py split`` on qstr.i.last. It's just used as a dependency to indicate that the step ran. This script outputs one file per input C file, ``genhdr/qstr/...file.c.qstr``, diff --git a/docs/develop/writingtests.rst b/docs/develop/writingtests.rst index 9bb5178f55f22..a7d033f17e9b9 100644 --- a/docs/develop/writingtests.rst +++ b/docs/develop/writingtests.rst @@ -60,7 +60,7 @@ Then to run on a board: .. code-block:: bash - $ ./run-tests.py --target minimal --device /dev/ttyACM0 + $ ./run-tests.py -t /dev/ttyACM0 And to run only a certain set of tests (eg a directory): diff --git a/docs/differences/index_template.txt b/docs/differences/index_template.txt index 41ddeb6d3e4aa..1bbfbfedce921 100644 --- a/docs/differences/index_template.txt +++ b/docs/differences/index_template.txt @@ -3,8 +3,22 @@ MicroPython differences from CPython ==================================== -The operations listed in this section produce conflicting results in MicroPython when compared to standard Python. -MicroPython implements Python 3.4 and some select features of Python 3.5. +MicroPython implements Python 3.4 and some select features of Python 3.5 and +above. The sections below describe the current status of these features. + +.. toctree:: + + ../differences/python_35.rst + ../differences/python_36.rst + ../differences/python_37.rst + ../differences/python_38.rst + ../differences/python_39.rst + ../differences/python_310.rst + +For the features of Python that are implemented by MicroPython, there are +sometimes differences in their behaviour compared to standard Python. The +operations listed in the sections below produce conflicting results in +MicroPython when compared to standard Python. .. toctree:: :maxdepth: 2 diff --git a/docs/differences/modules_preamble.txt b/docs/differences/modules_preamble.txt new file mode 100644 index 0000000000000..1958f0084d2db --- /dev/null +++ b/docs/differences/modules_preamble.txt @@ -0,0 +1,33 @@ +.. Preamble section inserted into generated output + +Positional-only Parameters +-------------------------- + +To save code size, many functions that accept keyword arguments in CPython only accept positional arguments in MicroPython. + +MicroPython marks positional-only parameters in the same way as CPython, by inserting a ``/`` to mark the end of the positional parameters. Any function whose signature ends in ``/`` takes *only* positional arguments. For more details, see `PEP 570 `_. + +Example +~~~~~~~ + +For example, in CPython 3.4 this is the signature of the constructor ``socket.socket``:: + + socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None) + +However, the signature documented in :func:`MicroPython` is:: + + socket(af=AF_INET, type=SOCK_STREAM, proto=IPPROTO_TCP, /) + +The ``/`` at the end of the parameters indicates that they are all positional-only in MicroPython. The following code works in CPython but not in most MicroPython ports:: + + import socket + s = socket.socket(type=socket.SOCK_DGRAM) + +MicroPython will raise an exception:: + + TypeError: function doesn't take keyword arguments + +The following code will work in both CPython and MicroPython:: + + import socket + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) diff --git a/docs/differences/python_310.rst b/docs/differences/python_310.rst new file mode 100644 index 0000000000000..f9d92db655372 --- /dev/null +++ b/docs/differences/python_310.rst @@ -0,0 +1,238 @@ +.. _python_310: + +Python 3.10 +=========== + +Python 3.10.0 (final) was released on the 4 October 2021. The Features for 3.10 are +defined in `PEP 619 `_ +and a detailed description of the changes can be found in +`What's New in Python 3.10 `_. + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **New syntax features** | **Status** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 634 `_ | Structural Pattern Matching: Specification | [#spm]_ | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 635 `_ | Structural Pattern Matching: Motivation and | [#spm]_ | + | | Rationale | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 636 `_ | Structural Pattern Matching: Tutorial | [#spm]_ | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `bpo-12782 | Parenthesized context managers are now officially | | + | `_ | allowed | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **New features in the standard library** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 618 `_ | Add Optional Length-Checking To zip | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **Interpreter improvements** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 626 `_ | Precise line numbers for debugging and other tools | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **New typing features** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 604 `_ | Allow writing union types as X | Y | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 613 `_ | Explicit Type Aliases | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 612 `_ | Parameter Specification Variables | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | **Important deprecations, removals or restrictions** | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 644 `_ | Require OpenSSL 1.1.1 or newer | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 632 `_ | Deprecate distutils module. | Not relevant | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 623 `_ | Deprecate and prepare for the removal of the wstr | Not relevant | + | | member in PyUnicodeObject. | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 624 `_ | Remove Py_UNICODE encoder APIs | Not relevant | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + | `PEP 597 `_ | Add optional EncodingWarning | | + +--------------------------------------------------------+----------------------------------------------------+--------------+ + + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-------------------------------------------------------------------------------------------------------------+---------------+ + | The :class:`int` type has a new method :meth:`int.bit_count`, returning the | | + | number of ones in the binary expansion of a given integer, also known | | + | as the population count. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | The views returned by :meth:`dict.keys`, :meth:`dict.values` and | | + | :meth:`dict.items` now all have a ``mapping`` attribute that gives a | | + | :class:`types.MappingProxyType` object wrapping the original | | + | dictionary. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | :pep:`618`: The :func:`zip` function now has an optional ``strict`` flag, used | | + | to require that all the iterables have an equal length. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Builtin and extension functions that take integer arguments no longer accept | | + | :class:`~decimal.Decimal`\ s, :class:`~fractions.Fraction`\ s and other | | + | objects that can be converted to integers only with a loss (e.g. that have | | + | the :meth:`~object.__int__` method but do not have the | | + | :meth:`~object.__index__` method). | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | If :func:`object.__ipow__` returns :const:`NotImplemented`, the operator will | | + | correctly fall back to :func:`object.__pow__` and :func:`object.__rpow__` as expected. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Assignment expressions can now be used unparenthesized within set literals | | + | and set comprehensions, as well as in sequence indexes (but not slices). | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Functions have a new ``__builtins__`` attribute which is used to look for | | + | builtin symbols when a function is executed, instead of looking into | | + | ``__globals__['__builtins__']``. The attribute is initialized from | | + | ``__globals__["__builtins__"]`` if it exists, else from the current builtins. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Two new builtin functions -- :func:`aiter` and :func:`anext` have been added | | + | to provide asynchronous counterparts to :func:`iter` and :func:`next`, | | + | respectively. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Static methods (:func:`@staticmethod `) and class methods | | + | (:func:`@classmethod `) now inherit the method attributes | | + | (``__module__``, ``__name__``, ``__qualname__``, ``__doc__``, | | + | ``__annotations__``) and have a new ``__wrapped__`` attribute. | | + | Moreover, static methods are now callable as regular functions. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Annotations for complex targets (everything beside ``simple name`` targets | | + | defined by :pep:`526`) no longer cause any runtime effects with ``from __future__ import annotations``. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Class and module objects now lazy-create empty annotations dicts on demand. | | + | The annotations dicts are stored in the object’s ``__dict__`` for | | + | backwards compatibility. This improves the best practices for working | | + | with ``__annotations__``. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Annotations consist of ``yield``, ``yield from``, ``await`` or named expressions | | + | are now forbidden under ``from __future__ import annotations`` due to their side | | + | effects. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Usage of unbound variables, ``super()`` and other expressions that might | | + | alter the processing of symbol table as annotations are now rendered | | + | effectless under ``from __future__ import annotations``. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Hashes of NaN values of both :class:`float` type and | | + | :class:`decimal.Decimal` type now depend on object identity. Formerly, they | | + | always hashed to ``0`` even though NaN values are not equal to one another. | | + | This caused potentially quadratic runtime behavior due to excessive hash | | + | collisions when creating dictionaries and sets containing multiple NaNs. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | A :exc:`SyntaxError` (instead of a :exc:`NameError`) will be raised when deleting | | + | the :const:`__debug__` constant. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | :exc:`SyntaxError` exceptions now have ``end_lineno`` and | | + | ``end_offset`` attributes. They will be ``None`` if not determined. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `asyncio `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add missing :meth:`~asyncio.events.AbstractEventLoop.connect_accepted_socket` | | + | method. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `array `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The :meth:`~array.array.index` method of :class:`array.array` now has | | + | optional *start* and *stop* parameters. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `gc `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add audit hooks for :func:`gc.get_objects`, :func:`gc.get_referrers` and | | + | :func:`gc.get_referents`. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `hashlib `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The hashlib module requires OpenSSL 1.1.1 or newer. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The hashlib module has preliminary support for OpenSSL 3.0.0. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The pure-Python fallback of :func:`~hashlib.pbkdf2_hmac` is deprecated. In | | + | the future PBKDF2-HMAC will only be available when Python has been built with | | + | OpenSSL support. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `os `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :func:`os.cpu_count()` support for VxWorks RTOS. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add a new function :func:`os.eventfd` and related helpers to wrap the | | + | ``eventfd2`` syscall on Linux. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :func:`os.splice()` that allows to move data between two file | | + | descriptors without copying between kernel address space and user | | + | address space, where one of the file descriptors must refer to a | | + | pipe. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :data:`~os.O_EVTONLY`, :data:`~os.O_FSYNC`, :data:`~os.O_SYMLINK` | | + | and :data:`~os.O_NOFOLLOW_ANY` for macOS. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `platform `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :func:`platform.freedesktop_os_release()` to retrieve operation system | | + | identification from `freedesktop.org os-release | | + | `_ standard file. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `socket `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The exception :exc:`socket.timeout` is now an alias of :exc:`TimeoutError`. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add option to create MPTCP sockets with ``IPPROTO_MPTCP``. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add ``IP_RECVTOS`` option to receive the type of service (ToS) or DSCP/ECN fields. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `ssl `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module requires OpenSSL 1.1.1 or newer. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module has preliminary support for OpenSSL 3.0.0 and new option | | + | :data:`~ssl.OP_IGNORE_UNEXPECTED_EOF`. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Deprecated function and use of deprecated constants now result in | | + | a :exc:`DeprecationWarning`. :attr:`ssl.SSLContext.options` has | | + | :data:`~ssl.OP_NO_SSLv2` and :data:`~ssl.OP_NO_SSLv3` set by default and | | + | therefore cannot warn about setting the flag again. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module now has more secure default settings. Ciphers without forward | | + | secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits | | + | weak RSA, DH, and ECC keys with less than 112 bits of security. | | + | :class:`~ssl.SSLContext` defaults to minimum protocol version TLS 1.2. | | + | Settings are based on Hynek Schlawack's research. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer | | + | officially supported. Python does not block them actively. However | | + | OpenSSL build options, distro configurations, vendor patches, and cipher | | + | suites may prevent a successful handshake. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add a *timeout* parameter to the :func:`ssl.get_server_certificate` function. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The ssl module uses heap-types and multi-phase initialization. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | A new verify flag :data:`~ssl.VERIFY_X509_PARTIAL_CHAIN` has been added. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `sys `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :data:`sys.orig_argv` attribute: the list of the original command line | | + | arguments passed to the Python executable. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Add :data:`sys.stdlib_module_names`, containing the list of the standard library | | + | module names. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `_thread `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | :func:`_thread.interrupt_main` now takes an optional signal number to | | + | simulate (the default is still :data:`signal.SIGINT`). | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + +.. rubric:: Notes + +.. [#spm] The structural pattern matching feature is discussed in `issue #7847 `_. + diff --git a/docs/differences/python_35.rst b/docs/differences/python_35.rst new file mode 100644 index 0000000000000..0fdc6121a1dd0 --- /dev/null +++ b/docs/differences/python_35.rst @@ -0,0 +1,193 @@ +.. _python_35: + +Python 3.5 +========== + +Below is a list of finalised/accepted PEPs for Python 3.5 grouped into their impact to MicroPython. + +.. table:: + :widths: 30 50 20 + + +--------------------------------------------------------------------------------------------------------------+--------------------+ + | **Extensions to the syntax** | **Status** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 448 `_ | Additional unpacking generalizations | Partial | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 465 `_ | A new matrix multiplication operator | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 492 `_ | Coroutines with ``async`` and ``await`` syntax | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | **Extensions and changes to runtime** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 461 `_ | % formatting for binary strings | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 475 `_ | Retrying system calls that fail with ``EINTR`` | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 479 `_ | Change ``StopIteration`` handling inside generators | Complete | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | **Standard library changes** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 471 `_ | ``os.scandir()`` | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 485 `_ | ``math.isclose()``, a function for testing | Complete | + | | approximate equality | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | **Miscellaneous changes** | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 441 `_ | Improved Python zip application support | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 486 `_ | Make the Python Launcher aware of virtual | Not relevant | + | | environments | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 484 `_ | Type hints (advisory only) | Complete [#fth]_ | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 488 `_ | Elimination of PYO files | Not relevant | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + | `PEP 489 `_ | Redesigning extension module loading | | + +--------------------------------------------------------+-----------------------------------------------------+--------------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Added the *namereplace* error handlers. The *backslashreplace* error handlers now work with decoding and | | + | translating. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Property docstrings are now writable. This is especially useful for collections.namedtuple() docstrings | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Circular imports involving relative imports are now supported. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + + +New Modules: + +* `typing `_ + +* `zipzap `_ + + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `collections `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *OrderedDict* class is now implemented in C, which makes it 4 to 100 times faster. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | *OrderedDict.items()* , *OrderedDict.keys()* , *OrderedDict.values()* views now support reversed() | | + | iteration. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The deque class now defines *index()*, *insert()*, and *copy()*, and supports the + and * operators. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Docstrings produced by namedtuple() can now be updated. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The UserString class now implements the *__getnewargs__()*, *__rmod__()*, *casefold()*, *format_map()*, | | + | *isprintable()*, and *maketrans()* methods to match the corresponding methods of str. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `heapq `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Element comparison in *merge()* can now be customized by passing a key function in a new optional key | | + | keyword argument, and a new optional *reverse* keyword argument can be used to reverse element comparison | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `io `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *BufferedIOBase.readinto1()* method, that uses at most one call to the underlying raw stream's | | + | *RawIOBase.read()* or *RawIOBase.readinto()* methods | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `json `_ | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | JSON decoder now raises JSONDecodeError instead of ValueError to provide better context information about | | + | the error. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `math `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Two new constants have been added to the math module: *inf* and *nan*. | Complete | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new function *isclose()* provides a way to test for approximate equality. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *gcd()* function has been added. The *fractions.gcd()* function is now deprecated. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `os `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The new *scandir()* function returning an iterator of DirEntry objects has been added. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *urandom()* function now uses the *getrandom()* syscall on Linux 3.17 or newer, and *getentropy()* on | | + | OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file | | + | descriptor exhaustion. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | New *get_blocking()* and *set_blocking()* functions allow getting and setting a file descriptor's blocking| | + | mode (O_NONBLOCK.) | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | There is a new *os.path.commonpath()* function returning the longest common sub-path of each passed | | + | pathname | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `re `_ | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | References and conditional references to groups with fixed length are now allowed in lookbehind assertions| | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The number of capturing groups in regular expressions is no longer limited to 100. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *sub()* and *subn()* functions now replace unmatched groups with empty strings instead of raising an | | + | exception. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *re.error* exceptions have new attributes, msg, pattern, pos, lineno, and colno, that provide better | | + | context information about the error | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `socket `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Functions with timeouts now use a monotonic clock, instead of a system clock. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *socket.sendfile()* method allows sending a file over a socket by using the high-performance | | + | *os.sendfile()* function on UNIX, resulting in uploads being from 2 to 3 times faster than when using | | + | plain *socket.send()* | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *socket.sendall()* method no longer resets the socket timeout every time bytes are received or sent. | | + | The socket timeout is now the maximum total duration to send all data. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The backlog argument of the *socket.listen()* method is now optional. By default it is set to SOMAXCONN or| Complete | + | to 128, whichever is less. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `ssl `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Memory BIO Support | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | Application-Layer Protocol Negotiation Support | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | There is a new *SSLSocket.version()* method to query the actual protocol version in use. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The SSLSocket class now implements a *SSLSocket.sendfile()* method. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *SSLSocket.send()* method now raises either the *ssl.SSLWantReadError* or *ssl.SSLWantWriteError* | | + | exception on a non-blocking socket if the operation would block. Previously, it would return 0. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *cert_time_to_seconds()* function now interprets the input time as UTC and not as local time, per RFC | | + | 5280. Additionally, the return value is always an int. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | New *SSLObject.shared_ciphers()* and *SSLSocket.shared_ciphers()* methods return the list of ciphers sent | | + | by the client during the handshake. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *SSLSocket.do_handshake()*, *SSLSocket.read()*, *SSLSocket.shutdown()*, and *SSLSocket.write()* | | + | methods of the SSLSocket class no longer reset the socket timeout every time bytes are received or sent. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *match_hostname()* function now supports matching of IP addresses. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `sys `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *set_coroutine_wrapper()* function allows setting a global hook that will be called whenever a | | + | coroutine object is created by an async def function. A corresponding *get_coroutine_wrapper()* can be | | + | used to obtain a currently set wrapper. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | A new *is_finalizing()* function can be used to check if the Python interpreter is shutting down. | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | `time `_ | + +-----------------------------------------------------------------------------------------------------------+---------------+ + | The *monotonic()* function is now always available | | + +-----------------------------------------------------------------------------------------------------------+---------------+ + +.. rubric:: Notes + +.. [#fth] The MicroPython parser correct ignores all type hints. However, the ``typing`` module is not built-in. diff --git a/docs/differences/python_36.rst b/docs/differences/python_36.rst new file mode 100644 index 0000000000000..3315b0594dafc --- /dev/null +++ b/docs/differences/python_36.rst @@ -0,0 +1,200 @@ +.. _python_36: + +Python 3.6 +========== + +Python 3.6 beta 1 was released on 12 Sep 2016, and a summary of the new features can be found here: + +.. table:: + :widths: 30 50 20 + + +-----------------------------------------------------------------------------------------------------------+-----------------+ + | **New Syntax Features** | **Status** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 498 `_ | Literal String Formatting | Complete | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 515 `_ | Underscores in Numeric Literals | Complete | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 525 `_ | Asynchronous Generators | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 526 `_ | Syntax for Variable Annotations (provisional) | Complete | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 530 `_ | Asynchronous Comprehensions | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **New Built-in Features** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 468 `_ | Preserving the order of *kwargs* in a function | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 487 `_ | Simpler customization of class creation | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 520 `_ | Preserving Class Attribute Definition Order | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **Standard Library Changes** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 495 `_ | Local Time Disambiguation | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 506 `_ | Adding A Secrets Module To The Standard Library | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 519 `_ | Adding a file system path protocol | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **CPython Internals** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 509 `_ | Add a private version to dict | Won't do | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 523 `_ | Adding a frame evaluation API to CPython | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | **Linux/Window Changes** | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 524 `_ | Make ``os.urandom()`` blocking on Linux | | + | | (during system startup) | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 528 `_ | Change Windows console encoding to UTF-8 | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + | `PEP 529 `_ | Change Windows filesystem encoding to UTF-8 | | + +--------------------------------------------------------+--------------------------------------------------+-----------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-------------------------------------------------------------------------------------------------------------+---------------+ + | A *global* or *nonlocal* statement must now textually appear before the first use of the affected name in | | + | the same scope. Previously this was a SyntaxWarning. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | It is now possible to set a special method to None to indicate that the corresponding operation is not | | + | available. For example, if a class sets *__iter__()* to *None* , the class is not iterable. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Long sequences of repeated traceback lines are now abbreviated as *[Previous line repeated {count} more | | + | times]* | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Import now raises the new exception *ModuleNotFoundError* when it cannot find a module. Code that currently | | + | checks for ImportError (in try-except) will still work. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Class methods relying on zero-argument *super()* will now work correctly when called from metaclass methods | | + | during class creation. | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `array `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Exhausted iterators of *array.array* will now stay exhausted even if the iterated array is extended. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `binascii `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline | Complete | + | character is appended to the return value | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `cmath `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new cmath.tau (τ) constant has been added | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | New constants: *cmath.inf* and *cmath.nan* to match *math.inf* and *math.nan* , and also *cmath.infj* and | | + | *cmath.nanj* to match the format used by complex repr | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `collections `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new Collection abstract base class has been added to represent sized iterable container classes | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *Reversible* abstract base class represents iterable classes that also provide the *__reversed__()* | | + | method. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *AsyncGenerator* abstract base class represents asynchronous generators. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *namedtuple()* function now accepts an optional keyword argument module, which, when specified, is used | | + | for the *__module__* attribute of the returned named tuple class. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The verbose and rename arguments for *namedtuple()* are now keyword-only. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Recursive *collections.deque* instances can now be pickled. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `hashlib `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | BLAKE2 hash functions were added to the module. *blake2b()* and *blake2s()* are always available and support | | + | the full feature set of BLAKE2. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The SHA-3 hash functions *sha3_224()*, *sha3_256()*, *sha3_384()*, *sha3_512()*, and *SHAKE* hash functions | | + | *shake_128()* and *shake_256()* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The password-based key derivation function *scrypt()* is now available with OpenSSL 1.1.0 and newer. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `json `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | *json.load()* and *json.loads()* now support binary input. Encoded JSON should be represented using either | | + | UTF-8, UTF-16, or UTF-32. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `math `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new math.tau (τ) constant has been added | Complete | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `os `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | A new *close()* method allows explicitly closing a *scandir()* iterator. The *scandir()* iterator now | | + | supports the context manager protocol. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | On Linux, *os.urandom()* now blocks until the system urandom entropy pool is initialized to increase the | | + | security. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The Linux *getrandom()* syscall (get random bytes) is now exposed as the new *os.getrandom()* function. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `re `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Added support of modifier spans in regular expressions. Examples: *'(?i:p)ython'* matches 'python' and | | + | 'Python', but not 'PYTHON'; *'(?i)g(?-i:v)r'* matches *'GvR'* and *'gvr'*, but not *'GVR'*. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Match object groups can be accessed by *__getitem__*, which is equivalent to *group()*. So *mo['name']* is | | + | now equivalent to *mo.group('name')*. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Match objects now support index-like objects as group indices. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `socket `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *ioctl()* function now supports the *SIO_LOOPBACK_FAST_PATH* control code. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *getsockopt()* constants *SO_DOMAIN* , *SO_PROTOCOL*, *SO_PEERSEC* , and *SO_PASSSEC* are now supported. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *setsockopt()* now supports the *setsockopt(level, optname, None, optlen: int)* form. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The socket module now supports the address family *AF_ALG* to interface with Linux Kernel crypto API. | | + | *ALG_*, *SOL_ALG* and *sendmsg_afalg()* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | New Linux constants *TCP_USER_TIMEOUT* and *TCP_CONGESTION* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `ssl `_ | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | 3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | *SSLContext* has better default configuration for options and ciphers. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | SSL session can be copied from one client-side connection to another with the new *SSLSession* class. TLS | | + | session resumption can speed up the initial handshake, reduce latency and improve performance. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *get_ciphers()* method can be used to get a list of enabled ciphers in order of cipher priority. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | All constants and flags have been converted to *IntEnum* and *IntFlags*. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Server and client-side specific TLS protocols for *SSLContext* were added. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | Added *SSLContext.post_handshake_auth* to enable and *ssl.SSLSocket.verify_client_post_handshake()* to | | + | initiate TLS 1.3 post-handshake authentication. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `struct `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | now supports IEEE 754 half-precision floats via the 'e' format specifier. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `sys `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The new *getfilesystemencodeerrors()* function returns the name of the error mode used to convert between | | + | Unicode filenames and bytes filenames. | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | `zlib `_ | | + +--------------------------------------------------------------------------------------------------------------+----------------+ + | The *compress()* and *decompress()* functions now accept keyword arguments | | + +--------------------------------------------------------------------------------------------------------------+----------------+ diff --git a/docs/differences/python_37.rst b/docs/differences/python_37.rst new file mode 100644 index 0000000000000..2b9a089217f6d --- /dev/null +++ b/docs/differences/python_37.rst @@ -0,0 +1,108 @@ +.. _python_37: + +Python 3.7 +========== + +New Features: + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | **Feature** | **Status** | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 538 `_ | Coercing the legacy C locale to a UTF-8 based | | + | | locale | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 539 `_ | A New C-API for Thread-Local Storage in CPython | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 540 `_ | UTF-8 mode | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 552 `_ | Deterministic pyc | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 553 `_ | Built-in ``breakpoint()`` | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 557 `_ | Data Classes | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 560 `_ | Core support for typing module and generic types | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 562 `_ | Module ``__getattr__`` and ``__dir__`` | Partial | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 563 `_ | Postponed Evaluation of Annotations | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 564 `_ | Time functions with nanosecond resolution | Partial [#ftimenanosec]_ | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 565 `_ | Show DeprecationWarning in ``__main__`` | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + | `PEP 567 `_ | Context Variables | | + +--------------------------------------------------------+--------------------------------------------------+--------------------------------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``async`` and ``await`` are now reserved keywords | Complete | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``dict`` objects must preserve insertion-order | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | More than 255 arguments can now be passed to a function; a function can now have more than 255 parameters | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``bytes.fromhex()`` and ``bytearray.fromhex()`` now ignore all ASCII whitespace, not only spaces | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``str``, ``bytes``, and ``bytearray`` gained support for the new ``isascii()`` method, which can be used to | | + | test if a string or bytes contain only the ASCII characters | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``ImportError`` now displays module name and module ``__file__`` path when ``from ... import ...`` fails | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | Circular imports involving absolute imports with binding a submodule to a name are now supported | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than ``format(str(self), '')`` | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | In order to better support dynamic creation of stack traces, ``types.TracebackType`` can now be | | + | instantiated from Python code, and the ``tb_next`` attribute on tracebacks is now writable | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | When using the ``-m`` switch, ``sys.path[0]`` is now eagerly expanded to the full starting directory path, | | + | rather than being left as the empty directory (which allows imports from the current working directory | | + | at the time when an import occurs) | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + | The new ``-X importtime`` option or the ``PYTHONPROFILEIMPORTTIME`` environment variable can be used to | | + | show the timing of each module import | | + +-----------------------------------------------------------------------------------------------------------------+----------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +------------------------------------------------------------------------------------------------------------+----------------+ + | `asyncio `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | Too many to list | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `gc `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | New features include *gc.freeze()*, *gc.unfreeze()*, *gc-get_freeze_count* | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `math `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | math.remainder() added to implement IEEE 754-style remainder | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `re `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | A number of tidy up features including better support for splitting on empty strings and copy support for | | + | compiled expressions and match objects | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `sys `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | sys.breakpointhook() added. sys.get(/set)_coroutine_origin_tracking_depth() added | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | `time `_ | | + +------------------------------------------------------------------------------------------------------------+----------------+ + | Mostly updates to support nanosecond resolution in PEP564, see above | | + +------------------------------------------------------------------------------------------------------------+----------------+ + +.. rubric:: Notes + +.. [#ftimenanosec] Only :func:`time.time_ns` is implemented. diff --git a/docs/differences/python_38.rst b/docs/differences/python_38.rst new file mode 100644 index 0000000000000..32fd68d2caeac --- /dev/null +++ b/docs/differences/python_38.rst @@ -0,0 +1,127 @@ +.. _python_38: + +Python 3.8 +========== + +Python 3.8.0 (final) was released on the 14 October 2019. The Features for 3.8 +are defined in `PEP 569 `_ and +a detailed description of the changes can be found in `What's New in Python +3.8. `_ + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | **Features** | **Status** | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 570 `_ | Positional-only arguments | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 572 `_ | Assignment Expressions | Complete | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 574 `_ | Pickle protocol 5 with out-of-band data | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 578 `_ | Runtime audit hooks | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 587 `_ | Python Initialization Configuration | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | `PEP 590 `_ | Vectorcall: a fast calling protocol for CPython | | + +--------------------------------------------------------+---------------------------------------------------+---------------+ + | **Miscellaneous** | + +------------------------------------------------------------------------------------------------------------+---------------+ + | f-strings support = for self-documenting expressions and debugging | Complete | + +------------------------------------------------------------------------------------------------------------+---------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +------------------------------------------------------------------------------------------------------------+-------------+ + | A *continue* statement was illegal in the *finally* clause due to a problem with the implementation. In | Complete | + | Python 3.8 this restriction was lifted | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The *bool*, *int* , and *fractions.Fraction* types now have an *as_integer_ratio()* method like that found | | + | in *float* and *decimal.Decimal* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Constructors of *int*, *float* and *complex* will now use the *__index__()* special method, if available | | + | and the corresponding method *__int__()*, *__float__()* or *__complex__()* is not available | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added support of *\N{name}* escapes in regular expressions | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Dict and dictviews are now iterable in reversed insertion order using *reversed()* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The syntax allowed for keyword names in function calls was further restricted. In particular, | | + | f((keyword)=arg) is no longer allowed | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Generalized iterable unpacking in yield and return statements no longer requires enclosing parentheses | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | When a comma is missed in code such as [(10, 20) (30, 40)], the compiler displays a SyntaxWarning with a | | + | helpful suggestion | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Arithmetic operations between subclasses of *datetime.date* or *datetime.datetime* and *datetime.timedelta*| | + | objects now return an instance of the subclass, rather than the base class | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | When the Python interpreter is interrupted by *Ctrl-C (SIGINT)* and the resulting *KeyboardInterrupt* | | + | exception is not caught, the Python process now exits via a SIGINT signal or with the correct exit code | | + | such that the calling process can detect that it died due to a *Ctrl-C* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Some advanced styles of programming require updating the *types.CodeType* object for an existing function | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | For integers, the three-argument form of the pow() function now permits the exponent to be negative in the | | + | case where the base is relatively prime to the modulus | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Dict comprehensions have been synced-up with dict literals so that the key is computed first and the value | | + | second | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The *object.__reduce__()* method can now return a tuple from two to six elements long | | + +------------------------------------------------------------------------------------------------------------+-------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +------------------------------------------------------------------------------------------------------------+-------------+ + | `asyncio `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | *asyncio.run()* has graduated from the provisional to stable API | Complete | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Running *python -m asyncio* launches a natively async REPL | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The exception *asyncio.CancelledError* now inherits from *BaseException* rather than *Exception* and no | Complete | + | longer inherits from *concurrent.futures.CancelledError* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added *asyncio.Task.get_coro()* for getting the wrapped coroutine within an *asyncio.Task* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Asyncio tasks can now be named, either by passing the name keyword argument to *asyncio.create_task()* or | | + | the *create_task()* event loop method, or by calling the *set_name()* method on the task object | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added support for Happy Eyeballs to *asyncio.loop.create_connection()*. To specify the behavior, two new | | + | parameters have been added: *happy_eyeballs_delay* and interleave. | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | `gc `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | *get_objects()* can now receive an optional generation parameter indicating a generation to get objects | | + | from. (Note, though, that while *gc* is a built-in, *get_objects()* is not implemented for MicroPython) | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | `math `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added new function *math.dist()* for computing Euclidean distance between two points | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Expanded the *math.hypot()* function to handle multiple dimensions | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added new function, *math.prod()*, as analogous function to *sum()* that returns the product of a "start" | | + | value (default: 1) times an iterable of numbers | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added two new combinatoric functions *math.perm()* and *math.comb()* | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Added a new function *math.isqrt()* for computing accurate integer square roots without conversion to | | + | floating point | | + +------------------------------------------------------------------------------------------------------------+-------------+ + | The function *math.factorial()* no longer accepts arguments that are not int-like | Complete | + +------------------------------------------------------------------------------------------------------------+-------------+ + | `sys `_ | + +------------------------------------------------------------------------------------------------------------+-------------+ + | Add new *sys.unraisablehook()* function which can be overridden to control how "unraisable exceptions" | | + | are handled | | + +------------------------------------------------------------------------------------------------------------+-------------+ diff --git a/docs/differences/python_39.rst b/docs/differences/python_39.rst new file mode 100644 index 0000000000000..f68abc50740ac --- /dev/null +++ b/docs/differences/python_39.rst @@ -0,0 +1,134 @@ +.. _python_39: + +Python 3.9 +========== + +Python 3.9.0 (final) was released on the 5th October 2020. The Features for 3.9 are +defined in `PEP 596 `_ +and a detailed description of the changes can be found in +`What's New in Python 3.9 `_ + +.. table:: + :widths: 20 60 20 + + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | **Features** | | **Status** | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 573 `_ | Fast access to module state from methods of C | Not relevant | + | | extension types | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 584 `_ | Union operators added to dict | Complete [#pep584]_ | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 585 `_ | Type hinting generics in standard collections | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 593 `_ | Flexible function and variable annotations | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 602 `_ | CPython adopts an annual release cycle. Instead of | Not relevant | + | | annual, aiming for two month release cycle | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 614 `_ | Relaxed grammar restrictions on decorators | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 615 `_ | The IANA Time Zone Database is now present in the | | + | | standard library in the zoneinfo module | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 616 `_ | String methods to remove prefixes and suffixes | | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + | `PEP 617 `_ | CPython now uses a new parser based on PEG | Not relevant | + +--------------------------------------------------------+----------------------------------------------------+----------------------+ + +Other Language Changes: + +.. table:: + :widths: 90 10 + + +-------------------------------------------------------------------------------------------------------------+---------------+ + | *__import__()* now raises *ImportError* instead of *ValueError* | Complete | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Python now gets the absolute path of the script filename specified on the command line (ex: *python3* | | + | *script.py*): the *__file__* attribute of the *__main__* module became an absolute path, rather than a | | + | relative path | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | By default, for best performance, the errors argument is only checked at the first encoding/decoding error | | + | and the encoding argument is sometimes ignored for empty strings | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | *"".replace("", s, n)* now returns *s* instead of an empty string for all non-zero n. It is now consistent | | + | with *"".replace("", s)* | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Any valid expression can now be used as a decorator. Previously, the grammar was much more restrictive | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Parallel running of *aclose()* / *asend()* / *athrow()* is now prohibited, and *ag_running* now reflects | | + | the actual running status of the async generator | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Unexpected errors in calling the *__iter__* method are no longer masked by TypeError in the in operator and | | + | functions contains(), indexOf() and countOf() of the operator module | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + | Unparenthesized lambda expressions can no longer be the expression part in an if clause in comprehensions | | + | and generator expressions | | + +-------------------------------------------------------------------------------------------------------------+---------------+ + +Changes to built-in modules: + +.. table:: + :widths: 90 10 + + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `asyncio `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Due to significant security concerns, the reuse_address parameter of *asyncio.loop.create_datagram_endpoint()*| | + | is no longer supported | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new coroutine *shutdown_default_executor()* that schedules a shutdown for the default executor that | | + | waits on the *ThreadPoolExecutor* to finish closing. Also, *asyncio.run()* has been updated to use the new | | + | coroutine. | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *asyncio.PidfdChildWatcher*, a Linux-specific child watcher implementation that polls process file | | + | descriptors | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | added a new *coroutine asyncio.to_thread()* | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | When cancelling the task due to a timeout, *asyncio.wait_for()* will now wait until the cancellation is | | + | complete also in the case when timeout is <= 0, like it does with positive timeouts | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | *asyncio* now raises *TyperError* when calling incompatible methods with an *ssl.SSLSocket* socket | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `gc `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Garbage collection does not block on resurrected objects | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new function *gc.is_finalized()* to check if an object has been finalized by the garbage collector | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `math `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Expanded the *math.gcd()* function to handle multiple arguments. Formerly, it only supported two arguments | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *math.lcm()*: return the least common multiple of specified arguments | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *math.nextafter()*: return the next floating-point value after x towards y | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *math.ulp()*: return the value of the least significant bit of a float | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `os `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Exposed the Linux-specific *os.pidfd_open()* and *os.P_PIDFD* | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The *os.unsetenv()* function is now also available on Windows | Complete | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | The *os.putenv()* and *os.unsetenv()* functions are now always available | Complete | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added *os.waitstatus_to_exitcode()* function: convert a wait status to an exit code | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `random `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new *random.Random.randbytes* method: generate random bytes | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | `sys `_ | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Added a new *sys.platlibdir* attribute: name of the platform-specific library directory | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + | Previously, *sys.stderr* was block-buffered when non-interactive. Now stderr defaults to always being | | + | line-buffered | | + +---------------------------------------------------------------------------------------------------------------+---------------+ + +.. rubric:: Notes + +.. [#pep584] PEP 584 ``dict`` union operator is only available on MicroPython builds with ``MICROPY_CPYTHON_COMPAT`` enabled. diff --git a/docs/esp32/quickref.rst b/docs/esp32/quickref.rst index 1709891265d8e..ccc01099d17f9 100644 --- a/docs/esp32/quickref.rst +++ b/docs/esp32/quickref.rst @@ -16,7 +16,10 @@ working with this board it may be useful to get an overview of the microcontroll :maxdepth: 1 general.rst - tutorial/intro.rst + tutorial/index.rst + +Note that there are several varieties of ESP32 -- ESP32, ESP32C3, ESP32C6, ESP32S2, ESP32S3 -- +supported by MicroPython, with some differences in functionality between them. Installing MicroPython ---------------------- @@ -57,55 +60,204 @@ The :mod:`esp32` module:: import esp32 - esp32.hall_sensor() # read the internal hall sensor esp32.raw_temperature() # read the internal temperature of the MCU, in Fahrenheit - esp32.ULP() # access to the Ultra-Low-Power Co-processor + esp32.ULP() # access to the Ultra-Low-Power Co-processor, not on ESP32C3/C6 Note that the temperature sensor in the ESP32 will typically read higher than ambient due to the IC getting warm while it runs. This effect can be minimised by reading the temperature sensor immediately after waking up from sleep. +ESP32C3, ESP32C6, ESP32S2, and ESP32S3 also have an internal temperature sensor available. +It is implemented a bit differently to the ESP32 and returns the temperature in +Celsius:: + + esp32.mcu_temperature() # read the internal temperature of the MCU, in Celsius + Networking ---------- -The :mod:`network` module:: +WLAN +^^^^ + +The :class:`network.WLAN` class in the :mod:`network` module:: import network - wlan = network.WLAN(network.STA_IF) # create station interface - wlan.active(True) # activate the interface - wlan.scan() # scan for access points - wlan.isconnected() # check if the station is connected to an AP - wlan.connect('essid', 'password') # connect to an AP - wlan.config('mac') # get the interface's MAC address - wlan.ifconfig() # get the interface's IP/netmask/gw/DNS addresses + wlan = network.WLAN() # create station interface (the default, see below for an access point interface) + wlan.active(True) # activate the interface + wlan.scan() # scan for access points + wlan.isconnected() # check if the station is connected to an AP + wlan.connect('ssid', 'key') # connect to an AP + wlan.config('mac') # get the interface's MAC address + wlan.ipconfig('addr4') # get the interface's IPv4 addresses - ap = network.WLAN(network.AP_IF) # create access-point interface - ap.config(essid='ESP-AP') # set the ESSID of the access point - ap.config(max_clients=10) # set how many clients can connect to the network - ap.active(True) # activate the interface + ap = network.WLAN(network.WLAN.IF_AP) # create access-point interface + ap.config(ssid='ESP-AP') # set the SSID of the access point + ap.config(max_clients=10) # set how many clients can connect to the network + ap.active(True) # activate the interface A useful function for connecting to your local WiFi network is:: def do_connect(): - import network - wlan = network.WLAN(network.STA_IF) + import machine, network + wlan = network.WLAN() wlan.active(True) if not wlan.isconnected(): print('connecting to network...') - wlan.connect('essid', 'password') + wlan.connect('ssid', 'key') while not wlan.isconnected(): - pass - print('network config:', wlan.ifconfig()) + machine.idle() + print('network config:', wlan.ipconfig('addr4')) -Once the network is established the :mod:`socket ` module can be used -to create and use TCP/UDP sockets as usual, and the ``urequests`` module for +Once the network is established the :mod:`socket ` module can be used +to create and use TCP/UDP sockets as usual, and the ``requests`` module for convenient HTTP requests. +After a call to ``wlan.connect()``, the device will by default retry to connect +**forever**, even when the authentication failed or no AP is in range. +``wlan.status()`` will return ``network.STAT_CONNECTING`` in this state until a +connection succeeds or the interface gets disabled. This can be changed by +calling ``wlan.config(reconnects=n)``, where n are the number of desired reconnect +attempts (0 means it won't retry, -1 will restore the default behaviour of trying +to reconnect forever). + +.. _esp32_network_lan: + +LAN +^^^ + +Built-in MAC (original ESP32) +""""""""""""""""""""""""""""" + +The original ESP32 SoC has a built-in Ethernet MAC. Using this MAC requires an +external Ethernet PHY to be wired to the chip's EMAC pins. Most of the EMAC pin +assignments are fixed, consult the ESP32 datasheet for details. + +If the PHY is connected, the internal Ethernet MAC can be configured via +the :class:`network.LAN` constructor:: + + import network + + lan = network.LAN(mdc=PIN_MDC, ...) # Set the pin and mode configuration + lan.active(True) # activate the interface + lan.ipconfig('addr4') # get the interface's IPv4 addresses + + +Required keyword arguments for the constructor: + +- ``mdc`` and ``mdio`` - :class:`machine.Pin` objects (or integers) specifying + the MDC and MDIO pins. +- ``phy_type`` - Select the PHY device type. Supported devices are + ``PHY_LAN8710``, ``PHY_LAN8720``, ``PHY_IP101``, ``PHY_RTL8201``, + ``PHY_DP83848``, ``PHY_KSZ8041`` and ``PHY_KSZ8081``. These values are all + constants defined in the ``network`` module. +- ``phy_addr`` - The address number of the PHY device. Must be an integer in the + range 0x00 to 0x1f, inclusive. Common values are ``0`` and ``1``. + +All of the above keyword arguments must be present to configure the interface. + +Optional keyword arguments: + +- ``reset`` - :class:`machine.Pin` object (or integer) specifying the PHY reset pin. +- ``power`` - :class:`machine.Pin` object (or integer) specifying a pin which + switches the power of the PHY device. +- ``ref_clk`` - :class:`machine.Pin` object (or integer) specifying the pin used + for the EMAC ``ref_clk`` signal. If not specified, the board default is used + (typically GPIO 0, but may be different if a particular board has Ethernet.) +- ``ref_clk_mode`` - Defines whether the EMAC ``ref_clk`` pin of the ESP32 + should be an input or an output. Suitable values are ``machine.Pin.IN`` and + ``machine.Pin.OUT``. If not specified, the board default is used + (typically input, but may be different if a particular board has Ethernet.) + +These are working configurations for LAN interfaces of some popular ESP32 boards:: + + # Olimex ESP32-GATEWAY: power controlled by Pin(5) + # Olimex ESP32 PoE and ESP32-PoE ISO: power controlled by Pin(12) + + lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), power=machine.Pin(5), + phy_type=network.PHY_LAN8720, phy_addr=0, + ref_clk=machine.Pin(17), ref_clk_mode=machine.Pin.OUT) + + # Wireless-Tag's WT32-ETH01 + + lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), + phy_type=network.PHY_LAN8720, phy_addr=1, power=None) + + # Wireless-Tag's WT32-ETH01 v1.4 + + lan = network.LAN(mdc=machine.Pin(23), mdio=machine.Pin(18), + phy_type=network.PHY_LAN8720, phy_addr=1, + power=machine.Pin(16)) + + # Espressif ESP32-Ethernet-Kit_A_V1.2 + + lan = network.LAN(id=0, mdc=Pin(23), mdio=Pin(18), power=Pin(5), + phy_type=network.PHY_IP101, phy_addr=1) + + +.. _esp32_spi_ethernet: + +SPI Ethernet Interface +"""""""""""""""""""""" + +All ESP32 SoCs support external SPI Ethernet interface chips. These are Ethernet +interfaces that connect via a SPI bus, rather than an Ethernet RMII interface. + +.. note:: The only exception is the ESP32 ``d2wd`` variant, where this feature is disabled + to save code size. + +SPI Ethernet uses the same :class:`network.LAN` constructor, with a different +set of keyword arguments:: + + import machine, network + + spi = machine.SPI(1, sck=SCK_PIN, mosi=MOSI_PIN, miso=MISO_PIN) + lan = network.LAN(spi=spi, cs=CS_PIN, ...) # Set the pin and mode configuration + lan.active(True) # activate the interface + lan.ipconfig('addr4') # get the interface's IPv4 addresses + +Required keyword arguments for the constructor: + +- ``spi`` - Should be a :class:`machine.SPI` object configured for this + connection. Note that any clock speed configured on the SPI object is ignored, + the SPI Ethernet clock speed is configured at compile time. +- ``cs`` - :class:`machine.Pin` object (or integer) specifying the CS pin + connected to the interface. +- ``int`` - :class:`machine.Pin` object (or integer) specifying the INT pin + connected to the interface. +- ``phy_type`` - Select the SPI Ethernet interface type. Supported devices are + ``PHY_KSZ8851SNL``, ``PHY_DM9051``, ``PHY_W5500``. These values are all + constants defined in the ``network`` module. +- ``phy_addr`` - The address number of the PHY device. Must be an integer in the + range 0x00 to 0x1f, inclusive. This is usually ``0`` for SPI Ethernet devices. + +All of the above keyword arguments must be present to configure the interface. + +Optional keyword arguments for the constructor: + +- ``reset`` - :class:`machine.Pin` object (or integer) specifying the SPI Ethernet + interface reset pin. +- ``power`` - :class:`machine.Pin` object (or integer) specifying a pin which + switches the power of the SPI Ethernet interface. + +Here is a sample configuration for a WIZNet W5500 chip connected to pins on +an ESP32-S3 development board:: + + import machine, network + from machine import Pin, SPI + + spi = SPI(1, sck=Pin(12), mosi=Pin(13), miso=Pin(14)) + lan = network.LAN(spi=spi, phy_type=network.PHY_W5500, phy_addr=0, + cs=Pin(10), int=Pin(11)) + +.. note:: WIZnet W5500 Ethernet is also supported on some other MicroPython + ports, but using a :ref:`different software interface + `. + Delay and timing ---------------- -Use the :mod:`time ` module:: +Use the :mod:`time