Fix duplicate changelog entries (#19759)

Co-authored-by: Tzu-ping Chung <[email protected]>

.pre-commit-config.yaml

@@ -690,6 +690,12 @@ repos:
         files: ^airflow/www/static/js/
         entry: scripts/ci/static_checks/www_lint.sh
         pass_filenames: false
+      - id: changelog-duplicates
+        name: Check changelogs for duplicate entries
+        language: python
+        files: CHANGELOG\.txt$|CHANGELOG\.rst$
+        entry: scripts/ci/pre_commit/pre_commit_changelog_duplicates.py
+        pass_filenames: true
         ## ADD MOST PRE-COMMITS ABOVE THAT LINE
         # The below pre-commits are those requiring CI image to be built
       - id: build

BREEZE.rst

@@ -2195,21 +2195,21 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
                  all airflow-config-yaml airflow-providers-available airflow-provider-yaml-files-ok
                  base-operator bats-tests bats-in-container-tests black blacken-docs boring-cyborg
-                 build build-providers-dependencies check-apache-license check-builtin-literals
-                 check-executables-have-shebangs check-extras-order check-hooks-apply
-                 check-integrations check-merge-conflict check-xml daysago-import-check
-                 debug-statements detect-private-key doctoc dont-use-safe-filter end-of-file-fixer
-                 fix-encoding-pragma flake8 flynt codespell forbid-tabs helm-lint identity
-                 incorrect-use-of-LoggingMixin insert-license isort json-schema language-matters
-                 lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending mypy mypy-helm
-                 no-providers-in-core-examples no-relative-imports pre-commit-descriptions
-                 pre-commit-hook-names pretty-format-json provide-create-sessions
-                 providers-changelogs providers-init-file providers-subpackages-init-file
-                 provider-yamls pydevd pydocstyle python-no-log-warn pyupgrade restrict-start_date
-                 rst-backticks setup-order setup-extra-packages shellcheck sort-in-the-wild
-                 sort-spelling-wordlist stylelint trailing-whitespace ui-lint update-breeze-file
-                 update-extras update-local-yml-file update-setup-cfg-file update-versions
-                 verify-db-migrations-documented version-sync www-lint yamllint yesqa
+                 build build-providers-dependencies changelog-duplicates check-apache-license
+                 check-builtin-literals check-executables-have-shebangs check-extras-order
+                 check-hooks-apply check-integrations check-merge-conflict check-xml
+                 daysago-import-check debug-statements detect-private-key doctoc dont-use-safe-filter
+                 end-of-file-fixer fix-encoding-pragma flake8 flynt codespell forbid-tabs helm-lint
+                 identity incorrect-use-of-LoggingMixin insert-license isort json-schema
+                 language-matters lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending
+                 mypy mypy-helm no-providers-in-core-examples no-relative-imports
+                 pre-commit-descriptions pre-commit-hook-names pretty-format-json
+                 provide-create-sessions providers-changelogs providers-init-file
+                 providers-subpackages-init-file provider-yamls pydevd pydocstyle python-no-log-warn
+                 pyupgrade restrict-start_date rst-backticks setup-order setup-extra-packages
+                 shellcheck sort-in-the-wild sort-spelling-wordlist stylelint trailing-whitespace
+                 ui-lint update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
+                 update-versions verify-db-migrations-documented version-sync www-lint yamllint yesqa
 
         You can pass extra arguments including options to the pre-commit framework as
         <EXTRA_ARGS> passed after --. For example:

CHANGELOG.txt

@@ -134,7 +134,6 @@ Improvements
 - Add note about params on trigger DAG page (#18166)
 - Change ``TaskInstance`` and ``TaskReschedule`` PK from ``execution_date`` to ``run_id`` (#17719)
 - Adding ``TaskGroup`` support in ``BaseOperator.chain()`` (#17456)
-- Advises the kernel to not cache log files generated by Airflow (#18054)
 - Allow filtering DAGS by tags in the REST API (#18090)
 - Optimize imports of Providers Manager (#18052)
 - Adds capability of Warnings for incompatible community providers (#18020)
@@ -143,8 +142,6 @@ Improvements
 - Refactor ``BranchDayOfWeekOperator``, ``DayOfWeekSensor`` (#17940)
 - Update error message to guide the user into self-help mostly (#17929)
 - Update to Celery 5 (#17397)
-- Avoid redirect loop for users with no permissions (#17838)
-- Improve graph view load time for dags with open groups (#17821)
 - Add links to provider's documentation (#17736)
 - Remove Marshmallow schema warnings (#17753)
 - Rename ``none_failed_or_skipped`` by ``none_failed_min_one_success`` trigger rule (#17683)
@@ -153,7 +150,6 @@ Improvements
 - Import Hooks lazily individually in providers manager (#17682)
 - Adding support for multiple task-ids in the external task sensor (#17339)
 - Replace ``execution_date`` with ``run_id`` in airflow tasks run command (#16666)
-- Avoid endless redirect loop when user has no roles (#17613)
 - Make output from users cli command more consistent (#17642)
 - Open relative extra links in place (#17477)
 - Move ``worker_log_server_port`` option to the logging section (#17621)
@@ -185,7 +181,6 @@ Improvements
 - When a task instance fails with exception, log it (#16805)
 - Set process title for ``serve-logs`` and ``LocalExecutor`` (#16644)
 - Rename ``test_cycle`` to ``check_cycle`` (#16617)
-- Set process title for Worker when using ``LocalExecutor`` (#16623)
 - Add schema as ``DbApiHook`` instance attribute (#16521, #17423)
 - Improve compatibility with MSSQL (#9973)
 - Add transparency for unsupported connection type (#16220)
@@ -234,41 +229,32 @@ Bug Fixes
 - Use try/except when closing temporary file in task_runner (#18269)
 - show next run if not none (#18273)
 - Fix DB session handling in ``XCom.set`` (#18240)
-- Sort adopted tasks in ``_check_for_stalled_adopted_tasks`` method (#18208)
 - Fix external_executor_id not being set for manually run jobs (#17207)
 - Fix deleting of zipped Dags in Serialized Dag Table (#18243)
 - Return explicit error on user-add for duplicated email (#18224)
 - Remove loading dots even when last run data is empty (#18230)
 - Swap dag import error dropdown icons (#18207)
 - Automatically create section when migrating config (#16814)
 - Set encoding to utf-8 by default while reading task logs (#17965)
-- Set task state to failed when pod is DELETED while running (#18095)
 - Apply parent dag permissions to subdags (#18160)
 - Change id collation for MySQL to case-sensitive (#18072)
 - Logs task launch exception in ``StandardTaskRunner`` (#17967)
 - Applied permissions to ``self._error_file`` (#15947)
-- Hide variable import form if user lacks permission (#18000)
 - Fix blank dag dependencies view (#17990)
 - Add missing menu access for dag dependencies and configurations pages (#17450)
 - Fix passing Jinja templates in ``DateTimeSensor`` (#17959)
 - Fixing bug which restricted the visibility of ImportErrors (#17924)
 - Fix grammar in ``traceback.html`` (#17942)
 - Fix ``DagRunState`` enum query for ``MySQLdb`` driver (#17886)
 - Fixed button size in "Actions" group. (#17902)
-- Fix log links on graph TI modal (#17862)
 - Only show import errors for DAGs a user can access (#17835)
 - Show all import_errors from zip files (#17759)
 - fix EXTRA_LOGGER_NAMES param and related docs (#17808)
 - Use one interpreter for Airflow and gunicorn (#17805)
 - Fix: Mysql 5.7 id utf8mb3 (#14535)
 - Fix dag_processing.last_duration metric random holes (#17769)
 - Automatically use ``utf8mb3_general_ci`` collation for MySQL (#17729)
-- Fix wrong query on running tis (#17631)
-- Add root to tree refresh url (#17633)
-- Do not delete running DAG from the UI (#17630)
-- Rescue if a DagRun's DAG was removed from db (#17544)
 - fix: filter condition of ``TaskInstance`` does not work #17535 (#17548)
-- Regression on pid reset to allow task start after heartbeat (#17333)
 - Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)
 - Remove locks for upgrades in MSSQL (#17213)
 - Create virtualenv via python call (#17156)
@@ -279,8 +265,6 @@ Bug Fixes
 - ensure task is skipped if missing sla (#16719)
 - Fix direct use of ``cached_property`` module (#16710)
 - Fix TI success confirm page (#16650)
-- Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
-- Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
 - Modify return value check in python virtualenv jinja template (#16049)
 - Fix dag dependency search (#15924)
 - Make custom JSON encoder support ``Decimal`` (#16383)
@@ -306,12 +290,9 @@ Doc only changes
 - Add ``sla_miss_callback`` section to the documentation (#18305)
 - Explain sentry default environment variable for subprocess hook (#18346)
 - Refactor installation pages (#18282)
-- Improves installing from sources pages for all components (#18251)
-- Doc: Use ``closer.lua`` script for downloading sources (#18179)
 - Improves quick-start docker-compose warnings and documentation (#18164)
 - Production-level support for MSSQL (#18382)
 - Update non-working example in documentation (#18067)
-- Improve the description of how to handle dynamic task generation (#17963)
 - Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)
 - Update max_tis_per_query to better render on the webpage (#17971)
 - Adds Github Oauth example with team based authorization (#17896)
@@ -328,7 +309,6 @@ Doc only changes
 - docs(dagowner): describe dag owner more carefully (#17699)
 - docs(impersonation): update note so avoid misintrepretation (#17701)
 - Docs: Make ``DAG.is_active`` read-only in API (#17667)
-- Improve cross-links to operators and hooks references (#17622)
 - Update documentation regarding Python 3.9 support (#17611)
 - Fix MySQL database character set instruction (#17603)
 - Document overriding ``XCom.clear`` for data lifecycle management (#17589)
@@ -352,7 +332,6 @@ Doc only changes
 - Small improvements for README.md files (#16244)
 - Fix docs for ``dag_concurrency`` (#16177)
 - Check syntactic correctness for code-snippets (#16005)
-- Improvements for Docker Image docs (#14843)
 - Add proper link for wheel packages in docs. (#15999)
 - Add Docs for ``default_pool`` slots (#15997)
 - Add memory usage warning in quick-start documentation (#15967)
@@ -378,10 +357,8 @@ Misc/Internal
 - Silence warnings in tests from using SubDagOperator (#18275)
 - Fix usage of ``range(len())`` to ``enumerate`` (#18174)
 - Test coverage on the autocomplete view (#15943)
-- Update version added fields in airflow/config_templates/config.yml (#18128)
 - Add "packaging" to core requirements (#18122)
 - Adds LoggingMixins to BaseTrigger (#18106)
-- Limit colorlog version (6.x is incompatible) (#18099)
 - Fix building docs in ``main`` builds (#18035)
 - Remove upper-limit on ``tenacity`` (#17593)
 - Remove  redundant ``numpy`` dependency (#17594)
@@ -403,7 +380,6 @@ Misc/Internal
 - Use ``DAG_ACTIONS`` constant (#16232)
 - Use updated ``_get_all_non_dag_permissions`` method (#16317)
 - Add updated-name wrappers for built-in FAB methods (#16077)
-- Pins ``docutils`` to <0.17 until breaking behaviour is fixed (#16133)
 - Remove ``TaskInstance.log_filepath`` attribute (#15217)
 - Removes unnecessary function call in ``airflow/www/app.py`` (#15956)
 - Move ``plyvel`` to google provider extra (#15812)
@@ -440,7 +416,7 @@ Bug Fixes
 - Do not delete running DAG from the UI (#17630)
 - Improve discoverability of Provider packages' functionality
 - Do not let ``create_dagrun`` overwrite explicit ``run_id`` (#17728)
-- BugFix: Regression on pid reset to allow task start after heartbeat (#17333)
+- Regression on pid reset to allow task start after heartbeat (#17333)
 - Set task state to failed when pod is DELETED while running (#18095)
 - Advises the kernel to not cache log files generated by Airflow (#18054)
 - Sort adopted tasks in ``_check_for_stalled_adopted_tasks`` method (#18208)
@@ -455,7 +431,6 @@ Doc only changes
 - Make the providers operators/hooks reference much more usable (#17768)
 - Update description about the new ``connection-types`` provider meta-data
 - Suggest to use secrets backend for variable when it contains sensitive data (#17319)
-- Improves quick-start docker-compose warnings and documentation (#18164)
 - Separate Installing from sources section and add more details (#18171)
 - Doc: Use ``closer.lua`` script for downloading sources (#18179)
 - Doc: Improve installing from sources (#18194)
@@ -1055,7 +1030,6 @@ Bug Fixes
 - Moved boto3 limitation to snowflake (#13286)
 - ``KubernetesExecutor`` should accept images from ``executor_config`` (#13074)
 - Scheduler should acknowledge active runs properly (#13803)
-- Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
 - Include ``airflow/contrib/executors`` in the dist package
 - Pin Click version for Python 2.7 users
 - Ensure all statsd timers use millisecond values. (#10633)
@@ -1303,7 +1277,6 @@ Doc only changes
 - Add Qingping Hou to committers list (#9725)
 - Updated link to official documentation (#9629)
 - Create a short-link for Airflow Slack Invites (#10034)
-- Fix docstrings in BigQueryGetDataOperator (#10042)
 - Set language on code-block on docs/howto/email-config.rst (#10238)
 - Remove duplicate line from 1.10.10 CHANGELOG (#10289)
 - Improve heading on Email Configuration page (#10175)
@@ -1957,7 +1930,6 @@ Improvements
 - [AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735)
 - [AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716)
 - [AIRFLOW-6144] Improve the log message of Airflow scheduler (#6710)
-- [AIRFLOW-6099] Add host name to task runner log (#6688)
 - [AIRFLOW-6045] Error on failed execution of compile_assets (#6640)
 - [AIRFLOW-5144] Add confirmation on delete button click (#6745)
 - [AIRFLOW-6099] Add host name to task runner log (#6688)
@@ -2287,7 +2259,6 @@ Misc/Internal
 - [AIRFLOW-5301] Some not-yet-available files from breeze are committed to master (#5901)
 - [AIRFLOW-5285] Pre-commit pylint runs over todo files (#5884)
 - [AIRFLOW-5288] Temporary container for static checks should be auto-removed (#5887)
-- [AIRFLOW-5326] Fix teething problems for Airflow breeze (#5933)
 - [AIRFLOW-5206] All .md files should have all common licence, TOC (where applicable) (#5809)
 - [AIRFLOW-5329] Easy way to add local files to docker (#5933)
 - [AIRFLOW-4027] Make experimental api tests more stateless (#4854)
@@ -2755,8 +2726,7 @@ Improvement
 - [AIRFLOW-3591] Fix start date, end date, duration for rescheduled tasks (#4502)
 - [AIRFLOW-3709] Validate `allowed_states` for ExternalTaskSensor (#4536)
 - [AIRFLOW-3522] Add support for sending Slack attachments (#4332)
-- [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www only) (#4373)
-- [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www_rbac only) (#4373)
+- [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (#4373)
 - [AIRFLOW-3044] Dataflow operators accept templated job_name param (#3887)
 - [AIRFLOW-3023] Fix docstring datatypes
 - [AIRFLOW-2928] Use uuid4 instead of uuid1 (#3779)
@@ -2869,7 +2839,6 @@ Bug fixes
 - [AIRFLOW-3923] Update flask-admin dependency to 1.5.3 to resolve security vulnerabilities from safety (#4739)
 - [AIRFLOW-3683] Fix formatting of error message for invalid TriggerRule (#4490)
 - [AIRFLOW-2787] Allow is_backfill to handle NULL DagRun.run_id (#3629)
-- [AIRFLOW-3780] Fix some incorrect when base_url is used (#4643)
 - [AIRFLOW-3639] Fix request creation in Jenkins Operator (#4450)
 - [AIRFLOW-3779] Don't install enum34 backport when not needed (#4620)
 - [AIRFLOW-3079] Improve migration scripts to support MSSQL Server (#3964)
@@ -2946,7 +2915,6 @@ Doc-only changes
 - [AIRFLOW-XXX] Fix typo (#4564)
 - [AIRFLOW-XXX] Add a doc about fab security (#4595)
 - [AIRFLOW-XXX] Speed up DagBagTest cases (#3974)
-- [AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)
 
 
 Airflow 1.10.2, 2019-01-19

STATIC_CODE_CHECKS.rst

@@ -148,6 +148,8 @@ require Breeze Docker images to be installed locally.
 ------------------------------------ ---------------------------------------------------------------- ------------
 ``build-providers-dependencies``       Regenerates the JSON file with cross-provider dependencies
 ------------------------------------ ---------------------------------------------------------------- ------------
+``changelog-duplicates``               Checks for duplicate changelog entries
+------------------------------------ ---------------------------------------------------------------- ------------
 ``check-apache-license``               Checks compatibility with Apache License requirements
 ------------------------------------ ---------------------------------------------------------------- ------------
 ``check-builtin-literals``             Require literal syntax when initializing Python builtin types

airflow/providers/amazon/CHANGELOG.rst

@@ -44,7 +44,6 @@ Bug Fixes
    appropriate section above if needed. Do not delete the lines(!):
    * ``More f-strings (#18855)``
    * ``Prepare documentation for RC2 Amazon Provider release for September (#18830)``
-   * ``Simplify strings previously split across lines (#18679)``
    * ``Doc: Fix typos in variable and comments (#19349)``
    * ``Remove duplicated entries in changelog (#19331)``
    * ``Prepare documentation for October Provider's release (#19321)``

airflow/providers/cncf/kubernetes/CHANGELOG.rst

@@ -79,10 +79,7 @@ Bug Fixes
 
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
-   * ``Fixed wrongly escaped characters in amazon's changelog (#17020)``
    * ``Simplify 'default_args' in Kubernetes example DAGs (#16870)``
-   * ``Enable using custom pod launcher in Kubernetes Pod Operator (#16945)``
-   * ``Prepare documentation for July release of providers. (#17015)``
    * ``Updating task dependencies (#16624)``
    * ``Removes pylint from our toolchain (#16682)``
    * ``Prepare documentation for July release of providers. (#17015)``

airflow/providers/elasticsearch/CHANGELOG.rst

@@ -100,7 +100,6 @@ Bug fixes
    appropriate section above if needed. Do not delete the lines(!):
    * ``Bump pyupgrade v2.13.0 to v2.18.1 (#15991)``
    * ``Updated documentation for June 2021 provider release (#16294)``
-   * ``More documentation update for June providers release (#16405)``
    * ``Docs: Fix url for ''Elasticsearch'' (#16275)``
    * ``Add ElasticSearch Connection Doc (#16436)``
    * ``More documentation update for June providers release (#16405)``

airflow/providers/microsoft/winrm/CHANGELOG.rst

@@ -64,7 +64,7 @@ Breaking changes
 Features
 ~~~~~~~~
 
-* ``WinRM Operator: Fix stout decoding issue (#13153)``
+* ``WinRM Operator: Allow running as powershell script (#13153)``
 
 Bug Fixes
 ~~~~~~~~~

airflow/providers/mongo/CHANGELOG.rst

@@ -46,7 +46,6 @@ Misc
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Update description about the new ''connection-types'' provider meta-data (#17767)``
-   * ``Import Hooks lazily individually in providers manager (#17682)``
 
 2.0.0
 .....

breeze-complete

@@ -85,6 +85,7 @@ blacken-docs
 boring-cyborg
 build
 build-providers-dependencies
+changelog-duplicates
 check-apache-license
 check-builtin-literals
 check-executables-have-shebangs