Skip to content

Flowchart of pytest test session states and hooks #3261

Open
@Sup3rGeo

Description

@Sup3rGeo
Member

Hi,

This is a documentation request to have a flowchart of all the pytest test session states (setup, conftest, collecting, run setup, run test, run teardown, make report, etc..) together with all the applicable hooks.

Something similar to the one on logging module documentation (I suppose that for pytest this would be way more complex):
https://docs.python.org/2/howto/logging.html#logging-flow

I surely read the page on the hooks but not knowing all the steps that are taken by pytest it becomes a bit hard to guess which one we should be using. This would really improve the understanding on how pytest works, especially for plugin developers.

Activity

nicoddemus

nicoddemus commented on Feb 26, 2018

@nicoddemus
Member

Hi @Sup3rGeo, this would definitely be useful.

I think we can use sphinx.ext.graphviz to generate the flowcharts.

added
type: docsdocumentation improvement, missing or needing clarification
on Feb 26, 2018
nicoddemus

nicoddemus commented on Feb 26, 2018

@nicoddemus
Member

Just played around with sphinx.ext.graphviz and this code:

.. digraph:: foo

    "pytest_cmdline_main" -> "pytest_cmdline_preparse" -> "pytest_load_initial_conftests";

    "pytest_cmdline_main" -> "pytest_cmdline_parse";

Produces this image:

graphviz-45befaaca8f80d2a294cbf4685e5fc9ca1222c57

So this is definitely doable. 👍

Sup3rGeo

Sup3rGeo commented on Feb 27, 2018

@Sup3rGeo
MemberAuthor

Nice!

I would be willing to help with documentation (with graphviz) but I would just need someone who knows pytest execution flow to define in any way (text or in a tool like draw.io for instance)

nicoddemus

nicoddemus commented on Feb 27, 2018

@nicoddemus
Member

@Sup3rGeo thanks for the offer, we appreciate it!

The hook order can be obtained by passing --debug, which will make pytest write a pytestdebug.log file with contents like this:

versions pytest-3.4.1.dev39+ga1c3c3d.d20180226, py-1.5.2, python-3.6.3.final.0
cwd=C:\pytest
args=['.tmp\\test_fail.py', '--debug', '-v']

  pytest_cmdline_main [hook]
      config: <_pytest.config.Config object at 0x0000016BDEBF8BE0>
    pytest_plugin_registered [hook]
        plugin: <Session '.tmp'>
        manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
    finish pytest_plugin_registered --> [] [hook]
    pytest_configure [hook]
        config: <_pytest.config.Config object at 0x0000016BDEBF8BE0>
      pytest_plugin_registered [hook]
          plugin: <_pytest.cacheprovider.LFPlugin object at 0x0000016BDEFF4358>
          manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
      finish pytest_plugin_registered --> [] [hook]
      pytest_plugin_registered [hook]
          plugin: <_pytest.logging.LoggingPlugin object at 0x0000016BDEFF4518>
          manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
      finish pytest_plugin_registered --> [] [hook]
    find_module called for: py._io [assertion]
    find_module called for: py._io.terminalwriter [assertion]
    find_module called for: termios [assertion]
    find_module called for: termios [assertion]
      pytest_plugin_registered [hook]
...

Which can be used as a starting point, and in any case if there are any questions we would gladly answer them over a WIP PR. 😁

Sup3rGeo

Sup3rGeo commented on Feb 28, 2018

@Sup3rGeo
MemberAuthor

That is really great, can't believe all this time I didn't know it!

I will try to come up with something then and create PR so you can review it.

Sup3rGeo

Sup3rGeo commented on Feb 28, 2018

@Sup3rGeo
MemberAuthor

I was thinking about doing something like this:
https://github.com/Sup3rGeo/pytest/blob/ace24468d336fdd4a8c2eaa3c3e9a32da961f98b/doc/en/img/Pytest%20hooks.png

However I am not sure how to do this with graphviz. Mainly to represent the fact that some hooks are executed inside the context of other hooks.

What do you think?

nicoddemus

nicoddemus commented on Feb 28, 2018

@nicoddemus
Member

I was thinking about doing something like this
However I am not sure how to do this with graphviz.

I never used graphviz myself 🤕

What do you think?

I think it looks great! Also, perhaps you can write a script to generate the graphviz output from a pytestdebug.log file? Then we can automate this graph generation for each release.

Sup3rGeo

Sup3rGeo commented on Mar 1, 2018

@Sup3rGeo
MemberAuthor

I think it looks great! Also, perhaps you can write a script to generate the graphviz output from a pytestdebug.log file? Then we can automate this graph generation for each release.

That is a great idea. From what I could see, not all the hooks are shown in the log file. Is that dependent on what pytest features the test is actually using?
If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?

nicoddemus

nicoddemus commented on Mar 1, 2018

@nicoddemus
Member

If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?

Not offhand, but IMHO don't worry about it too much: having the script in place generating a graph with the most common hooks and integrated with our release process is already a huge addition. We can always improve that bit later. 😁

Sup3rGeo

Sup3rGeo commented on Mar 1, 2018

@Sup3rGeo
MemberAuthor

So basically I wrote something to parse the log file and create a tree using anytree module:
https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.py

The result:

root
└── pytest_cmdline_main
    ├── pytest_plugin_registered
    ├── pytest_configure
    │   └── pytest_plugin_registered
    ├── pytest_sessionstart
    │   ├── pytest_plugin_registered
    │   └── pytest_report_header
    ├── pytest_collection
    │   ├── pytest_collectstart
    │   ├── pytest_make_collect_report
    │   │   ├── pytest_collect_file
    │   │   │   └── pytest_pycollect_makemodule
    │   │   └── pytest_pycollect_makeitem
    │   │       └── pytest_generate_tests
    │   │           └── pytest_make_parametrize_id
    │   ├── pytest_collectreport
    │   ├── pytest_itemcollected
    │   ├── pytest_collection_modifyitems
    │   └── pytest_collection_finish
    │       └── pytest_report_collectionfinish
    ├── pytest_runtestloop
    │   └── pytest_runtest_protocol
    │       ├── pytest_runtest_logstart
    │       ├── pytest_runtest_setup
    │       │   └── pytest_fixture_setup
    │       ├── pytest_runtest_makereport
    │       ├── pytest_runtest_logreport
    │       │   └── pytest_report_teststatus
    │       ├── pytest_runtest_call
    │       │   └── pytest_pyfunc_call
    │       ├── pytest_runtest_teardown
    │       │   └── pytest_fixture_post_finalizer
    │       └── pytest_runtest_logfinish
    ├── pytest_sessionfinish
    │   └── pytest_terminal_summary
    └── pytest_unconfigure

The challenge was to not repeat hook calls and merge internal hooks from some calls that repeat but not exactly one after the other. But it seems to be working fine.

I faced just one problem though - the last pytest_runtest_setup [hook] line did not have a corresponding finish pytest_runtest_setup. But I am going to run the test again to see.

It also has the ability to generate dot graphs out of it, although the tree text representation is already very good and this graph is just unreadable:

https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.png

Sup3rGeo

Sup3rGeo commented on Mar 1, 2018

@Sup3rGeo
MemberAuthor

So I ran a test again and same thing happened: One of the pytest_runtest_setup hooks did not have a corresponding finish pytest_runtest_setup line. This will confuse the script I wrote.

One possible solution is to consider the leading whitespaces to do the nesting.

Sup3rGeo

Sup3rGeo commented on Mar 1, 2018

@Sup3rGeo
MemberAuthor

This might be also useful for #3113 and #2670

RonnyPfannschmidt

RonnyPfannschmidt commented on Mar 2, 2018

@RonnyPfannschmidt
Member

looks fabulous, can we add the scripts somewhere to the pytest repo so we can update in future (maybe it makes sense to put this into pluggy)

Sup3rGeo

Sup3rGeo commented on Mar 2, 2018

@Sup3rGeo
MemberAuthor

@RonnyPfannschmidt nice to hear that!

Merging repeated calls inside a parent is problematic because it does not capture the fact that, for instance, pytest__runtest_makereport runs three times, after setup, call, and teardown respectively.

On the other hand, if we leave everything as is then typically you can have much useless repetition, for example many many pytest_plugin_registered and then the hooks from pytest_logstart to pytest_logfinish for every test case if you have many.

So we might have to craft a test that will call just the right amount of hooks, so it looks good on the script, or we should perhaps think of other way.

If we do it manually then it is easier to indicate that a certain hook sequence is repeated for every item collected, etc.

26 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: docsdocumentation improvement, missing or needing clarification

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @hectorv@flub@RonnyPfannschmidt@The-Compiler@nicoddemus

        Issue actions

          Flowchart of pytest test session states and hooks · Issue #3261 · pytest-dev/pytest