Open
Description
Hi,
This is a documentation request to have a flowchart of all the pytest test session states (setup, conftest, collecting, run setup, run test, run teardown, make report, etc..) together with all the applicable hooks.
Something similar to the one on logging module documentation (I suppose that for pytest this would be way more complex):
https://docs.python.org/2/howto/logging.html#logging-flow
I surely read the page on the hooks but not knowing all the steps that are taken by pytest it becomes a bit hard to guess which one we should be using. This would really improve the understanding on how pytest works, especially for plugin developers.
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Activity
nicoddemus commentedon Feb 26, 2018
Hi @Sup3rGeo, this would definitely be useful.
I think we can use sphinx.ext.graphviz to generate the flowcharts.
nicoddemus commentedon Feb 26, 2018
Just played around with
sphinx.ext.graphviz
and this code:Produces this image:
So this is definitely doable. 👍
Sup3rGeo commentedon Feb 27, 2018
Nice!
I would be willing to help with documentation (with graphviz) but I would just need someone who knows pytest execution flow to define in any way (text or in a tool like draw.io for instance)
nicoddemus commentedon Feb 27, 2018
@Sup3rGeo thanks for the offer, we appreciate it!
The hook order can be obtained by passing
--debug
, which will make pytest write apytestdebug.log
file with contents like this:Which can be used as a starting point, and in any case if there are any questions we would gladly answer them over a WIP PR. 😁
Sup3rGeo commentedon Feb 28, 2018
That is really great, can't believe all this time I didn't know it!
I will try to come up with something then and create PR so you can review it.
Sup3rGeo commentedon Feb 28, 2018
I was thinking about doing something like this:
https://github.com/Sup3rGeo/pytest/blob/ace24468d336fdd4a8c2eaa3c3e9a32da961f98b/doc/en/img/Pytest%20hooks.png
However I am not sure how to do this with graphviz. Mainly to represent the fact that some hooks are executed inside the context of other hooks.
What do you think?
nicoddemus commentedon Feb 28, 2018
I never used
graphviz
myself 🤕I think it looks great! Also, perhaps you can write a script to generate the
graphviz
output from apytestdebug.log
file? Then we can automate this graph generation for each release.Sup3rGeo commentedon Mar 1, 2018
That is a great idea. From what I could see, not all the hooks are shown in the log file. Is that dependent on what pytest features the test is actually using?
If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?
nicoddemus commentedon Mar 1, 2018
Not offhand, but IMHO don't worry about it too much: having the script in place generating a graph with the most common hooks and integrated with our release process is already a huge addition. We can always improve that bit later. 😁
Sup3rGeo commentedon Mar 1, 2018
So basically I wrote something to parse the log file and create a tree using
anytree
module:https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.py
The result:
The challenge was to not repeat hook calls and merge internal hooks from some calls that repeat but not exactly one after the other. But it seems to be working fine.
I faced just one problem though - the last
pytest_runtest_setup [hook]
line did not have a correspondingfinish pytest_runtest_setup
. But I am going to run the test again to see.It also has the ability to generate dot graphs out of it, although the tree text representation is already very good and this graph is just unreadable:
https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.png
Sup3rGeo commentedon Mar 1, 2018
So I ran a test again and same thing happened: One of the
pytest_runtest_setup
hooks did not have a correspondingfinish pytest_runtest_setup
line. This will confuse the script I wrote.One possible solution is to consider the leading whitespaces to do the nesting.
Sup3rGeo commentedon Mar 1, 2018
This might be also useful for #3113 and #2670
RonnyPfannschmidt commentedon Mar 2, 2018
looks fabulous, can we add the scripts somewhere to the pytest repo so we can update in future (maybe it makes sense to put this into pluggy)
Sup3rGeo commentedon Mar 2, 2018
@RonnyPfannschmidt nice to hear that!
Merging repeated calls inside a parent is problematic because it does not capture the fact that, for instance, pytest__runtest_makereport runs three times, after setup, call, and teardown respectively.
On the other hand, if we leave everything as is then typically you can have much useless repetition, for example many many pytest_plugin_registered and then the hooks from pytest_logstart to pytest_logfinish for every test case if you have many.
So we might have to craft a test that will call just the right amount of hooks, so it looks good on the script, or we should perhaps think of other way.
If we do it manually then it is easier to indicate that a certain hook sequence is repeated for every item collected, etc.
26 remaining items