I'm having trouble understanding how this conceptually works. The presence of a plugin makes me think the grass is hand-placed objects, while the Density config parameter makes it seem as though it's placed procedurally. What determines what grass appears where and by how much?

What determines what grass appears where and by how much?

Now statics which ID's have the "grs" substring are considered to be a grass. 
We need a more reliable approach to determine grass objects.

Attach grass nodes directly to cell node, without creating 
a cellref and animation object just for every grass shape. 

Basically, it is possible just to skip part of grass objects from plugin during cell loading.
For example, with density = 0.5 I skip every second grass object.

Yes, but you also mention a density parameter, and that there's no batching so high density means lower performance. If grass is instead a static object (which is detected with the "grs" substring and given a "fast path" of sorts), that means grass is hand-placed by the plugin, so where does the density come in and what does it do?

I do not understand the question.
I already told that I just skip part of grass objects during cell load (depending on selected grass density). MGE does pretty the same thing, but it uses a random generator instead (so 50% means that every grass object has a 50% chance to be rendered).
I doubt that there is a simpler explanation, which would be shorter than code itself.

I don't think we should be merging this, no matter the cool-factor, until at the minimum we're accepting changes to the ESM format, so could tag grass with its own field.

I don't see how we could be adding grass to the scene without it being capable of receiving lighting. , but I guess you could be skipping whatever system adds a light update callback to each object when you're skipping doing things.

so 50% means that every grass object has a 50% chance to be rendered

That's what I was unsure about. So density = 0.5 means each placed grass object in the plugins has a 50% chance to "spawn"?

The density is just a performance setting.

So density = 0.5 means each placed grass object in the plugins has a 50% chance to "spawn"?

If you re-read my message, you may notice that it was the comparison with MGE:

I already told that I just skip part of grass objects during cell load (depending on selected grass density).

MGE [in comparison] does pretty the same thing, but it uses a random generator instead (so 50% means that every grass object has a 50% chance to be rendered).

OpenMW just loads all objects in cell one by one during cell load.
With 75% density I drop every 4th encountered grass object, with 50% - every second, with 10% I handle only every 10th encountered grass object.
Since grass generators usually sequentially place next grass object nearby previous grass object, grass density remain more-or-less constant in different places (it is not a thing when using a random generator).

So what did we decide to do?

In my opinion this isn't the way to add grass, and it's no use to add it if we can't get on par with MGE XE:

1. The most common complaint of the users is performance, not how the grass looks. The performance is only changed slightly.
2. Even then, the grass still looks not that good.
3. The way the grass is integrated into ESM is awful, crap mistap. And it'll have to be updated anyways breaking existing content. Though I understand that this is just a proof of concept.
4. The performance optimizations are questionable still.
   All in all, not how grass should be approached. Give it another time, in a better world.

Yes, do nothing. It's better than doing something hacky to satisfy users for only a brief moment. We're OpenMW, we have full source code access, we should surpass MGE XE which works with a proprietary engine.

The performance is only changed slightly.

It is not true. Difference from master:

1. Grass density is configurable.
2. Grass rendering distance is configurable, and is not affected by the "exterior cell load distance" value.
3. Grass does not cast shadows.
4. Grass almost does not affect cell loading speed, while in master it cripples loading speed.
5. Memory consumption is lesser (about 300MB lesser consumption with the 100% density and about 750MB with the 30% density with the "exterior cell load distance = 1").

With 30% density and 8192 units distance grass costs me about 2ms of Draw and GPU time on GTX1050 while still looks a quite good. Actually, trees replacer for West Gash region costs me more FPS than grass.
With more powerful GPU the FPS drop with same settings more likely will be negligable, so it will be possible to use a higher density.

In the upstream master performance drop is MUCH larger, but many players still enable grass. Some modpacks for OpenMW even enable grass mods by default.

Couldn't come to an agreement in a discussion on the Fullrest Discord server. I cannot override potential executive decisions, but I stand by my opinion that we still need full parity with MGE XE before either us or some of the players stand advertising "grass support" that cannot satisfy all players that could move from Morrowind, and having a placeholder solution for everything in place because "some grass is better than no grass!" isn't good. I'll avoid continuing arguing.

I want to label this: technical debt

It's great you wanted to tackle this problem, but I'm not convinced that this will get us closer to having a working "Grass solution" and that we'll just end up ditching this code in the future. It just isn't a complete solution and doesn't help us carry OpenMW towards 1.0

Another thought came to me too... people want their mods to be able to work on both OpenMW and Morrowind (and other games). So being able to handle existing mods isn't out of the question.

So unless the majority of developers (stake holders?) think this is a good idea; I'm not keen on signing off on it. If you really think this is the way forward, you'll need to convince more developers that you're on to something. I'm abstaining on this issue in either case.

but I stand by my opinion that we still need full parity with MGE XE

We already violated that rule when merged a Distant Land which supports only terrain, or shaders which do not implement all features which MGE XE's shaders have.

That's a false equivalence. There's no way to have a better terrain system than the terrain paging system, so we'll never have to replace it. The vertex and fragment shaders we use aren't remotely the same thing as post process shaders, and the vanilla engine uses vertex and pixel shaders under some circumstances, so even if they were related, it's not exclusively an mge thing

There's no way to have a better terrain system than the terrain paging system, so we'll never have to replace it.

I do not understand the logic.
From the one hand, it is OK to have a partially implemented Distant Land (which supports only terrain) without the "full parity with the MGE XE" just because no one provided an implementation which supports objects yet.
From the other hand, it is not OK to have an optimization for already existing and widely used assets because this optimization does not provide the "full parity with the MGE XE", despite it is even not claimed to be a part of MGE's features re-implementation.

Distant terrain was never intended to be a full replication of MGE XE's distant land functionality, it's a terrain system first and foremost.

Distant terrain was never intended to be a full replication of MGE XE's distant land functionality, it's a terrain system first and foremost.

This PR was never intended to be a full replication of MGE XE's grass functionality, it's an optimization for already used assets first and foremost.

IIRC, the "already used assets" are explicitly for MGE XE's grass system. The existing grass mods aren't supposed to be used on a running game, but instead are given to MGE's LOD generator and that handles it from there. So proper use of these assets would depend on a system like MGE's since that's what they're designed for. OpenMW's Distant Terrain, in contrast, is a system for handling terrain with far distances, providing similar functionality to MGE's, but in a different way that doesn't share resources/assets with it and can continue to be built upon separately.

I'll just reiterate what I said before about this PR, though. I don't mind it if it's functioning, and it doesn't get in the way of proper grass handling later. It would be a stop-gap to try and use some MGE grass mods to pretty up the terrain a bit, but shouldn't be considered the way forward with how to handle grass. It just depends if the added maintenance is worth it.

Any ideas how the grass batching should be done? That is to my understanding the main culprit for bad performance.

Can this PR be developed further to become the solution or part of the solution that everyone wants?

I lack understanding of the technical details, but my five cents: Supporting grass mods for Morrowind is probably a good goal before extended features, but I agree that performance for the grass is most important goal. Animating grass has to be possible, but having a system that supports adding animation support later should be acceptable. Making procedural generation for grass shouldn't be a problem, but it's useless without performance.

Any ideas how the grass batching should be done? That is to my understanding the main culprit for bad performance.

Can this PR be developed further to become the solution or part of the solution that everyone wants?

Unlikely. This PR depends on having a mod manually place each individual grass mesh, with some engine-side recognition of what objects count as "grass" to give it a more efficient render path. A more appropriate solution would likely depend on extending the esm format to define grass volumes (which themselves define the type/texture and height of the grass), and then the engine automatically generates the appropriate grass meshes on the ground in those volumes, batching as appropriate.

That's just my initial thoughts, though. It might be helpful to look into how other engines like Unity or UE handle dense dynamic foliage.

Batching for legacy-style grass mods should be the same as batching for anything else. It's just one of the worst offenders right now for bazillions of tiny draw calls.

I don't think it's unreasonable to handle grass separately to other objects (e.g. to make shadows optional), either, just I think it should be handled by a flag in the ESP rather than a check in the name for a substring. Whether or not the grass is placed manually or procedurally, and whether or not the procedural placement is done at runtime or by the CS doesn't change that.

I can think of two potential approaches for procedural grass, and both would end up driven by grass density textures either in the ESP like how we currently have blendmaps for terrain textures, or inferred by assigning each terrain texture a grass density for each grass type. Either we pass the terrain tile to a geometry shader with the grass density texture bound and generate fully procedural grass based on that, or we define collections of grass meshes (e.g. we have one collection with six different tufts of green grass and one dandelion, and another with four tufts of ashlands grass) and have a density texture for each that gets used for each terrain tile, and place meshes when loading the world based on these textures. Both of these options are used elsewhere and can look good. They also both allow for an overall grass density setting that gets multiplied into the texture to reduce the density on lower-end machines.

It might be helpful to look into how other engines like Unity or UE handle dense dynamic foliage.

IIRC, UE4 games usually place vegetation objects randomly when you visit a location in the first time and store positions in savegames. It allows trees/bushes/grass/etc to be in the same place when you re-visit location later.
Technically, foliage is a special kind of object with an arbitrary mesh:

As I understood, a basic approach is pretty similar to MGE's - you create vegetation records with assigned meshes, place them to scene (manually via game editor, by an external generation tool, or you embed generation tool to the game executable). Engine uses a some kind of batching to render vegetations properly.

the code is right there

It does not really help.

You or someone else could also work with OP/AG to allow for the same technique of keying off the 'grass' directory name and giving a different (shorter) rendering distance than the camera view distance.

I tried, but

OP was designed to be a black box rather than readable and extendable module, so it is very hard to work with it

As I said, I'd like to make OP to work with groundcover, but I did not find a way to do it without major rewrite. A major rewrite is out of question sinse no one excepts of bzzt knows about how this thing works and can modify it without breaking anything.

Also I still did not get an aswer how you are going to make grass animation to work with pages, without adding an additional code.

The object paging MR doesn't actually change that much stuff. I'm pretty sure it's just leveraging OSG's database paging system, plus merging suitable geometries. The details shouldn't be too opaque.

I'd expect adding a special case for grass to probably work something like:

• Find the identical grass meshes in a cell.
• Use instanced rendering by having a position vector and Euler rotation vector for each of them instead of statically merging them.
• If they're past the fade distance, kill them in the vertex shader.
• The mesh origin is the same as before, so animation is the same as before.

All we really need is the loader to tag the grass meshes somehow to say that we want to use the special case. The instancing bit might not need that, as we can probably work out the threshold at which it's better to use instanced rendering than plain merging and have the merging function decide that generically.

If they're past the fade distance, kill them in the vertex shader.

By using "discard"? I tried to do it, but it decreased performance a lot (IIRC, bzzt's implementation of radial clipping had the same issue), so I do it on CPU now by setting node masks to 0.
Also probably it would be better to "kill" such nodes on the QuadTreeWorld level (as we do for terrain chunks), but the whole QuadTreeWorld is supposed to have a single viewing distance.

discard only works in the fragment shader, but I said to kill it in the vertex shader. If you know a vertex and every vertex it's connected to aren't going to be visible, you can set the location of the vertex outside the viewing frustum (or, even better, set its clip position to the wrong side of a clipping plane, or one of its clip distances negative), and it should get ignored much earlier in the pipeline. You kind of have to have the object's bounds in its vertex data, a uniform, or a texture, though, otherwise you can't determine the every vertex it's connected to until the geometry shader.

you can set the location of the vertex outside the viewing frustum (or, even better, set its clip position to the wrong side of a clipping plane

Can you provide an example, please? Articles which I found either require newer GLSL version or suggest to cull objects on CPU instead.

An only thing which I managed to get working with GLSL 1.2 is such thing:

    if (euclideanDepth > @grassFadeEnd)
       gl_Position.z = 10000000;

While it works, it does not improve performance at all.

Use instanced rendering by having a position vector and Euler rotation vector for each of them instead of statically merging them.

I thought about it, but:

1. I did not manage to find any reusable instancing example for OSG, which doesn not require OpenGL 3.0 or higher.
2. More likely instancing will add a good chunk of new code and will work slower than static batching on psi29a's hardware, so drawbacks will remain the same.
3. If we are going to use instancing instead of batching, we do not need to use ObjectPaging for groundcover objects at all anyway.
4. Judging by docs, GLES 2 does not support instancing, so Android users may have issues with grass.

I only managed to find something like this, but I did not find a way to assign an arbitrary mesh to the instanced geometry. Probably we need somehow to convert the whole mesh to the vertex buffer and color buffer.

1. I did not manage to find any reusable instancing example for OSG, which doesn not require OpenGL 3.0 or higher.

Wouldn't this be a case where we are okay with this? We have stated before that for some features, we can fence them off based on GL extension.

We do not support GLES 2 nor 3 and make no claims to do so.

Static batching, works well but not with animations. So animated foliage is probably not going to be a thing in OP/AG unless we implement either dynamic batching or GPU instancing.

Dynamic batching works for very small meshes but is CPU intensive.

GPU Instancing is meant to be used by modern hardware. According to Unity (for example): GPU Instancing is available on OpenGL >= 4.1+ or ES >= 3.0

According to Unity (for example): GPU Instancing is available on OpenGL >= 4.1+ or ES >= 3.0

glDrawElementsInstanced / glDrawArraysInstanced should be available since OpenGL 3.1 and GLES 3.0, in OpenGL 2.1 instancing seems to be available via ARB extensions, but OSG does not handle them out of box.

Static batching, works well

Unfortunately, only in some cases.

I keep seeing people claim that OSG doesn't support instancing out of the box (including its own mailing list), but it actually lives in the exact same place as all the rest of the rendering. If you have any vertex attributes bound per-instance rather than per-draw-call or per-vertex, it automatically turns on instancing. It'll only work if the OpenGL version is high enough or the ARB or EXT extension exists, but they're present on some seriously old hardware. In the compatability database I'm looking at, most of the cards that are listed as incompatible are seriously old ATi cards with Mesa or Gallium drivers that are also listed as compatible when a newer driver is used.

I don't have too much data for how often mobile GPUs expose the extension if they're in ES 2.0 mode but are ES 3.0 capable, but as Psi said, we don't officially support it in the first place, so I think this is a decent candidate for a situation where the fast path is only available on reasonably modern hardware.

As for whether this should be Object Paging's job to handle, I don't see instancing as wildly different from merging objects into the same draw call any other way. It'll be a good tool for it to have in its arsenal, and if it turns out that a cell with the same rock fifty times can be drawn as fast with less memory pressure, that's a win we'll get for free by adding it.

@AnyOldName3, do you have ideas how to make animations to work with object paging?

So far the main blocker is that we do not need to animate weeds roots. Since they are near or below mesh pivot point, there is a good assumption "higher gl_Vertex.z -> higher animation speed". Unfortunately, it does not work for pages since technically every grass page is just a single huge plant, partially located underground.

I need somehow to know a relative Z coordinate of each vertex from original mesh in grass shaders to use it instead of gl_Vertex.z.

That's one of the things that'll become easier if instancing is added as a type of batching to object paging. As far as (that part of) the vertex shader's concerned, it's the same mesh as it always was.

that'll become easier if instancing is added as a type of batching to object paging

I know, but we do not have instancing, and it more likely will not work on all platforms where OpenMW exists.
I suppose that we will need to use animations for batches anyway in such case.

It'll work on every platform OpenMW supports, so I feel the discussion is academic as it's dumb to throw away performance to make things easier for unsupported platforms. Adding one extra float as a vertex attribute for the Z threshold when merging the buffers would be a solution, though. Alternatively, and this leans pretty heavily towards doing things entirely differently for grass, the terrain heightmap could be bound as a texture, and then the distance from the ground could be computed.

Not sure if already talked about, but here is an OSG example:
https://github.com/openscenegraph/OpenSceneGraph/blob/master/examples/osgdrawinstanced/osgdrawinstanced.cpp
that is makes use of the GL_ARB_draw_instanced extension.

From the doc: https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_draw_instanced.txt

OpenGL 2.0 is required.

The extension is written against the OpenGL 2.1 Specification.

EXT_gpu_shader4 or NV_vertex_program4 or OpenGL 3.0 is required.

There is also a bit more in the OSG3 Cookbook about instancing if anyone is interested.

Digging deeper is this: https://github.com/openscenegraph/OpenSceneGraph/tree/master/examples/osggpucull

• instanced rendering capable of drawing thousands of different objects with almost no CPU intervention ( cull and draw times are close to 0 ms ).
• input objects may be sourced from any OSG graph ( for example - information about object points may be stored in a PagedLOD graph. This way we may cover the whole countries with trees, buildings and other objects ). Furthermore if we create osgDB plugins that generate data on the fly, we may generate information for every grass blade for that country.
• every object may have its own parameters and thus may be distinct from other objects of the same type.
• relatively low memory footprint ( single object information is stored in a few vertex attributes ).
• no GPU->CPU roundtrip typical for such methods ( method uses atomic counters and glDrawArraysIndirect() function instead of OpenGL queries. This way information about quantity of rendered objects never goes back to CPU. The typical GPU->CPU roundtrip cost is about 2 ms ).
• this example also shows how to render dynamic objects ( objects that may change its position ) with moving parts ( like car wheels or airplane propellers ) . The obvious extension to that dynamic method would be the animated crowd
  rendering.
• rendered objects may be easily replaced ( there is no need to process the whole OSG graphs, because these graphs store only positional information ).

But! The cost of this is...

The example uses various OpenGL 4.2 features such as texture buffer objects, atomic counters, image units and functions defined in GL_ARB_shader_image_load_store extension to achieve its goal and thus will not work on graphic cards with
older OpenGLversions.

The example was tested on Linux and Windows with NVidia 570 and 580 cards.

It's clear that by sticking to GL2, we're just hamstringing ourselves. To reiterate @AnyOldName3, adding features and having them fenced off by extension checks is preferable than no feature at all.

instancing is added as a type of batching to object paging.

Honestly, I see no point to use instancing in object paging.
From what I can tell, the whole purpose of QuadTreeWorld is to be a terrain implementaion, which is able to load large shapes (chunks) from cached providers (e.g. ChunkManager or ObjectPaging), attach them to the game world and detach them when they are not needed (so such terrain can work with large viewing distance). Instancing works with original meshes and has no relation to chunks, so it is quite strange to use it in the ObjectPaging, which is just a chunk provider.

I see two real alternatives here:

1. Do not take in account Android port and add instancing to this PR. In this case we will get pretty the same implementation as in MGE, with MGE-like animations.
2. Use batching instead of instancing. In this case it is possible to use a batch of hacks to improve performance (for example, setup a separate QuadTreeWorld for groundcover objects with a separate viewing distance), something like this. But it is quite hard to make animations to work with this approach.

the terrain heightmap could be bound as a texture

I suppose that it will not work since by design there can be not-animated meshes (e.g. stones, the whole mesh is below pivot point), or partially animated meshes (e.g. trees, bushes or mushrooms, a part of mesh is below pivot point).
Vertex attributes may work, but I did not find where OSG converts vertex transformations when it merges shapes. Probably it would be better to ask bzzt about it, but it is up to @psi29a.

I think you're misunderstanding what instancing is. It's just another kind of batching, which, for things like grass, might give better performance than dumb stick-everything-into-the-same-vertex-buffer-pre-transformed batching, and will definitely use less memory. Object paging has two jobs, and one of them is putting things into batches.

It's just another kind of batching

It is an another kind of batching, which does not need neither ChunkManager (and ObjectPaging is a ChunkManager) nor QuadTreeWorld (which works with chunks).

Object paging has two jobs, and one of them is putting things into batches.

Chunk manager has only one job - provide QuadTreeWorld chunks (which are just osg::Node's to attach to scene) for given cells by its request.
The rest is just details of implementation (for example, basic ChunkManager generates chunks by using terrain storage, while ObjectPaging iterates over target cells and merges objects meshes to chunks). The point is that chunks are static, so they can be cached and used for a quite large amount of time. What do you suggest to use as chunks for QuadTreeWorld, when using it with instancing?
As for me, ObjectPaging is complex enough even without instancing.

What I'm suggesting is to add an extra feature to the optimiser part (which I think might live in sceneutil rather than object paging itself) to do the instancing if it makes more sense as a merge strategy than sticking everything into the same model space and putting it in one big vertex buffer, and then some other stuff either in the loader or object paging's create chunk function that sets up the grass-specific stuff.

@psi29a, if you do not like a check which is based on mesh folder, there is a possibility is just to register groundcover mods separately from normal content files, for example:

content=Morrowind.esm
groundcover=Grass Vanilla 1.esp

In this case any static object record, added by such content file, is treated as groundcover object (by setting the flag in the ESM::Static). In this case groundcover objects detection can become more reliable and fast. After OpenMW-CS 1.0 release this flag can become a part of ESM format and be modified via editor.

I actually like that a lot more than I like the mesh path check. Everything's explicit and there's no chance of picking things up by mistake.

Actually yes, that is a lot nicer to deal with.

Superseded by #3010.