GSX Level 2 - Stutters with pax animation [SOLVED]

[kmax59]

Hello,

After a fully clean install of windows and P3D, I am facing an annoying issue when using GSX level 2, as soon as the passengers are boarding or deboarding, I'm having heavy stutters every few seconds.
I don't remember seeing this issue before my clean install but I could be wrong as I didn't fly a lot between GSX Lvl 2 release and my new installation. It seems it also brings some stutters here and there with other animations but not as frequently as passengers.

Before explaining anything in detail, here are some essential informations about my computer and my sim:

- Asus z170 pro gaming
- i7 6700k OC @4.5Ghz (confirmed stable)
- Hyper Threading is enabled
- MSI GTX 1070 Founders Edition
- Corsair Vengeance @2666Mhz 2x8Gb
- Windows 10 x64 Family

- I'm using Prepar3D v4.3 (I'm using the whole Orbx global products as well as some regions and airports with well known advanced aircraft like PMDG, iFly...etc)
- Both windows and P3D running on their own Samsung 860 Evo,
- Addon manager, and sceneries are installed on another 3rd 860 Evo,
- Game mode is disabled
- All P3D related processes are run as an admin
- P3D is running with full screen optimizations disabled (it creates heavy graphic stutters for me, not the same kind of stutters I'm facing with GSX though)
- I don't use any tweak for P3D (no affinity mask or anything else),
- My P3D settings are not over-estimated and are set in order to be running smooth even in worst circumstances, I rarely have less than 30 fps and it only happens on extremely heavy areas / addon sceneries,
- All P3D related folders and processes are excluded from Windows Defender
- I'm externally limiting fps to 30 with the help of Nvidia Inspector

I initially thought it was only GSX itself that was stuttering but it's actually the whole sim (I can see that as panning aroung is also stuttering and animated things like people flow from Orbx also stutters).
After openning the process manager I finally realized that the CPU core 0 was used at 100% when GSX passengers are displayed (and an additional load is obviously expected as no addition is free on performances).

Trying to solve this I managed to partially find a solution, but it's still stuttering:
I opened the task manager and disabled Core 0 in the affinity settings of P3D then I enabled it again. This forced the cpu to balance the load of the sim on all core, preventing the overload of Core 0. Doing this lowered the stutters frequency but are still existing when passengers animations are being played by GSX (about every 10/15 seconds from 4/5 seconds before this test).
This makes me think it's not only about the cpu load but something in GSX could possibly create these stutters.

I tried several other things trying to find the culprit already without real success:

- First of all I updated GSX using the live updater but it didn't make any difference,
- I redownloaded the full installer and updated from there but it didn't make any difference either,
- I crosschecked that all folders and process are already excluded from windows defender (both processes and folders are correctly excluded for all P3D applications),
- I disabled "completely" windows defender without success,
- I tried to lower the passengers density but it didn't make a difference or if any it was a really minor difference,
- I tried to cap the simconnect update without success,
- I tried disabling other addons one by one (the Immersion Manager released a few days ago, chaseplane, active sky and such addons potentially overloading simconnect) without success
- I went back to my cpu stock clock (it was overclocked to 4.5Ghz) without success
- Disabling totally my Nvidia Inspector settings didn't help either (it just makes the sim less smooth)

So as you can see I already explored quite a lot of possibilities without finding any real solution, I definitely think my computer should handle this load especially when the pax density is set to normal.
Do you have any idea where I could still have a look? Could a recent update of GSX be responsible of this?

Thanks a lot by advance

Edit: Here are 3 videos that may help understanding:

First video, GSX is already enabled and is preparing for deboarding, baggage deboarding already began but awaiting passengers bus, the simulator is pretty smooth appart from 1 or 2 stutter during the whole video, not sure if it's GSX or P3D loading something: https://youtu.be/nLgv2Z9H_mo

Second video, doors are opened, GSX passengers are now deboarding, the whole sim is severly stuttering whether I'm looking to the pax or not : https://youtu.be/QzK2vrPj5Xc

Third video, applying the tweak explained above to the cpu affinity reduces the frequency of stutters and balances the cpu load but there are still stutters: https://youtu.be/mWz-YvGEMSE

[virtuali]

After openning the process manager I finally realized that the CPU core 0 was used at 100% when GSX passengers are displayed (and an additional load is obviously expected as no addition is free on performances).

It's difficult to evaluate this information, without knowing how much CPU occupation you had *just* before passengers started to appear.

The issue is, the simulator cannot create objects in multi-threading: the only thing the sim does in multi threading is the terrain generation. If you were already close to 100% CPU utilization before passengers started, and the creation of passengers made you go over 100%, the extra required resources won't be taken from the remaining cores, and the simulator will just stall (stuttering), until enough core cycles will be freed again.

This is just nothing we can do about it: every addon that creates objects (you said yourself animated people flow from Orbx also stutters) will have to compete with the same limited single-core cpu power, and add-ons don't have any control over this, other than just doing less, which is what we can do with the "Passenger Density" slider.

And, there's a difference. While addons that create objects and are in-process (= they are a .DLL, like OrbX Object flow) CAN slow down the sim even with their own logic (their code runs on the single main code too), GSX cannot slow down the simulator with its own logic, since its code runs entirely externally to the sim, under the Couatl.exe interpreter that, being an .EXE, CAN take advantage of the automatic allocation of spare cpu cores made by the OS, for separate processes.  That is, unless you start playing with the Affinity, which gives you back some control over this.

So, you can be sure it's not GSX which is causing this, not its own code, for sure (unless you purposely force Couatl.exe to run on the same core of the sim, by overriding the automatic selection made by the OS, with the Task Manager affinity).

It's the very act of creating objects, which is done by the sim *internally*, following a standard request from addons made with Simconnect, that is not multi-threaded so, if your cpu is already high on utilization, anything that might cross over 100%, will cause stuttering.

The only way to fix this, would reducing the number of objects created, or reducing what is causing the high cpu utilization.

[kmax59]

Hi Umberto,

About the orbx people flow, that's not exactly what I meant, they actually don't stutter on their own, I meant that I can see the whole  sim is stuttering as everything in the sim makes a short stop and it's visible by looking at other animations like people flow. So it's not GSX itself only which is stuttering, it makes the whole sim stuttering, however I've never seen people flow itself stuttering.

Thanks for the explanations about how P3D manages the objects creation and it makes sense indeed.
My cpu load on core 0 is approximately 80 before GSX starts displaying passengers so it could be the reason, however balancing the CPU load with affinity actually doesn't help and stutters are still there whether the cpu load is close to 100% or not.
I definitely have the feeling something is going wrong. Not saying it's GSX itself but may be some bad interactions or settings. As I said previously I don't remember such a behaviour from GSX before my clean install and I'm having a bad time finding the culprit as I don't have a lot of addons installed yet appart orbx, utlive and some aircraft.

I posted 3 small videos to show you in my previous edited message but you probably read it before I add them, please take a quick look at them (tried to integrate the video in the forum but it's reporting invalid links).

- First video, GSX is already enabled and is preparing for deboarding, baggage deboarding already began but awaiting passengers bus, the simulator is pretty smooth appart from 1 or 2 stutter during the whole video, not sure if it's GSX or P3D loading something:

- Second video, doors are opened, GSX passengers are now deboarding, the whole sim is severly stuttering whether I'm looking to the pax or not :

- Third video, applying the tweak explained above to the cpu affinity reduces the frequency of stutters and balances the cpu load but there are still stutters:

[virtuali]

My cpu load on core 0 is approximately 80 before GSX starts displaying passengers so it could be the reason

That explains everything. I bet you have OrbX vector installed ? We noticed on one of our testers system, that just installing OrbX went from about 25-30% core 0 utilization (with default scenery), to 80%, without anything else going on.

This very high impact on cpu load can be explained by the way the simulator creates textures: they are created in real-time by taking the base landclass textures, and rendering roads, railways, rivers, coastlines, and everything other similar vector data, to create tons of new textures, AS IF the scenery was a (synthetic) satellite image, just that is created in realtime.

This means, the more varied the scenery is, the more textures will have to be created. The more accurate the landclass is, the more the process affects the cpu load. And, the more detailed the vector data is, the more it impacts the cpu.

Note the frame rate: if you have enough VRAM, once these textures are created, the fps can still be good, but its their creation that takes an heavy tool on the CPU load.

Nothing is "free" in computers: the convenience of having the whole world using a "reasonable" amount of data (if it was a true photorealistic, it will take MANY TERABYTES of data for the whole world), because is synthesized with vectors, takes an heavy impact on the CPU.

So, if you were already that high ( 80% IS very high ), and GSX passengers took you OVER 100%, since the creation of objects is NOT multi-threaded, the performance monitor won't show you any higher than 100%, but the sim will stutter.

As I've said, there's just nothing we can do about it, other than reducing the number of objects created, and the main setting is the "Passengers Density" slider.

[virtuali]

- Third video, applying the tweak explained above to the cpu affinity reduces the frequency of stutters and balances the cpu load but there are still stutters:

Have you tried with NO affinity setting in the Prepar3d.cfg file (which is the default) and NO affinity tweaks in the Task manager ?

[kmax59]

I am using FTX vector indeed and I know it's a huge performance eater (I don't use all the options), I fully understand that nothing is free on performances and that's why I'm balancing my P3D settings on the performance side instead of the eye candy side. However that's the first time I ever notice such an impact with GSX, while it wasn't the case before my clean install so I'm wondering if I missed something during the reinstall.
I'm using the same settings for months and never had such an issue previously even with early versions of GSX level 2 before performing a clean install.

Reducing the amount of passengers almost does nothing as I explained in my first post.
As for the affinity, I don't use any tweak usually in prepar3D.cfg, it was only for testing purposes.
With or without an affinity mask in the cfg, with or without tweaking the affinity from the tasks manager, it's just stuttering as soon as the boarding or deboarding begins and it's the first time I notice such an impact.
The stutters are much heavier without trying to tweak (but I'm not searching to tweak, this usually brings more downsides than enhancements)

[kmax59]

Hi again Umberto,

I deleted my prepar3D.cfg and let the default P3D settings that are really lower than what I'm usually using considering the PC I'm using.
I started a flight at default KLAS with all orbx addons still installed. CPU load starts much lower (at around 60%) but it doesn't prevent GSX to create stutters during boarding/deboarding, it just still pushes the CPU Core 0 up to 100% even with the sparse setting. Same remark as previously, when balancing the load with affinity, the stutters are reduced but do not disappear.
I definitely think this is not normal, that wasn't happening before on the same rig.

EDIT: Please take a look at this video, you'll see the cpu usage on core 0 is not at 100, GSX pax setting is set to sparse but it's still stuttering when a passenger is being created, it's obviously less often but there's always a short pause which is quite annoying (sorry for constantly panning in the video but that's the best way to detect a pause): https://youtu.be/GFf7oEy1LhQ
You should see a stutter around 0:07/0:08 and another around 0:28/0:29. As you'll see on the video there's not even a spike on the cpu load corresponding to these stutters.

There is definitely something other than the cpu load which is creating a stutter, whether a conflict, something in gsx, a gsx setting or system setting. I'll keep investigating but if you have any idea it would really be welcome, you can be sure that if I'm posting here it's because I really feel something is wrong, I always manage to solve performance issues by myself but here I'm facing something I really cannot solve even by lowering settings. (And as a developer I also know quite well the simulator limits, I really don't want to sound like someone complaining about something which is totally normal, I'm pretty sure it's not here).

[virtuali]

There is definitely something other than the cpu load which is creating a stutter, whether a conflict, something in gsx, a gsx setting or system setting.

Every object will create a very small stutter when it's created, that's just how the sim works. It can be visible or not, depending on:

- The other load on the sim

- Your own perception of stuttering

- The size of the object

- The size of the texture. You can control this: does the stutter become less visible if you lower the maximum texture size ?

[kmax59]

Indeed I know you're not supposed to know but I am aware of these things but here the stutter is far too important to be normal, the initial loading of the whole AI traffic system of UTlive creates a lower stutter than what you see in my videos.
A stutter free P3D is not possible for now (let's hope it will one day) but here we're talking about a 1/2 sec pause, which is really huge for such small objects with such a computer.

Reducing the maximum texture size doesn't help, I decreased all settings to default and the result is exactly the same, the stutters aren't even lower, it's as hard with the lowest setting as with the higher settings. Same applies for the passengers density, decreasing the density decreases the frequency of the stutters but they're as hard as with the highest density.

However, I am probably about to find a reason as I just was able to deboard the PMDG 737 with the density set to insane in the same conditions without a single stutter. I'll try to reproduce this and will let you know as soon as I find the real culprit. It could be helpful for you if it's a conflict with another 3rd party addon.

[virtuali]

here we're talking about a 1/2 sec pause, which is really huge for such small objects with such a computer.

That's not normal, of course. With small stuttering, I mean less than 1/30th of a second.

[kmax59]

Thanks for confirming

I'm still unable to find a precise reason for my issue but after running a few tests may be you would be able to indicate me the right direction
I am running the payware version of FSUIPC and I use it to start my weather programs automatically (Active Sky, ASCA and REX Skyforce). I already tried to close them previously but I had another idea:

- I closed them after they were automatically started by FSUIPC but this time I reset the position in the GSX window, after that I started the deboarding again and it was fine even on the insane density. I reset it several times and started a few other deboarding with different settings and densities and it worked perfectly.

- After that I closed the sim and deleted the lines that start the weather programs, one by one in order to see if a single program is the reason of my issue or if it's because they're started from FSUIPC (which would be surprising as I'm doing this for years). Unfortunately I was unable to get the results I had in the previous session, it was stuttering with all my tests.

- I deleted the FSUIPC dll just to check if it wasn't fsuipc itself (I have the latest version) but it didn't make a difference either.

- I put everything as it was before and tried to replicate the good result I had by proceeding the same way as my first try but without any luck.

- After that I tried to reset UTlive (not closing it, just reset) during a deboarding and it suddenly became perfectly smooth, not a single stutter anymore. I didn't need to reset the position in GSX window.

- In order to be sure it was not because of the amount of programs using simconnect, I started again ASP4, ASCA and Skyforce and all other program that use simconnect, it was still perfectly smooth.

So, following these tests, it sounds like it has something to do with SimConnect but it seems completely random, not due to one specific program and not due to the number of programs using simconnect.

I'm going to uninstall all simconnect version and reinstall them and will let you know.

Other than that, do you have any idea about what could be happening there? Now I'm 500% sure such a thing wasn't happening before my clean reinstall but may be this is due to one of the latest updates? (I didn't perform the latest updates before reinstall).

Thanks for your help

EDIT: I'm starting to get a headache after this day of testing but completely disabling ASP4, ASCA, Skyforce, UTlive, Accu-feel, Immersion manager, then starting the sim and start a deboarding just creates the same stutters, and there's nothing left that is still using simconnect (but gsx and sode of course). I managed to make the stutters disappear again by just playing with the GSX/youcontrol windows, the customize menus etc. I don't know how exactly or why this is happening but it's simply extremely weird. After that, I can restart couatl, rebuild airport cache, reset position as much as I want, GSX doesn't create any stutters until I restart the sim.

[virtuali]

The only thing your tests shows is that, with so many addons running, Simconnect is likely being overflown with too many commands. No, it's not "random", it's just that not all addons sends lots of commands always at the same time so, it looks random to you, but it's not.

UT Live, for example, sends *a lot* of commands, but only in the first 2-3 minutes after starting it, then it settles down.

Active Sky might send lots of commands, but only if there's a new weather reports.

FSUIPC constantly reads a lot of data from Simconnect either.

GSX, of course, hast its turn in sending lots of commands, when animating passengers.

Taken singularly, they wouldn't be able to overflow Simconnect with commands but, if it happens they are all doing their thing all at the same time, you get stuttering.

[kmax59]

Indeed but it's obviously not the reason of the stutters here as I disabled everything which uses SimConnect as I said at the end of my post (in the EDIT part) and it didn't solve anything unfortunately.
The issue comes from somewhere else, in the end it's not simconnect which is overflowed, nor P3D settings, nor GSX density setting, I'm completely out of ideas.

Another thing I've seen, there wasn't any FSDT_jetwaynumber.lua in the scripts subfolder of the addonmanager earlier today, now there's one, and after a reinstall the addon.xml of the addon manager includes the scripts folder (which wasn't part of the addon beforehand).

[virtuali]

Could simply be the overclock which is causing the cpu to overheat, so it's throttling down for its own safety ?

[kmax59]

That's not the case either, I have a very good watercooling and temps are difficultly raising to 70°C at full load, that's really cool for an overclocked 6700k.
And I also tried without overclock during my early testing about this issue but it didn't make a difference, well actually it made it worse.

The GPU temps and clocks are fine as well as the memory so I doubt it could be related to hardware, and there's nothing else that would indicate there's a hardware problem as the sim is running very smoothly.
Everything else than Boarding/deboarding of GSX is running as it was used to. I'm running 3 SSD, 1 for windows, 1 for P3D/Orbx/Aircraft and another 1 for sceneries (like yours and GSX as well) so it couldn't be caused by a slow HDD.

I know by experience it doesn't necessarily mean it's GSX itself however, but here I really have explored everything that came to my mind.