This holiday took a different route from what I had planned on. I was planning to learn vCloud Directory and NSX, and heading off to visit some other countries. I was mentally tired from work, and I felt like dead person, but I came across topic so interesting that I could not let go before pushing to the limits. It kept me awake night and day. My holiday became three weeks of playing and tinkering with Horizon and different GPUs, and exploring VMware Horizon’s limits—and zero weeks traveling.
I would like to thank VMware vExperts and Scottish VMUG community members for advising and thinking with me when I was playing and tinkering with Horizon and different GPUs. I especially want to give a big thanks to vExperts Stephen Daniel Wagner, Dan Skwara, Stephen Jesse, and Scottish VMUG members Iain Balmer and Craig Dalrymple, among the many others.
I did not keep a journal during this exploration, and there has been a lot of action in the background and interesting conversation. Some of these thoughts are mine, and some are others’; it’s difficult to keep track of it now. I’ll try to sum up all the information, and I hope it will be useful for you. This is not a science paper. Just me tinkering and playing with Horizon.
FPS—Frames Per Second.
FPS is considered as the key measurement for gamers, and it should describe how smoothly a given game runs on your PC. Lower framerates—typically lower than 30 FPS or so—will appear choppy or slow. It’s a useful metric for evaluating your hardware’s gaming performance, and often touted by PC enthusiasts looking to boast about their system. For example, if your game is running slowly, displaying the frame rate can help you figure out which graphics settings to turn down for the most effective performance improvements.
(Source: pcmag.com Whitson Gordon)
Monitoring the framerate alongside other hardware stats—like CPU, GPU, and VRAM usage—can even tell you which component is the bottleneck in your system, and where you’d benefit most from an upgrade.
Playable framerate is 60 FPS and above for single-player games.
120 FPS and higher is standard for multiplayer games and first-person shooters.
Here is a good Linus Tech Tips video about this
My experience with game framerates inside a Virtual Machine
I found that FPS-measuring software is not able to measure always accurately framerates inside a VM. When I was playing Horizon Zero Dawn, the FPS was always very low, around 26–36 FPS. But the game was running very smoothly, and I did not have any issues with shooting or playing, so it seemed that the FPS measurement was not correct. I tried a few more games, and found a similar situation.
Here’s a video of my gameplay where I use the Nvidia Data Center GPU Tesla K80.
At the time of writing, there are 43 videos, but I will add more later. (Let me know in YouTube comments if it’s good or bad, sluggish, and so on.)
Even though I was playing and testing with different GPUs (Tesla K80, GeForce 1090, GeForce 3070 Ti and GeForce 3090) and changing the Graphics settings between Off, Low, and Max, the FPS rate stayed the same. No matter what I changed, it had no effect on framerate.
When I played Crysis Remastered, however, increasing the framerate made a much bigger difference. When the framerate was under 60 FPS, shooting was difficult or even impossible—everyone killed me before I was able to shoot them.
Turning down the shadows brought the framerate up to 60 FPS, and I was finally able shoot someone! Here are the settings I used:
You can watch a complete gameplay experience here: https://youtu.be/bAw5bv21vCM (Let me know in YouTube comments whether it’s good or bad, sluggish, and so on.) This game was played on my Mac Mini M1 with its original 1GbE network adapter, before I upgraded to 10 GbE.
Here, I am testing Crysis Remastered with the Nvidia Tesla K80, at full-screen 5120 × 1440 resolution. (https://youtu.be/RyKy6KXGfUk) The game does not actually support this resolution, so everything is stretched out.
If you like to play a Deus EX: Mankind Divided with the Nvidia Tesla K80 over a 1 GbE network connection to the VM, turn shadows to low and MSAA off. Then the framerate is around 35–45 FPS.
Here is YouTube video showing the gaming experience: (https://youtu.be/KNq7wWB3C9M)
Different FPS-measuring software shows completely different framerates during a 3DMark Benchmark Fire Strike test
And now we get to the most interesting part. I have been using three different apps to measure framerate—FRAPS, GeForce Overlay, and XboxGame Bar. They do not all work in every game, so I was using what I could, and for some games I was able to use two framerate apps at the same time. FRAPS and Xbox Game Bar FPS would not work at the same time.
But back to the topic… this screenshot shows the different results—180 FPS from FRAPS in the top left corner, and 80 FPS from GeForce Overlay in the bottom right corner.
And here is the video of this Benchmark (https://youtu.be/mOMEPdoxe1Q)
I was planning to do a very specific different test to compare the VM gaming experience (vSphere+Horizon) vs. using a bare-metal server but, after finding out how wrong the framerate can be inside a VM, I didn’t see any point anymore. The test would be very time consuming and painful anyway, because it would involve turning my 1 ESXi server to PC. When I can get a testing bench, I’ll be able to do that kind of test very quickly, but for now it would be nice to find software that works accurately inside a VM.
Making these discrepancies even more science-fiction is that I cannot find any explanation behind them. After I bought a gaming monitor (LG UltraGear 34GP950G-B 34”), I noticed that the framerates in CyberPunk2077 and Horizon Zero Dawn increased rapidly. Yes, this photo and the previous ones show different GPUs in use but, even before, with the GeForce 3090 24G, the framerate was around 30 FPS.
Horizon Zero Dawn 31 FPS
Horizon Zero Dawn 84 FPS
How your PC-to-VM network speed affects FPS
I have a Mac Mini M1, which originally came with only a 1 GbE network card. After I upgraded with an OWC Thunderbolt Pro Dock with 10 GbE, game framerates increased by around 30 FPS, and, in some games, the mouse response seemed better. CyberPunk 2077 mouse response improved, and shooting improved a little, but overall the Tesla K80 struggles to handle shooting. What I found out is that the GPU must be tested during both shooting and action. Then you can see what the GPU is capable of.
Framerates can also be improved by expanding the network bandwidth. My ESXi servers have a 100 GbE NIC and, for the VM, I use the VMXNET3 Ethernet Adapater. VMXNET3 nominally runs at 10 Gbps but, based on my testing, it can get up to around 30 Gpbs.
1 GbE connection– Overwatch 60FPS
10 GbE connection– Overwatch 90FPS
This is because higher framerates and refresh rates need a higher network bandwidth (between the PC and the server). The situation is similar to that with display cables. For example, for a DisplayPort to support 4K resolution @ 120 Hz, we need at least DisplayPort cable type 1.3 or above to carry a data rate of at least 25.92 Gbps.
Take a look at the table below showing DisplayPort speeds, resolutions and refresh rates.
Now, you could wonder, would you get any performance boost by increasing to 25 GbE or even 100GbE network speed? The answer is NO. Let me explain why.
What is the maximum VMware Horizon Blast Extreme’s framerate?
VMware Blast Extreme’s framerate limit is 60 FPS; this is technical limitation of the Horizon protocol. Your games can do 120 Hz and your monitor can do 120 Hz, but the Horizon client is only 60 Hz. So, Horizon is the bottleneck.
The default setting is 30 FPS, but this can easily be changed using the Windows Registry Editor under
HKEY_LOCAL_MACHINE\SOFTWARE\Vmware, Inc.\Vmware Blast\Config
Increast the EncoderMaxFPS key from 30 to 60. If there is no EncoderMaxFPS key, then create it, set its type to REG_SZ, and its value to 60.
Here is mine. I set it to 100 just for fun and testing.
Is it possible to achieve 120 FPS with VMware Horizon?
The answer is yes. PCoIP can achieve 120 FPS according to VMware, but I have not tested this yet. I will let you know if there are any benefits over BLAST or not when I have got a free time to try it out. (Link: https://docs.vmware.com/en/VMware-Horizon/2203/horizon-remote-desktop-features/GUID-0AD7962F-22DC-4FC1-B31B-D48946BF1D47.html)
Use the Maximum Frame Rate value to manage the average bandwidth consumed per user by limiting the number of screen updates per second. You can specify a value between 1 and 120 frames per second.
With PCoIP I am not able to use Nvidia NVENC HEVC. If you are using Nvidia GPU Tesla or GeForce, though, you can use NVENC HEVC.
Passing through an Nvidia GeForce card or Tesla, can you use NVENC HEVC with Blast Extreme?
Yes. It works with all Nvidia GPUs (Tesla K80, GeForce 1060,3070 TI and 3090)
What are the NVENC HEVC limitations?
NVENC HEVC can do 120 FPS, and Nvidia builds their own streaming client—the GeForce Now—that supports 120 FPS from a VM. (https://www.nvidia.com/en-us/geforce-now/ and https://nvidianews.nvidia.com/news/nvidia-introduces-the-next-generation-in-cloud-gaming )
And there is another interesting way…
GeForce Card can be used for streaming to another computer/TV/ Tablet or over the Internet by using, for example, the Moonlight Open-source Client to play games. This client supports 90–120 FPS. (https://moonlight-stream.org/). The server side is built right into the Nvidia Drivers and GeForce Experience. The host is the PC, and Moonlight is the client.
I tested this and found the experience worse than with my Horizon Client. There is another problem as well—it is streaming my desktop and the features that are supported are what my desktop supports. So, if my monitor does not support HDR, then I can’t stream in HDR. My monitor resolution would be the streaming resolution as well. In other words, this does not help me much, because I do not get past Horizon’s limitations and I still have to login with Horizon to the VM, and then login with Moonlight Client. There could be another way, if I were to use it like people do on AWS or GCP. Maybe I will test this.
At the end of the github pages you can find installation tutorial videos. https://github.com/acceleration3/cloudgamestream I have not gone deeply into this, and do not have time for that at the moment.
Another solution that could be used is Parsec https://parsec.app/cloud-gaming
After this interesting side story, let’s come back to the Horizon Client VM.
What is the gaming experience like inside a Virtual Machine (VM)?
My humble opinion is that VM gaming feels like a high-end Gaming PC. For me, it could replace the Gaming PC. But I am not a pro gamer, nor I do not call myself a gamer. It would be interesting to hear pro-gamer opinions about what it’s like to sit at my computer and play over a VM. Also, if people need framerates above 60 FPS, then I think this is not for them. This is the only limitation at the moment, and there is another thing that was disturbing, that I will write little bit about later.
I have always believed that Microsoft Windows runs better on vSphere than on bare metal. The VMware guys have done a very good job and that’s the reason why it leads in the Hypervisor market.
But what was a big surprise for me is that, when installing games, I needed to install so many different company apps inside my Windows installation, and all of those needed accounts and so on. The Xbox App, then game maker apps, and some games even had multiple apps. Because of that, I felt that my Gaming VM was being filled up like some kind trash bin. Because of that, I feel the gaming must be done only inside a VM. I do not want to use my daily-use PC for games that need a hundred apps installed. Everything must be kept minimal and very organized.
I am actually considering creating a new YouTube Channel where I stream my gaming inside a Virtual Machine so you can see the performance and what it feels like to play games inside a VM. I have recorded tests with many games and uploaded those to YouTube.
I did not expect the performance to be so good; I was expecting higher mouse latency and worse video quality.
Here are 3DMark Benchmarks for Time Spy, showing my score in the range of a good high-end gaming PC, close to a Premium gaming PC. I have included the video (let me know in the YouTube comments what score you get with your gaming PC or Virtual Machine)
3D Mark Time Spy Nvidia GeForce 3090
3D Mark Time Spy Nvidia Tesla K80
3D Mark Fire Strike Nvidia Tesla K80
3D Mark Fire Strike Nvidia Geforce 3090
And here are videos of my VM 3D Mark Benchmark. I would get a better score if my CPU were faster. It runs at only 1.8 Ghz.
I have been testing all kinds of different games on different screens and at different resolutions. Mostly, I am now playing games at full specs. I have even tried out a full-screen 5120 × 1440 49” monitor. Need for Speed is fun to play with a 49” monitor. (Watch and let me know via YouTube comments what you think. Negative, positive or if you have some questions.)
Need For Speed Deluxe Edition GF 3070 TI 49inch monitor 5120×1440
Need For Speed Heat GF 3070 TI 49inch monitor 5120×1440
Star Wars Jedi Fallen Order GF 3070 TI 49inch monitor 5120×1440
Far Cry 5 GF 3070 TI 49inch monitor 5120×1440
Far Cry 5 GF 3090 49inch monitor 5120×1440
Outriders Gameplay Geforce 3090 5120×1440 (49″monitor)
Cyberpunk 2077
Is it possible to play all games inside VM?
Of course, I don’t know about every individual game, but only those that I have tried over these past few weeks. The answer, though, is NO. Apex Legends and Lost Ark can’t be played this way. They will be blocked and not allowed. And there are few more games I have not be able to play, but I am not sure whether is it because I am using a VM, or the game thinks my CPU is not powerful enough or does not recognize the Intel® Xeon® Scalable Platinum CPU. Like the GeForce Experience does not recognize the CPU.
Watch Dogs 2 plays well, but is letting me know that my CPU does not meet its minimum requirements.
I did not try very hard to find the solution, but here are few examples, and I found there are many people with same problem even though they are not running it inside a VM.
What did you not like when playing inside VM?
Because I am using Mac Mini M1, keystrokes that are meant to be in the game end up triggering Apple shortcuts, for example resizing a window, or hiding all applications, or the dock. This happens mainly when I am playing in Fullscreen, and often during the most important shooting times. Sometimes screen goes blank because I trigger screen resizing. I have not found a solution for changing those keyboard shortcuts with mouse combinations.
Here is good example:
What about power consumption from not shutting down?
Not many people talk about the minimum power that those cards consume. Usually, everyone speaks about the maximum. When the VM is not in use, it is not so bad at all. And with many games, they do not consume maximum power either, because all new CPUs and GPUs consume only as much power as they need to do the work. I do not have a monitor connected to the video card. Maybe, if I did, then it would consume more power. The big difference between the Tesla K80 and the GeForce 3090 was that most of the games took the Tesla K80 to 99% use, but the GeForce 3090 to only 40%.
Tesla K80 min power—30 W per GPU (Tesla K80 has 2 GPUs.)
Geforce 3090 24G min power—10W
Geforce 1060 min power—3W
What GPU you would recommend?
Definitely Nvidia. With AMD, the website is not so user-friendly, and you could have driver problems. With AMD, you cannot use VMware Horizon Client NVENC HEVC. I do have the AMD Radeon INSTINCT MI25 16GB HBM2, which is twice as powerful as the Nvidia Tesla K80, but I could not figure out how to get this working as a graphics card. It is only a computing card, and I could not find drivers. When I have a test bench, I can start experimenting on this.
Tesla K80 vs. GeForce 30×0
In my humble opinion, Tesla K80, for the price that I paid on eBay (115$ US) and the performance I’ve had, is a very good card. I was very blown away the performance, which was able to run those heavy, demanding games. In some games I did not notice any difference. For example, in Horizon Zero Dawn, no difference at all. CyberPunk 2077 showed no graphical difference but it was impossible to shoot and difficult to play, while just walking on a street or driving a car was fine 😊. The GeForce 3090 improves gaming performance. But, in Dying Light 2 and Far Cry 5, there is a clear graphical difference. GeForce 30×0 completely changes the game environment into something much more enjoyable and realistic. The grass, especially, looks more real. Just my opinion—GeForce is little bit too hyped-up, and I was expecting to see a bigger difference between the Tesla K80 and the Geforce 3090. The Tesla K80 does not support DLSS ray tracing. Tesla K80 reached 99% usage on most of the games, and GeForce 3090 only 40%. So, I have maxed out the Tesla K80 performance on most of the games, but not the GeForce 3090. Another reason why I could not see much difference is that my server motherboard and CPU support only PCIe 3.0. But the GeForce 3090 is a PCIe 4.0 card. The Tesla K80 is, technically speaking, two GPUs sharing one physical board, and it is possible passthrough each GPU to a different VM. So, totally, 2 VMs can use this GPU
Is GPU RAM size very important?
Based on my experience with the GeForce 3070 TI 8Gb, I can say it is very important. When I was testing with games, the performance was very bad. Even when the display was, graphically speaking, okay, many games were unplayable because, after few minutes someone, like, hit a break and everything froze for a second. Many games maxed out the memory usage. So, I would suggest, try to get as much GPU RAM as possible. I would say a minimum of 12GB, because my Tesla K80 has 12GB per GPU and was running games just fine. I returned the 3070 and bought a 3090 24Gb.
Another thing to consider is noise and cooling
GeForce cards are quiet, and they can handle their own cooling. Tesla cards rely on the server chassis. This could mean, you must spend extra money for fans, and keeping it cool can be challenging. GPU temperature can go up very quickly and, when it reaches its limit, it will shut down the VM. When you want to use the GPU again, you have to restart your server. Just restarting the VM would not help.
What you would do differently next time?
I would buy the most powerful GeForce or Quadro card, with at least 24GB RAM, that I could afford.
Quadro A5000 and above cards are good because they can be used for vGPU and graphics.
GeForce do only graphics
Tesla cards do vGPU, computing and graphics. When choosing a Tesla card, you must double-check what it supports.
With the GeForce and Quadro cards, I do not have to waste extra money on cooling, and my ears can enjoy the quiet from less fan noise. Newer cards use less power and they give better performance. Usually, the idle power is less than for older-generation models.
If you don’t have to worry about cooling or fan noise, then things are easier. But keep in mind that, when start buying quiet fans and 3d-printed ducts, they could cost as much as getting a quieter, newer, more powerful card. Also consider power usage costs in the long term. Sometimes it is smarter to buy a newer card because GPU power usages can just burn your money in the long run.
I do not know why, but I like the Nvidia RTX A2000. With this, is not possible to use a vGPU. Tesla T4 supports vGPU.
What you suggest if I would like to give it a try?
Start what you have; do not rush to buy. Test and play with what you have. See how it is working. If your servers run with a 1 gig network and you use HDD/SSD storage, I would not suggest going on this road. Only do it if you have a 10 gig network or higher and NVMe storage. Storage speed is important. I am using only NVMe storage and max out my PCIe 3.0, using an Intel® Xeon® Platinum CPU with a 100 GbE connection.
What are your ESXi server specs?
CPU: Xeon Platinum 28/56
RAM: 256GB
Storage: PNY XLR8 CS3040 M.2 NVMe Gen4 4TB SSD (Read: 5600, Write 3900)
Motherboard: Supermicro X11SPi-TF
Chassis: Be Quiet PURE BASE 500DX
Network: HP Mellanox CX455A 100GbE QSFP28 Network Controller PCIe x16
100GbE Switch: Dell S4112T-ON
What are your Gaming VM specs?
Would it actually make a big difference to have a real gaming monitor?
At the beginning I thought, yes, but now I believe not much. Of course, it would depend on what your current monitor specs are. The only advantages you will get are if your monitor supports true HDR10 and high color accuracy. I still believe that a response time of 1 ms (GtG at Faster) would be beneficial as well. But the Refresh Rate will not get over 60 because this is a Horizon limit, and I am not sure VMware will consider improving it, because the software is not meant for gaming—it is meant for thin clients and office work. Office work does not need a high refresh rate. In other words, your monitor might be able to refresh as 240 Hz, and your game also, but you would still get only 60 Hz.
And for the closing, enjoy watching my gaming videos and leave a comment on YouTube
Cyberpunk 2077
Dying Light 2: Stay Human
Fortnite
Mass Effect Legendary Edition
Overwatch
Outriders
Star Wars Jedi Fallen Order
Tom Clancy’s Ghost Recon Wildlands
Fart Cry 6
Far Cry 5