BIOS Update Bricked My Server: How I Recovered It

Posted on

I recently managed to brick my server motherboard by updating it to the latest BIOS version. The BIOS update itself completed successfully, but after rebooting, the server would no longer POST properly. It was getting stuck during initialization and never reached the operating system.

The motherboard is a Gigabyte MS33-AR0. Everything worked before the BIOS update. After installing the newer BIOS, it didn’t. Fortunately, this is a server motherboard, and server hardware gives you several recovery options that you normally do not have on a desktop PC.

I ended up testing different BIOS versions, BMC firmware, BIOS recovery, BIOS swap, CMOS resets, minimal hardware configurations and several different flashing methods. Eventually, I recovered the server by going back to an older working BIOS and then moving upward through the BIOS versions.

This is what I did, what worked, what didn’t work and the recovery process I would use if it happened again.

First, Understand BIOS and BMC

There are two important pieces of firmware involved here: the BIOS and the BMC.

The BIOS is responsible for starting the actual server. It initializes the CPU, memory, chipset, PCIe devices and other hardware before handing control over to the operating system.

The BMC, or Baseboard Management Controller, is basically a small independent computer built into the server motherboard. It has its own processor, memory, firmware and network interface. You can access it remotely over the network, normally through its IP address using a web browser, and on some systems also through SSH if SSH access is enabled in the BMC.

The BMC keeps working even when the main server is shut down. As long as the server is connected to power and the PSU is switched on, the BMC normally remains available. From the BMC you can power the server on, shut it down, restart it, open a remote KVM console, mount ISO files and install an operating system remotely.

This is one of the biggest advantages of proper server hardware. The operating system does not need to work, and in many cases the main BIOS does not need to work either. The server itself can be completely shut down. It only needs to be connected to power with the PSU switched on so that the BMC receives standby power.

How You Can Access the BMC

The easiest way is through the BMC web interface. The BMC has its own IP address. You enter that IP address into a browser and log in.

From there you can normally access hardware monitoring, logs, firmware information, power controls, KVM and firmware updates.

Another option is SSH. SSH needs to be enabled in the BMC first. It gives you command line access to the BMC operating system and can be useful for advanced troubleshooting and firmware recovery.

There are also management interfaces such as IPMI and Redfish that can be used for remote management and automation.

For BIOS recovery, the BMC web interface is normally the easiest place to start. SSH becomes more interesting when the normal web based firmware update mechanism cannot do what you need.

Different Ways to Install a Server BIOS

There are several ways to install a BIOS on a server motherboard.

BMC Web Interface

This is the method I prefer when the motherboard supports it. The main server can be shut down while the BMC remains online.

On my Gigabyte MS33-AR0, the BIOS package contains an RBU folder with an image.RBU file. This is the file used for BIOS updates through the BMC.

The process is straightforward:

  1. Shut down the server.
  2. Leave the power cables connected.
  3. Leave the PSU switched on.
  4. Log in to the BMC web interface.
  5. Open the firmware update section.
  6. Select BIOS as the firmware type.
  7. Select the correct BIOS update file.
  8. On my Gigabyte board, this is RBU/image.RBU.
  9. Start the firmware update.
  10. Do not remove power while the update is running.
  11. Wait until the BMC reports that the update has completed.
  12. Power on the server and check whether it completes POST.

BIOS or UEFI Interface

If the server can still reach BIOS setup, the motherboard may provide a built in firmware update function.

  1. Download the correct BIOS package for the exact motherboard model.
  2. Extract the firmware package.
  3. Copy the required files to a USB drive if necessary.
  4. Boot the server.
  5. Enter BIOS or UEFI setup.
  6. Open the firmware update utility.
  7. Select the BIOS image.
  8. Start the update.
  9. Do not interrupt the power.
  10. Reboot when instructed.

UEFI Shell

Another option is flashing from a UEFI shell. Server BIOS packages can include EFI flashing utilities and scripts. The Gigabyte BIOS packages I was working with contained AfuEfix64.efi together with Gigabyte supplied scripts.

  1. Download the correct BIOS package.
  2. Extract it.
  3. Copy the required files to a FAT32 USB drive.
  4. Boot into the UEFI shell.
  5. Find the USB filesystem.
  6. Navigate to the BIOS update directory.
  7. Run the vendor supplied update script.
  8. Wait for the update to complete.
  9. Follow the manufacturer’s shutdown or restart instructions.

I would use the flashing utility and scripts supplied with the actual BIOS package rather than downloading a random version of AFU and experimenting with commands.

Windows or Linux

Some manufacturers also provide Windows or Linux BIOS update utilities. These are useful during normal maintenance, but they do not help much when the server cannot boot an operating system.

For a server motherboard, I prefer using the BMC whenever possible.

BIOS Recovery Options

1. Clear CMOS

The first thing I would try is clearing CMOS. This removes the existing BIOS configuration from the equation.

  1. Shut down the server.
  2. Disconnect the AC power cables.
  3. Switch the PSU off.
  4. Wait until the BMC shuts down completely.
  5. Check the motherboard manual for the CMOS clear jumper.
  6. Move the jumper into the CMOS clear position.
  7. Leave it there for the time specified by the manufacturer.
  8. Return the jumper to its normal position.
  9. If necessary, remove the CMOS battery as well.
  10. Press the server power button several times while AC power is disconnected.
  11. Leave the battery out for several minutes.
  12. Reinstall the battery.
  13. Make sure the CMOS jumper is back in its normal position.
  14. Reconnect AC power.
  15. Switch the PSU on.
  16. Wait for the BMC to start.
  17. Power on the server.
  18. Check whether the server completes POST.

In my case, clearing CMOS did not fix the problem.

2. Reduce the Server to Minimum Hardware

The next step is removing everything that is not required for POST. This helps determine whether the problem is actually BIOS related or caused by another component.

  1. Shut down the server.
  2. Disconnect AC power.
  3. Switch the PSU off.
  4. Remove all GPUs.
  5. Remove NVMe drives.
  6. Remove SATA drives if possible.
  7. Remove additional PCIe cards.
  8. Remove unnecessary USB devices.
  9. Remove the TPM module if it is removable and not required for the test.
  10. Remove all but one memory module.
  11. Check the motherboard manual for the correct slot for a single DIMM.
  12. Install one known good memory module in that slot.
  13. Leave only the motherboard, CPU, one DIMM and required power connections.
  14. Reconnect AC power.
  15. Switch the PSU on.
  16. Wait for the BMC to start.
  17. Open KVM.
  18. Power on the server.
  19. Watch the POST process.
  20. Check the BMC POST history.

On the MS33-AR0, I used A0 for the single DIMM test.

The goal here is not to boot Windows or Linux. You only want to know whether the motherboard can initialize the CPU, memory and chipset.

3. Check the Full POST History

Do not rely only on what you see on the KVM screen.

In my case, KVM showed:

Chipset Initialize...

with POST code EE.

It looked like the motherboard had stopped at EE. When I checked the POST history, however, I could see that the system had continued through additional POST codes.

This meant the KVM screen was not showing the actual final point of the failure.

To check this:

  1. Log in to the BMC.
  2. Find the POST code or diagnostic section.
  3. Export the POST history if your BMC supports it.
  4. Start the server.
  5. Let it run until it stops or resets.
  6. Check the complete POST sequence.
  7. Look for repeating sequences.
  8. Check whether the machine resets and starts POST again.
  9. Record the final POST code.
  10. Compare it with previous attempts or a successful boot if you have one.

Be careful with generic POST code lists found online. Do not assume a generic AMI POST code definition is automatically correct for your exact motherboard and firmware.

4. Reflash the Same BIOS Through the BMC

If the BMC still works, try flashing the same BIOS again before moving to another version.

  1. Shut down the server.
  2. Leave AC power connected.
  3. Leave the PSU switched on.
  4. Log in to the BMC.
  5. Open the firmware update section.
  6. Select BIOS update.
  7. Select the correct vendor supplied BIOS file.
  8. Start the update.
  9. Do not restart the server or remove power.
  10. Wait until the BMC reports that the update has completed.
  11. Power on the server.
  12. Check whether it completes POST.

In my case, reflashing the newer BIOS completed successfully, but the server still did not boot.

This is an important distinction. A successful firmware flash does not necessarily mean that the BIOS version can successfully initialize your hardware.

5. Try an Older BIOS

This eventually became the important part of my recovery.

If the server worked before a BIOS update, find the last known working BIOS version.

  1. Find the BIOS version that was installed before the problem started.
  2. Download it from the manufacturer if it is still available.
  3. If it is no longer publicly available, contact the manufacturer and ask for it.
  4. Make sure you have the correct update package for your exact motherboard.
  5. Shut down the server.
  6. Leave AC power connected and the PSU switched on.
  7. Log in to the BMC.
  8. Open the BIOS firmware update section.
  9. Upload the older BIOS image.
  10. Start the update.
  11. Wait until the BMC reports that the flash has completed.
  12. Power on the server.
  13. Check whether it completes POST.
  14. If it works, record that version.
  15. Move upward through the BIOS versions one version at a time and test after each update.

This is what became particularly interesting in my case.

I initially tried R15 and R15 did not work.

Instead of stopping there, I went further back and installed R09. The server POSTed successfully with R09. I then started moving upward through the older BIOS versions and testing them.

My results eventually looked like this:

BIOSResult
R09Works
R10Works
R15 first attemptFails
R15 after moving up from older BIOSWorks
R21Fails
R23Fails

So R15 was not simply a BIOS version that worked from the beginning. On my first attempt it failed. I needed to go back to R09, get the server booting again and then move upward through the BIOS versions. When I installed R15 again after doing that, it worked.

That is how I eventually got the server running on R15.

This is also why I would not randomly jump between BIOS versions. If the latest BIOS breaks the server, find an older version that definitely works and move forward from there.

6. Update the BMC

Gigabyte support also provided me with a newer BMC firmware and recommended updating it before continuing with the BIOS testing.

My original BMC version was:

13.06.27

Gigabyte provided:

13.06.29

The update process was:

  1. Download the correct BMC firmware for the exact motherboard.
  2. Shut down the host server.
  3. Keep AC power connected.
  4. Keep the PSU switched on.
  5. Log in to the BMC web interface.
  6. Open the BMC firmware update section.
  7. Select the correct BMC firmware image.
  8. In the package Gigabyte supplied to me, the web update image was rom_v130629.ima_enc.
  9. Start the update.
  10. Do not remove power.
  11. Wait for the BMC to restart.
  12. Reconnect to the BMC web interface.
  13. Confirm that the new BMC version is installed.
  14. Test the BIOS again.

After updating my BMC to 13.06.29, I tested the latest BIOS again. It still failed in the same way.

That gave me useful information. The BMC was now updated to the version Gigabyte wanted me to use, but the latest BIOS still did not work with my configuration.

7. Use the BIOS Recovery Jumper

Many server motherboards have a dedicated BIOS recovery mechanism. On the MS33-AR0 it is called BIOS_RCVR.

The exact procedure depends on the motherboard, so check the manual first.

The general process is:

  1. Download the correct BIOS recovery package.
  2. Prepare a USB drive in the format required by the manufacturer.
  3. Copy the required recovery BIOS file to the USB drive.
  4. Shut down the server.
  5. Disconnect AC power.
  6. Locate the BIOS_RCVR jumper.
  7. Take a photo or record its original position.
  8. Move the jumper into the recovery position.
  9. Connect the USB drive.
  10. Reconnect AC power.
  11. Switch the PSU on.
  12. Power on the server.
  13. Watch the local display or KVM.
  14. Wait for the BIOS recovery environment to start.
  15. Follow the recovery instructions.
  16. When recovery is complete, shut down the server.
  17. Disconnect AC power.
  18. Return the BIOS_RCVR jumper to its normal position.
  19. Remove the USB drive.
  20. Reconnect power.
  21. Test normal boot.

In my case, the motherboard recognized that recovery mode was enabled and displayed:

Please wait into BIOS recovery

But it never actually entered the BIOS recovery environment.

So this method did not recover my server.

8. Try BIOS Swap

My motherboard also has a BIOS_SWAP jumper.

Some server motherboards have redundant BIOS functionality or multiple firmware regions. If your motherboard provides a documented BIOS swap function, it can provide another recovery option.

  1. Shut down the server.
  2. Disconnect AC power.
  3. Switch the PSU off.
  4. Locate the BIOS swap jumper.
  5. Take a photo of its original position.
  6. Change it to the alternate position.
  7. Reconnect AC power.
  8. Switch the PSU on.
  9. Wait for the BMC to start.
  10. Power on the server.
  11. Check whether the machine starts POST.
  12. If it does not work, shut the server down again.
  13. Disconnect AC power.
  14. Return the jumper to its original position.

BIOS swap did not recover my server either.

Do not start randomly moving service jumpers. Server motherboards can have BIOS recovery, BIOS swap, ME recovery, ME update and CMOS clear jumpers very close together. Know exactly what you are changing before touching them.

9. Recover Through the UEFI Shell

If the server can still reach a UEFI shell, you have another recovery option.

The Gigabyte BIOS packages I was working with contained an EFI firmware utility called AfuEfix64.efi together with Gigabyte supplied flashing scripts.

The general process is:

  1. Download the correct BIOS package.
  2. Extract the package.
  3. Copy the required update files to a FAT32 USB drive.
  4. Connect the USB drive to the server.
  5. Boot into the UEFI shell.
  6. Identify the USB filesystem, for example fs0:.
  7. Change to that filesystem.
  8. Navigate to the BIOS update directory.
  9. Confirm that the vendor supplied EFI flashing utility is present.
  10. Confirm that the vendor supplied update script is present.
  11. Run the vendor supplied script.
  12. Wait for the flash to complete.
  13. Follow any shutdown instructions included with the BIOS package.
  14. Remove the USB drive.
  15. Test the server.

I would not create my own AFU commands unless I knew exactly which firmware regions they were going to modify.

10. Recover Through BMC SSH

If the BMC web interface cannot perform the recovery you need, SSH may provide another route.

This is a more advanced recovery method and depends on the BMC implementation. Do not assume that commands from another server motherboard will work on yours.

First, SSH must be enabled on the BMC.

  1. Log in to the BMC web interface.
  2. Check whether SSH access is available.
  3. Enable SSH if required.
  4. Find the BMC IP address.
  5. From another computer, connect to the BMC:
ssh username@BMC_IP
  1. Log in using the BMC credentials.
  2. Look for BIOS related firmware utilities.

For example:

find / -name "*bios*" -type f -executable 2>/dev/null
  1. If you find the appropriate utility, inspect its available commands:
/path/to/tool --help
  1. Determine which firmware file format the utility expects.
  2. Confirm that the utility and procedure are intended for your motherboard and BMC firmware.
  3. Transfer the correct vendor supplied firmware image to the BMC if required.
  4. Run the vendor documented flashing command.
  5. Wait until the process completes.
  6. Power cycle the host if required.
  7. Check whether the server completes POST.

I would not use undocumented force, skip signature or similar options simply because a firmware image is being rejected.

A rejected firmware file can be the wrong format, an image for a different platform or a raw firmware dump rather than the signed update package expected by the BMC.

11. A Raw BIOS Backup Is Not the Same as a BIOS Update Package

Before the problem happened, I had created a full BIOS dump through the BMC.

My backup was a complete 64 MiB SPI image.

That backup was useful, but the BMC would not accept it as a normal BIOS update. The BMC expected the correct signed update package.

If you have a raw BIOS dump:

  1. Keep the original file unchanged.
  2. Make several copies.
  3. Calculate a SHA256 hash.
  4. Store the hash together with the backup.
  5. Do not modify your only copy.
  6. Do not assume renaming the file turns it into a normal BIOS update package.
  7. Keep the image in case direct SPI recovery becomes necessary later.

A raw SPI backup and a normal BMC BIOS update package serve different purposes.

12. Contact the Motherboard Manufacturer

This was one of the most useful steps in my recovery.

I contacted Gigabyte and explained exactly what had happened. They supplied a newer BMC firmware and several older BIOS versions that I could test.

When contacting the manufacturer, provide useful technical information instead of simply saying that the server does not boot.

Send them:

  1. Exact motherboard model.
  2. Current BIOS version.
  3. Previously working BIOS version.
  4. Current BMC version.
  5. POST behavior.
  6. Last POST code if known.
  7. Whether BMC is accessible.
  8. Whether KVM works.
  9. Whether BIOS flashing through BMC still works.
  10. Which BIOS versions you have already tested.
  11. Whether CMOS reset was tested.
  12. Whether minimum hardware configuration was tested.
  13. Whether BIOS recovery mode worked.

If you need an older BIOS, specifically ask for the older BIOS versions in the correct BMC flashable format.

In my case, getting the official older firmware from Gigabyte was much more useful than trying to force an unsupported firmware image into the motherboard.

13. External SPI Programming

If the BMC cannot recover the BIOS, the recovery jumper does not work, BIOS swap does not work, UEFI cannot start and there are no other vendor recovery methods available, the last option may be programming the physical BIOS flash chip directly.

This requires an external SPI programmer suitable for the flash chip on the motherboard.

The process should be approached carefully:

  1. Shut down the server.
  2. Disconnect all AC power.
  3. Locate the BIOS flash chip.
  4. Read the exact chip markings.
  5. Find the chip datasheet.
  6. Confirm its operating voltage.
  7. Determine whether the motherboard uses one or multiple firmware flash chips.
  8. Select a programmer that supports the chip.
  9. Connect the programmer correctly.
  10. Read the existing flash.
  11. Save the dump.
  12. Read it again.
  13. Save the second dump.
  14. Read it a third time.
  15. Calculate hashes of all three dumps.
  16. Confirm that all three reads are identical.
  17. Make multiple backups of the original firmware.
  18. Prepare the known good firmware image.
  19. Write the image to the flash.
  20. Read the flash again after writing.
  21. Verify that the written data matches the intended image.
  22. Disconnect the programmer.
  23. Return the motherboard to its normal configuration.
  24. Clear CMOS.
  25. Reconnect power.
  26. Test POST.

If repeated reads from the chip do not produce identical files, stop. Do not write anything until the reading problem has been solved.

External SPI programming would be my last recovery method, not the first one.

My BIOS Recovery Troubleshooting Order

If another server stopped booting immediately after a BIOS update, I would now troubleshoot it in this order:

  1. Check whether the BMC is still accessible.
  2. Record the BIOS and BMC versions.
  3. Check KVM and the complete POST history.
  4. Try reflashing the same BIOS through the BMC.
  5. Clear CMOS completely.
  6. Reduce the server to the minimum hardware configuration.
  7. Find the last known working BIOS and install it.
  8. If it works, move upward through BIOS versions and test them one at a time.
  9. Update the BMC if the manufacturer recommends it.
  10. Try the motherboard’s BIOS recovery mode.
  11. Try documented BIOS swap functionality if available.
  12. Use the vendor supplied UEFI flashing utility if the server can reach UEFI.
  13. Investigate BMC SSH recovery if the platform provides the required firmware utilities.
  14. Contact the motherboard manufacturer and request the required older BIOS packages.
  15. Use an external SPI programmer only when the normal recovery methods are exhausted.

How I Finally Recovered My Server

In my case, the BMC was the key to recovering the motherboard. The main server could not complete POST, but the BMC remained fully operational. I could access it remotely, use KVM, read POST history, control the server’s power and continue flashing BIOS firmware even while the host itself was shut down.

Gigabyte provided me with BMC version 13.06.29 and several older BIOS releases. I updated the BMC and tested the latest BIOS again, but the server still did not boot.

I then started testing the older BIOS versions. My first attempt with R15 did not work. I went further back to R09, and the server finally completed POST again. From there, I started moving upward through the older BIOS versions and testing them instead of jumping directly back to the latest release.

R09 worked. R10 worked. When I reached R15 again, R15 now worked as well. R21 and R23 did not work with my configuration.

That was how I recovered the server. I did not simply downgrade directly to R15 and get a working machine. I first had to go back to R09, establish a known working BIOS and then move upward until I reached R15.

The biggest practical lesson for me is what I will do before the next BIOS update. I will keep a copy of the currently working BIOS package, record the current BIOS and BMC versions, download any older firmware I may need and make a full BIOS backup if the BMC supports it.

If a new BIOS turns a working server into a machine that cannot POST, having the previous firmware already available makes the recovery much easier.