Unfortunately, Atmel does not ship a hex file for their STK500 development system that could be used to update the firmware using 3rd party tools like avrdude. Instead, an encrypted ebn-file comes with the latest Atmel Studio. The tools that is used to flash the firmware onto the ATmega8535 is a bit dull, though.

Preparing serial connection

First, hook up the DB9 port on your STK500 that is marked as RS232 CTRL to your PC. Since the flash tool AvrProg.exe only searches for devices on COM1, make sure that you connected it to COM1 or if you use a USB-to-Serial converter, change its address to COM1 (Device Manager → Right-click on the COM port you wish to reassign → Properties → Port Settings → Advanced → COM Port Number → COM1). The port settings should read as 9600 8N1.

Preparing the STK500

Now, power off the STK500 and make sure, the push button labeled PROGRAM is pressed while turning on the power again. The LED labeled STATUS shouldn’t be lit permanently (like it would in normal operation mode).

Flashing the firmware

Startup Atmel’s flash tool AvrProg.exe that is installed along with Atmel Studio. The default installation location is %ProgramFiles%\Atmel\AVR Tools\AvrProg\AvrProg.exe.
%ProgramFiles% is localized and depends on your architecture. It’s most likely going to be C:\Program Files or C:\Program Files (x86).

Load the firmware file using the ‘Browse’ button and flash it.

Next, load the STK500.ebn firmware file by pushing the Browse button. The default location is %ProgramFiles%\Atmel\AVR Studio 6.0\tools\STK500\STK500.ebn. After hitting Flash → Program, a progress bar should appear and shortly after the status indicator should tell you that the devices has been successfully programmed.

Device was successfully programmed.

If you restart the STK500 by turning off the power and turning it back on again, the firmware should be up and running. You can double-check with avrdude, for example.

$ avrdude -p atmega32 -c stk500v2 -P /dev/ttyUSB0 -n -v
[...]
         Programmer Type : STK500V2
         Description     : Atmel STK500 Version 2.x firmware
         Programmer Model: STK500
         Hardware Version: 2
         Firmware Version Master : 2.10
         Topcard         : Unknown
         Vtarget         : 5.1 V
         SCK period      : 10.9 us
         Varef           : 5.0 V
         Oscillator      : 3.686 MHz