Hello dear ethereum-community,
with this post, i want to share my journey to mining on a 6 GPU rig. I haven't found a complete guide, that suits my needs, so I figured I could write one myself (or at least try). This is a repost to Version 1!
This post is divided into a section where I'm explaining, what i did to make it work (I.) and a section of things that failed (II.) up to the point of success. At the end i will write down, what i did to increase the hashrate (III.).
I. How did i get it working
First of all, lets see what I bought:
- 1 x MSI Z170A GAMING PRO CARBON
- 1 x Intel Celeron G3930 2x 2.90GHz BOX
- 2 x 8GB Crucial CT2K4G4DFS8213 DDR4-2133 DIMM CL15 (overall 4x 4GB =16GB)
- 1 x 120GB Kingston SSD Now V300 2.5" SATA
- 6 x XFX 480 RS 8GB
- 1 x 1200 Watt Seasonic Prime Modular 80+ Platinum
- 6 x PCI-E Express 1x to 16x Extender Riser
Getting started:
After you have assembled everything except the GPUs and their risers. You should be able to boot the system into the BIOS. That is where we start. You have a keyboard and a DVI or HDMI cable in the corresponding output plugged into your motherboard. Prepare a USB thumbdrive with the newest BIOS (Version 1.7) provided by MSI (link). Plug that USB drive into the motherboard. Start the computer and press DEL on your keyboard to enter the BIOS. Once you are seeing the BIOS, you choose Advanced Mode (F7) and M-Flash. This will require the system to reboot into flash mode. The next window you see, will show a folder structure. Find your way to the file you've just put onto the USB thumbdrive. Once you've selected the file it will start flashing the BIOS. This was the very first step
Now we need a working bootable windows 10 pro usb-thumbdrive (at least thats what I use). I recommend rufus for creating such drive. Plug the bootable USB drive into the motherboard and boot it. I assume that the installation is a UEFI installation.
Once W10 is installed, make sure that all necessary drivers (i.e. Chipset-Driver) are installed (not those that are needed for the GPUs!). I've allowed RDP access on my miner to check if everything is alright on my main computer. You can find that option via System, Advanced System Settings, Tab: Remote, Check: "Allow Remote ....". Next setting we want to touch, is the "install drivers automatically"-feature. Go to System, Advanced System Settings, Tab: Hardware, Driver Installation Settings and choose No. If your Windows is updated and configured, we can proceed by downloading the AMD drivers. I am currently using the WHQL-Drivers version 16.10.1 (link). Just download it, don't install it yet. Shutdown your system.
Take one of your GPUs and plug it directly onto the 16x PCIe slot onto your motherboard. Pull the DVI/HDMI cable from the motherboard and plug it into the GPU. Boot your system. It should detect, that the display is now on the GPU rather than the iGPU and you should see the BIOS booting up. Change the setting: Settings, Advanced, Integrated Graphics Configuration, Initiate Graphic Adapter to PEG. Save the settings and reboot.
Once the system is successfully booted, go to your downloads folder where you've put the AMD driver installer and install the driver. I chose to install everything. Your display will flicker during this process. As soon as the installation is complete, try to start "AMD Settings" from the windows menu. If that worked we are good to proceed. (I uninstalled the iGPU via Device Manager at this point, but I'm not sure if thats needed). Reboot your system and enter the BIOS.
In the BIOS go to Settings, Advanced, PCI Subsystem Settings. Here you change PEG0 and PEG1 to Gen1 and enable "Above 4G encoding". Save the settings. The system will try to reboot now, but i shut it down via pressing the power button.
Now its time for the 5 remaining GPUs. Remove the GPU that is sitting on the PCIe 16x slot and plug all risers on the motherboard. Plug the GPUs into the risers and connect everything accordingly. Picture 4 shows my rig. My risers have a SATA to 4-Pin Molex cable, where they are pulling power from. The AMD cards are power hungry and they are not only pulling from the 8-Pin connector, they are also pulling from the PCIe/riser. So i did only connect two risers on one SATA power cable coming from the Power Supply (PSU). If you are connecting more than two risers you can feel the cable getting warm. I do not want that. The GPUs came with a 2x6-Pin to 1x8-Pin adapter. The PSU came with four cables that have two 8-Pin connectors. As you can see in the picture 5, I connected two of the mentioned cables to three GPUs.
Is everything set? Boot your machine, Windows will install the drivers for the remaining cards and you will see flickering for the next 5 minutes. Check the Device Manager if you can see 6 x RX 480. If yes, "high five"!, if not, let me know, I may have got an idea.
The next step is to download mining-software, I am using claymore's miner v9.1 at the moment. Once downloaded and extracted, go to the folder containing EthDcrMiner64.exe and search for the start.bat. Edit the file to your preferences. This is what my start.bat looks like:
setx GPU_FORCE_64BIT_PTR 0
setx GPU_MAX_HEAP_SIZE 100
setx GPU_USE_SYNC_OBJECTS 1
setx GPU_MAX_ALLOC_PERCENT 100
setx GPU_SINGLE_ALLOC_PERCENT 100
EthDcrMiner64.exe -epool eu2.ethermine.org:4444 -ewal 0xfD4d1B9ec7C75Af47787d3449d6e1f85c88B3eF1.Miner01 -epsw x -asm 0Double click start.bat and the machine should start working. Congratz, you are mining Ethereum! First thing I did was to open "AMD Settings" and reduced the power limit. Go to Gaming -> Global Settings and choose the first RX 480 Wattman. Scroll down until you see power limit and set it to -15%. Do that for all GPUs. Most systems should be stable at -15%. You can fine-tune that to your liking. I will explain more on that topic in section III.
II. What failed and problems I've encountered
This section is ment to be like a "PSA" or "TroubleShooting"-Section.
Which GPU is which?
First of all, GPU0 as shown in the Miner-Console, is not necessarily the card on the most right, or the card on the first PCIe-Slot. To determine which GPUx shown in the Miner-Console is which in reality, I disabled mining on them. You will see the fan stop spinning after a few seconds. I've then created a list with the shown name and their corresponding slot on my rig.
For example: GPU3: 6 P6 2100 -20. GPU3 is shown in the console, 6 is the corresponding number in WattTool (I explain later how to use WattTool to overclock), P6 is the self-given name of the slot on the rig. 2100 is the Mhz of the RAM and -20 is the PowerLimit.
If you press a number inside the Miner-Console, the GPU with the corresponding number will stop mining. As soon as you do that, the "Activity" in WattTool will reduce to ~0% and shortly after that, the fan will stop spinning. Thats how I determined which card is which!
Out of the 6 GPU's I've bought 2 were damaged. And also the first motherboard I've bought was damaged. I had to RMA them. How did I find out which GPU was broken: look for the log in the Claymore Miner-Directory. The very last lines will contain a "Hang-Message" (see below). The mainboard was totally non-functional, so that was easy to guess. It has debug-LEDs, none of them were on, even though the computer didn't boot. I was quite upset to have so many malfunctioning "new" computer-parts.
16 Gb RAM?
Since I had to RMA alot of the stuff, I haven't had all parts together when I started mining. I started with 4 GPUs. So I set everything up accordingly and it worked. As soon as I've gotten the remaining GPUs I've added them to the rig and booted. BSOD right after Windows was trying to install the drivers for the new cards (“ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY"). I tried some things, but nothing solved the Problem, then I've found a recommendation on some forum, that I should install more than 8GB of RAM. I did so. Unfortunately it didn't work. I got it working with the suggested method, which i described in part I. Basicly, install one card on PCIex16 and then plug all other cards in right after. Not one by one. Since it worked, I didn't research much further to explain "WHY", but im guessing Windows is allocating resources and there is no more space for other devices to claim the same amount, but don't trust me on that. ;-)
Remote Desktop Protocol and Hashrate?
I thought I had another broken GPU, because the hashrate of that card was always bouncing. It turned out, when i was checking my miner with the remote console from another pc, "http://<ip-of-miner>:3333" the hashrate was fine and stable. So I assumed, that it had to do something with the RDP-Session on the miner.
There are different RAMs on RX480
There are, as far as I know, three different RAMs installed on RX480s. Which RAM-Manufactor has built the RAM for your card is distributor specific! My cards have Samsung RAM. Other cards can have Elpida and Hynix RAM.
III. Advanced
In this section, I want to cover "advanced" things beyond the installation and troubleshooting.
Overclock / Tuning:
To increase my hashrate I eventually decided to flash the Bios of my cards and optimize their settings. The RX 480 cards have two BIOSs, which are accessable via a small switch near the power pins of each card (see the following picture).
Picture 1
The switch position in the picture is supposed to be the "silent"-position. It is towards the Display-Outputs.
So I shutdown my miner and switched all GPUs to the "silent"-position and rebooted it, then I backed up the BIOS of my cards. I used GPU-Z to do that.
Since I am always sceptical when it comes to downloading stuff from boards and video descriptions I didn't want to download the BIOS someone had prepared, I wanted to modify it myself. That is easier than I expected.
I copied the backup'd BIOS to a USB drive to have it safe, in case i would need to flash it again. Then I took another copy of it to make modifications to it. This is done with the tool PolarisBiosEditor.
Picture 2
You have to copy the value of the 1750 timings on the lower right of the PolarisBiosEditor to the 2000 timings and save it. Thats it. You have your own modified BIOS.
To flash the cards with the modified BIOS, I used ATIFLASH. I had to do it via console, because the program did not like 6 GPUs at once via GUI. This is the command to flash all cards with the given bios via cmd (run as administrator).
atiflash -pa biosname.rom
Now the cards have better/faster RAM timings. Its approximately a jump from 24 Mh/s to 28 Mh/s
This video helped me alot!
After flashing the BIOS i've made some tweaks to my GPUs via WattTool.
Picture 3
I've made changes to the yellow marked values in the picture above. First of all, its important to know, that Ethereum-Mining does not need a high GPU-Clock. It is much more RAM intensive than it is one the GPU. Thats why we can safe money by not wasteing it on a higher power bill. Reducing the Power Limit and the GPU clock itself reduces the power draw of the card.
Tuning the cards up to the sweetspot of maximum performance while being stable is a time-consuming task. All of my cards can run on -18 power limit, I would suggest you try reducing power draw first.
Set the state P4 on WattTool to 1100 and set the mV of P4 to P5-P7. For me, the value of P4's mV was around 1010-1070. Set the fanspeed to what ever you like, but keep in mind, a high rpm consumes more power than a lower one. This one depends on how hot it is where you live. I've set the Max Temp to 75°C, I'm assuming that the card will clock down once it reaches that temp, im not sure on this one. Reduce the Power Target as much as you can, -20 is the lowest you can do with this BIOS. Three out of my six cards can do -20. So the first step should be to go with -18 and see if it is stable.
If your rig is stable with those settings, you could go for an even lower Power Target. After that you should increase the Memory MHz. I increased it in +25MHz steps and once it got instable I reduced it by 5 MHz. By instable I mean a "Hang-Message" (see below).
These are my GPUs settings:
| Miner-Console | WattTool | Physical Slot | RAM MHz | Power Limit | Hashrate copied from console |
| GPU0 | 12 | P4 | 2040 | -19 | 28.546 |
| GPU1 | 0 | P5 | 2125 | -18 | 29.059 |
| GPU2 | 40 | P1 | 2125 | -20 | 29.290 |
| GPU3 | 6 | P6 | 2100 | -20 | 28.952 |
| GPU4 | 18 | P3 | 2065 | -19 | 28.582 |
| GPU5 | 24 | P2 | 2085 | -20 | 28.718 |
Hang-Message:
...
19:01:28:280 1030 WATCHDOG: GPU 0 hangs in OpenCL call, exit
...I've created a simple PowerShellScript. It reboots the computer once there is a "hang" message in the log file of the claymore miner.
$logfile = 'C:\Users\Miner01\Desktop\claymore\1494602312_log.txt'
Get-Content -Path $logfile -wait | %{if($_ -match "WATCHDOG: GPU"){shutdown /r /t 0}}Now you would just see the Miner console and an empty blue'ish PowerShell-Console.
I've had trouble setting up the Task Scheduler to start the miner via "start.bat". Once thats running reliable, I'm going to share that aswell.
Picture 5
UPDATED:
- in I: Using Claymore v9.1 ( -asm 0)
- in II: created
- in III: created
I hope you liked to read my story! I'm open for questions!
If you feel like you want to buy me a beer: 0xfD4d1B9ec7C75Af47787d3449d6e1f85c88B3eF1
Best wishes Dave2k