Thank you for all the information. We've finished the vCenter upgrade to 5.5 now, so I tried converting to the vCenter instead of directly to the ESXi host. Still, same error.
I tried increasing the timeout interval to 10 minutes, but it didn't help.
I then tried editing the vpxa.cfg like the KB said, but that gave me some trouble with the service not starting afterwards for some reason. I couldn't figure out why, so instead of wasting time on troubleshooting I just reinstalled the ESXi and started conversion with volume splitting instead. I create 2-3 jobs on the same server with different volumes and run them parallel, then it works just fine. I guess that'll do for now.. 