There are lot of guidelines available on how to create a bootable USB stick with VMware installer. Some of them use manual copying e.g http://www.ivobeerens.nl or http://www.techhead.co.uk or use the UNetbootin tool in Windows to create a bootable USB stick: https://community.emc.com.
But one major thing is to be obeyed: The USB stick must be formatted in FAT32 file system and the partition must be made active. Formatting the USB stick in Windows, all of this is done more or less automatically, but if you have only Linux at hand, you must follow a few steps that are also described in the vSphere 5 Documentation Center:
2 Create a partition table on the USB flash device.
/sbin/fdisk /dev/sdb
a Type d to delete partitions until they are all deleted.
b Type n to create primary partition 1 that extends over the entire disk.
c Type t to set the type to an appropriate setting for the FAT32 file system, such as c.
d Type a to set the active flag on partition 1.
e Type p to print the partition table.The result should be similar to the following text:
Disk /dev/sdb: 2004 MB, 2004877312 bytes
255 heads, 63 sectors/track, 243 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 243 1951866 c W95 FAT32 (LBA)f Type w to write the partition table and quit.
3 Format the USB flash drive with the Fat32 file system.
/sbin/mkfs.vfat -F 32 -n USB /dev/sdb1