I hope this helps you . EXE custom actions show this error when their return value is not 0. So make sure that your BAT file doesn't return a non-zero value. Also, check if your command line works correctly on older Windows versions.
How to stop an .MSI installation Using a wix custom action
like below fixes the issue If your checks can not be done using Condition element then in your Custom Action you should return ActionResult.Failure if your checks fail or return ActionResult.Success if the checks pass. Returning Failure should cause your setup to go in RollBack mode and it will not progress any further. This will work if the Return attribute of CustomAction element is set to check which is the default.