Pastor responds to backlash amid lavish anniversary gift to wife