In this situation it is appropriate to conduct two analyses. One analysis would compare offers to applicants, and would include those offers which were denied. A second analysis would compare hires to applicants, not including those applicants who denied offers. I have seen cases in which several offers were made to minority and/or female applicants and the offers were declined; the ultimate hire was a nonminority male. If there is adverse impact in hires (using a hires-to-applicants analysis), there may not be any adverse impact when all offers are included (using an offers-to-all applicants analysis).
You can use this OFCCP audit checklist to ensure you're doing what is required to maintain OFCCP's regulations including VEVRAA, Section 503, and EO 11246. Or request a demo to streamline your compliance and recruiting efforts.