To ask for proof of there 'being' a 'physical world' is to frame the problem with respect to the fundamental conceits of philosophical ontology specifically and Western physico-materialism generally - i.e., it's necessarily all about both (a) entitative objects of reference ('things') and (b) some transcendental quality or state or characteristic inherent in all such 'things' we allude to as 'being' or 'existence'. This is self-defeating from the skeptical position to which it seems you've arrived.
Phrased another way - to ask for proof of a physical world:
- from a vantage point of at least partial acceptance of the notion the physical world as-it-is isn't the physical world as-a-given-observer-sees / engages-it, and ...
- on the same terms as the conceptual presumptions ('things'; 'being') underpinning the naive notion there is a physical world as-it-is and all-there-is ...
... is an exercise in futility owing to self-contradiction.
Unless you wish to totally abandon and deny the relevance of subjective / phenomenological experience, the way out of this conundrum isn't to embrace the conceits that obscure or bypass this relevance, but rather to figure out what may be framed within a subjectively-framed perspective to serve as the analogues / corollaries / surrogates for traditional ontology's fundamental conceits ((a) and (b) above).
One more point ... 'Consensus reality' isn't a product of the physical realm per se. It's a construct of the interpersonal / social realm devised and maintained through language.
... And it's even more vastly overrated than plain old physical reality ...