Chemistry deals with matter. Physics deals with matter, its motion, energy, and force. There is an obvious overlap between the two, especially in areas such as quantum chemistry and physical chemistry.
You can do a lot of chemistry without thinking much about the underlying physics. Mixing hydrochloric acid and sodium hydroxide will always give you salt water. Understanding why they react is important, and it is based on physics, but once we know why, we can rise above the physics and use the rules of chemistry to build new molecules or analyze crime scene data without using our understanding of particle physics.
Some physicists like to think that all of chemistry is physics, since physics determines everything that goes on in reactions between molecules. But that would be to ignore the value of higher level abstractions. A chemist can understand proteins as chains of amino acids, forming strings, sheets, and corkscrew shapes that interact to control the shape, and thus the function, of the protein. To try to understand protein synthesis by starting with quarks and photons would be to make it needlessly complicated. It would be like trying to understand a hurricane from individual air molecules. If what you need is a rain forecast, higher level abstractions will get you there with a lot less trouble.
The parts of physics that are not chemistry are huge. All of high energy physics, the study of what goes on inside nuclei, the study of gravity, and things like relativity are all physics.