Speaking of canon in fiction

I’m actually thinking about breaking canon within the novel I’m writing right now. Different chapters will state things that don’t match, on purpose of course. Subtle at first, then more and more. It’s crazy, I know, it shouldn’t be done, but my brain works this way… for some reason. What do you think, is it crazy, should I stop this idea right now? Or did someone say genius?