A shifting point of view in a story is for some strange reason usually taught as a thing to be avoided by writers, almost as if it has to do with incorrect grammar or at least something along those lines. Perhaps that's the crux of the problem right there—that they are often done incorrectly. Actually, I think incorporating changing POVs in stories is a wonderful lost art today.
Mark Twain was one of the more prolific writers to integrate changing POVs into his stories. They usually would start off with someone relating how he had met someone, often a stranger, and go into some detail about their meeting, and then at some point this stranger would launch into an anecdote and take over the POV for most of the remainder of the story, perhaps switching back to the original narrator's POV right before the end.
A good example might be Twain's short story, "The Dying Man's Confession". This story opens with Twain relating how he and a small company of friends were traveling by boat (presumably a large paddle-wheeler). Twain himself has some kind of errand to do at his destination and decides it should be done at night rather than during the day, so he asks his traveling companions if they will relent to get off the boat in Napoleon, Arkansas and layover for a bit. They protest this, so he goes on to give an account of what happened to him the previous year in Bavaria in order to show them why this strange errand would best be done at night.
After a few pages of this, we find Twain talking about a man he met in Bavaria who was confined to bed because of a severe bout of consumption. This man has a story to tell pertaining to something in Napoleon, Arkansas and the reason why Twain is doing this errand. The POV then switches to this man as he recounts his tale like this:
When I had been this Karl Ritter's daily and sole intimate during two months, he one day said abruptly:
"I will tell you my story."
Then he went on as follows:
I have never given up until now. But now I have given up. I am going to die. I made up my mind last night that it must be....
Nearly 2/3 of Twain's story is taken up by the account told by Karl Ritter. You'll notice that Twain left a blank line between the place where he quit narrating and where Ritter picks up the narration. This is a good way to reinforce to the reader that the POV is about to change. Later in the story, when Ritter is done talking and the narration is taken up again by Mark Twain, another blank line is inserted between the two POVs. This is the thing you must remember about changing POVs in a story. You need to make it plain and obvious that you're doing it, and doing it on purpose for effect. If the reader sits scratching his/her head wondering why someone else suddenly seems to be doing the narration, then you've done it incorrectly. Twain didn't really need the blank line, but it helped to reinforce his objective of making the changing POV known, and that's a good thing to do. Rather than simply use a blank space you might consider using three centered bold dots, each separated by a space:
I can think of many examples of Mark Twain using the art of shifting POVs, but the best example I know is a terrific novel by James Hogg called The Private Memoirs and Confessions of a Justified Sinner. I consider this to be one of the ten or twenty best stories in the English language. Its narration is mainly told from the POV of someone who refers to himself as "the editor." The first half or so of the book is called "The Editor's Narrative," detailing a spooky story about an evil doppelganger that took place starting in 1687 and ending around 1710. These events are apparently as the world remembers them. However, after this, the editor offers us a document that has come into his possession that is purported to be an autobiographical account of those same events told from the POV of the story's antagonist in real time. They throw a very different light on the story. This takes up nearly the second half of the book. Thereafter is yet another short section from the editor's POV again in the present day (when this book was written in 1863), and here he begins with a letter said to have been published in Blackwood's Magazine that same year—a letter written by one James Hogg. (The author of the book.) This lengthy letter tells about the exhumation of the antagonist's mummified corpse by two boys. Then the tale finishes with "the editor" attempting to determine the authenticity of Hogg's letter.
In real life that letter really did appear in the magazine in 1863, a year before the book came out. It's a chilling novel that doubles as a railing against Calvinism. (John Carey's 1981 Oxford Press version is the only one worth having.) It's a real testament to the cleverness of switching POVs in a story when done right.
So don't be afraid to do things differently. The very fact that authors are taught to avoid shifting POVs nowadays is in my opinion a very good reason to use them. Not only will it make you stand out from the crowd, but changing POVs can be both clever and engaging.