As Roose Bolton matter of factly reminds Theon Greyjoy in the book (not sure if this happened on screen and it's not that important) Robb Stark's cause was permanently lost the moment Theon took Winterfell. The eternal symbol of Stark authority had been captured and was later burned. Robb's heirs were supposedly dead. Theon's actions (and the Bolton secret backstab) were not only serious symbolic blows to Robb's cause, they also caused both Stark allies and enemies to pause and wonder about The Young Wolf's judgment. If Robb couldn't protect his own castle and smallfolk, how could he protect his retainers? If Robb misjudged Theon's loyalty, would he make other similar strategic wartime errors? Well, we know that Robb did indeed make horrible strategic blunders. Chief among these errors were trusting the Boltons. The Boltons ruthlessly exploited Robb's mistakes in order to realize their long dreamed of goal to supplant the Starks as the Wardens of the North. None of this would have been possible without Theon. And Theon knows it. He verbalizes in both book and show that he should have died with Robb at the Red Wedding, and that he sees Robb and Ned as a better brother and father to him than his own. By then it's too late. All of Westeros knows him as a traitor. Many people start using the alliterative appellation "Theon Turncloak". And that's the nicest thing they call Theon. He's despised and mocked throughout the land. A man who could have been the living symbol of reconciliation and/or alliance between the North and the Iron Islands became the ultimate icon of Iron Islander treachery. What went wrong?
Both Theon Greyjoy and Sansa Stark were forcibly separated from their families and made to live with people who either had been or were currently in a state of war with their respective relatives. Both Theon and Sansa had to worry, needlessly or not, that they, children or not, would be killed to send a message to their relatives. But that is where the similarities end. From pure sadism, the Lannisters made Sansa look at her father's and tutor's corpses. They regularly beat her, stripped her and mocked her. She couldn't leave and was kept under constant guard. For every setback in Lannister fortunes that occurred, the Lannisters made Sansa pay a psychological or physical price. Theon Greyjoy, although presumably he can't just sail back to the Iron Islands, has extensive freedom of movement in and around Winterfell. Although from time to time he's reminded, sometimes bluntly, that he's not in fact a Stark, he's considered loyal enough to carry weapons around Stark family members and retainers, including Robb, the heir. The Stark master-of-arms trains Theon along with Jon and Robb in swordplay and combat techniques.
Rodrik: For ten years you have been a ward of Stark.
Theon: Hostage and prisoner, I call it.
Rodrik: Then perhaps Lord Eddard should have kept you chained to a dungeon wall. Instead he raised you among his own sons, the sweet boys you have butchered, and to my undying shame I trained you in the arts of war. Would that I had thrust a sword through your belly instead of placing one in your hand.
At Winterfell no bored psychopath beats Theon for amusement or threatens him with rape. And although as a child Theon no doubt feared Ned Stark and his stern nature, from what we know of Ned it's unlikely that Ned would have executed an underage Theon for his father's actions. When Catelyn Stark shares her suspicions about Bran's "accident" and her plan of action with Robb and the Stark retainers, Theon Greyjoy is there to listen and pledge his fealty to the Stark cause. Would Catelyn have done that if she had serious doubts about Theon? Can you imagine Tywin, Tyrion and Cersei having a Lannister war council and letting Sansa sit in? Would anyone let Sansa carry weapons around Joffrey? Of course not. So while Sansa Stark was unambiguously a hostage, and a poorly treated one at that, Theon Greyjoy is something a little different or rather, something a little bit more. It's the confusion that both Theon and Robb shared over Theon's status which caused the actions which ultimately led to Theon's downfall and Robb's murder. There's no guarantee that Balon Greyjoy knows that Ned is generally opposed to killing children. In fact he may not. Ned and Balon weren't friends and didn't meet under pleasant circumstances. Ned Stark, Stannis Baratheon, Tywin Lannister and King Robert Baratheon suppressed Balon Greyjoy's rebellion. Ned Stark took Balon's last living son and heir, Theon, as a ward/hostage. This is pretty obviously meant to make Balon think twice about revolting again. But the secondary reason is that once Balon passes away, Theon Greyjoy, schooled in the Stark/Northern ways and raised as a "semi-Stark" ,would hopefully be a friendlier Lord to the North, the Starks and to all of Westeros than his father had been. But that's the long game. Even kind decent Ned hoped that Theon could be used to compel Balon to do things he might otherwise not do. He told Catelyn as much when he saw her in King's Landing: And from this day on, I want a careful watch kept over Theon Greyjoy. If there is war, we shall have sore need of his father’s fleet.
Again, Ned would probably not have executed Theon under any but the most extreme situation. But Ned was okay with letting Theon and especially Balon Greyjoy think that Theon could be in harm's way. So although Ned was obviously not the best game player, he at least understood that Theon was a piece to be moved and used. So while Theon is initially a hostage, when war breaks out, he's clearly something else. Although Jon Snow is not that friendly with Theon, Robb certainly is, while the younger Stark children apparently view him as a foster brother, if a somewhat annoying one. Nevertheless Theon saved Bran's life from a wildling attack and later covered himself with glory fighting at Robb's side during the initial battles with the Lannisters. It's Theon's actions and his seeming loyalty which make Robb forget that Theon is not in fact his brother and commit the mistake of sending Theon, bound to him by oaths, back to the Iron Islands to carry Robb's offer to Balon Greyjoy. The problem with oaths, as Jaime "Kingslayer" Lannister mused, is that they often conflict with good sense and with each other.
Robb put Theon in an impossible position. Although oathbreaking and treason are horrible sins, kinslaying is even worse. Theon's a psychologically fragile young man, who upon returning home to his family, is met with indifference by his uncle and retainers, mocked by his sister and openly attacked by his father, who views Theon's continued existence as an unpleasant reminder of his own past failings. In Balon Greyjoy's mind the Starks, not the Baratheons or Lannisters, were the primary source of his humiliations. Now that the Northern leader was dead/imprisoned and his inexperienced son had taken most of the Northern armies south, what better time to strike at an undefended North? Strategically, attacking the only other force who is willing to recognize you as an independent entity is extra special stupid but clearly Balon Greyjoy's resentments outweighed his good sense. And Robb's friendship with Theon outweighed his mother's warnings that giving something for nothing to Balon Greyjoy was a very stupid move. Although Catelyn Stark couldn't know that Balon had already written Theon off and was preparing for war, she did know that showing kindness or good faith to Balon Greyjoy was futile. When Theon returns home he's reminded repeatedly that his family thinks he's a failure, a weakling and a punk. His family questions his manhood and loyalty. Balon showers Theon with abuse, verbal and physical. Theon is shocked to learn that his father considers Theon's sister Asha (Yara in show) to be the Greyjoy heir. When he learns that his father intends to make war on the North, Theon does not feel himself capable of doing anything about it. Think about it. What would you have done in Theon's place? Would you have continued to argue against it to no avail? Would you have raised hands against your own father and sister? Would you have refused to participate and thus confirmed your family's low opinion of you? Would you have sent a warning to Winterfell? Theon had to pick a side.
Considering your father had already written you off as dead, kinslaying taboo or not, he might have been willing to bring that status about. Theon had no good options. He chose his blood family over his foster one and you know the rest. But in Westeros, this is not an unreasonable choice. In fact it's the only choice. Blood is always thicker than water. Theon's "betrayal" came as a shock because Robb didn't heed his mother's warning about the Greyjoys or consider the possibility that Balon Greyjoy might not be eager to bleed FOR the people who had suppressed his last rebellion and kidnapped his last son. Hindsight tells us that Theon's choices were the wrong ones, both morally and consequentially. But at the time, Theon's decisions made sense from his pov. He chose blood over friendship. He tried to cover himself with glory and impress his father by taking Winterfell. And he trusted the wrong people (his own men in the show, "Reek/Ramsay" in the book). The ultimate problem is that the Great Houses need to find a different way of enforcing peace and good behavior than taking hostages. Hostages, even long standing ones who are well treated, may have deep resentments. When Balon rebelled, Theon's use as a deterrent was at an end. Robb's naivete and Theon's desperate need to impress and be accepted led to disaster. It's ironic that Theon's confusion over whether he's a Stark or a Greyjoy led to him losing his identity completely and being turned into "Reek". Theon paid a horrible cost for his crimes. No one should endure what he's had to go thru. Is Theon evil? Well, he's killed children. But so has fan favorite The Hound. Theon's switched allegiances but again so has The Hound. Theon betrayed the "good guys" House Stark but would you have had loyalty to people who kidnapped you from your family? It's complicated. I hope that Theon has, no matter how rocky, a redemption arc. He's done evil things. But as Rodrick Cassel and Maester Luwin pointed out, Theon is more lost than evil. He continues to be a fascinating character.