James’ first notable girlfriend was Emma Nielson or Emma King. He dated Emma for 6 years and broke up in 2003. This was when he met Anne Marie Duff and started dating. James and Anne denied to talk about their affair initially. In 2006, he married Anne in a private ceremony. No one was informed until James was asked a question in an interview about when he is planning to get married. He replied that he had already married his dream girl. They have a son together. James refused to talk about his son, the birth date and also did not inform that his wife was pregnant with his child. When his estranged father wanted to meet him, he was unmoved.

It is said that James is shy when it comes to facing the camera, when he is not acting and thus, he refuses to talk a lot about his personal details. There were a lot of rumors about their divorce when he stated that he would not work with Anne in 2009. He then stated that he did not want to share the screen with her, as it would put a lot of pressure on their relationship and they would become the target for gossips. In 2013, there were again rumors about their separation, when he was seen at a function, alone.

It was said that James dumped Emma to be with Anne and Emma was heartbroken as she did not see this coming. It was said that James did not mention the reason, when he broke up with her and when Emma came to know about the pair after a few months, he was heartbroken. A close friend of Emma said that she was very upset and angry with everyone around, when she came to know about James and Anne. When asked James about it, he said that he did not want to talk about it as the things between him and Emma were done, a long back. There was a rumor that Anne was angry when Angelina was demanding to have more intimate scenes in the movie with James. James then talked with her and took the next year off from acting to be with her to prove that there was nothing going on between Angelina and him.

Though James never talked about Angelina and the reason for taking a year off, the co-stars on his set assured that the reason for the break was Angelina as he was trying make things smooth between Anne and him. Anne never talked about her husband or her relationship with him. She was never known to have cheated on him with any secret boyfriend. With no strong evidences to support the rumors, and no indications from James and Anne, it can be considered that their marriage life is smooth without any problems and there are very less chances of their relationship to end up in separation, in the near future.