A statement from him added: "This is not a knee-jerk reaction. I have thought long and hard about my stance, but my integrity prevents me from supporting anything which I do not feel comfortable with."
Cllr Milner said he voted Tory before the party introduced business rates, and as businessman became a Liberal Democrat when first standing for office. He said they then started to 'diminish' in the town so he turned to Labour.
He added that New Labour's policies were becoming increasingly similar to those of the Conservatives.
Cllr Milner said he would not consider becoming an Independent because he didn't want to be a 'maverick'.
His statement read: 'To fully support the residents you represent, I believe you must be part of a team'.
It added he shared many political views with his Conservative colleagues.
Cllr Milner said he had considered resigning and going back to the polls, but with just another year before elections he felt it would be a waste of tax payers' money. "I think a lot of my votes are personal votes anyway because I do a lot of work in the town," he added.
Cllr Milner said he would continue with existing projects and anything which came along in the future.
He hoped to stay friends with people of all political persuasions he had met despite his bizarre turncoat behaviour.
CONSERVATIVE: Chairman of the Skegness Conservatives Cllr Dick Edginton said he was delighted to welcome Coun Milner to an 'expanding team'.
He added: "I am delighted that Cllr Ken Milner has joined the Conservative group on both the town and district council. He is a very able and conscientious councillor.
"The fact that he feels unable to continue as a Labour councillor proves the Conservatives have the best policies to serve the people of Skegness and nationally."
LABOUR: Chairman of the Skegness Labour group Cllr Phil Kemp said 'no one is indispensable'.
He added: "Although it's a bit of a loss for us, Skegness Labour councillors will continue to work as a team and I don't think it will effect the way we are trying to improve the town for the of residents.
"It seems he may think his chances of being re-elected next May are better served with the Conservatives, which is maybe what he did when he joined us from the Liberals."