Training Convolutional Neural Network (CNN) models is difficult when there is a lack of labeled training data and no unlabeled data is available. A popular method for this is domain adaptation where the weights of a pre-trained CNN model are transferred to the problem setup. The model is pre-trained on the same task but in a different domain that has plenty of labeled data samples available. In a CNN model, we can rearrange the weights of a convolutional layer by permuting them along the input channel dimension. This work shows that certain weights that are learned in the pre-trained model work well in the problem setup when the weights are rearranged in this manner. Computing the set of all possible rearrangements of the weights is computationally intractable. This work proposes two algorithms to find a good rearrangement of the weights in reasonable computation time. The solutions from the algorithms perform equally well or better than fine-tuning in the domain adaptation between SVHN and MNIST data.