Domain adaptation is a fundamental concept in machine learning that addresses the challenge of applying models trained on one dataset to different but related datasets. When we have a source domain with abundant labeled data and a target domain with different characteristics, domain adaptation techniques help transfer knowledge effectively. Common examples include adapting computer vision models from daytime to nighttime images, or transferring medical AI models between different hospitals with varying equipment and patient populations.
Domain shift is the fundamental challenge in domain adaptation. It occurs when the probability distributions of source and target domains differ. There are three main types: covariate shift where the input distribution changes, label shift where the output distribution changes, and concept drift where the relationship between inputs and outputs changes. When a classifier trained on the source domain is applied to the target domain, the decision boundary becomes suboptimal, leading to poor performance due to the distributional mismatch.
Domain adaptation can be categorized into three main types based on label availability in the target domain. Supervised domain adaptation assumes we have labeled data in the target domain, making it the easiest scenario. Semi-supervised domain adaptation has only partial labels in the target domain, requiring techniques to leverage both labeled and unlabeled target data. Unsupervised domain adaptation is the most challenging case where no target labels are available, relying entirely on distributional alignment and other unsupervised techniques to bridge the domain gap.
There are four key strategies for domain adaptation. Feature alignment maps features from different domains to a common representation space. Adversarial training uses a minimax game where a feature extractor tries to fool a domain discriminator, leading to domain-invariant features. Self-training iteratively generates pseudo-labels for unlabeled target data and retrains the model. Domain-invariant representation learning extracts shared features while discarding domain-specific information, creating robust representations that work across domains.
The mathematical framework for domain adaptation is built on the target error bound theory. The target domain error is bounded by three terms: the source domain error, the H-divergence between domains, and the ideal joint error. The H-divergence measures the difference between domain distributions in the hypothesis space. By minimizing the source error through good training, reducing the domain discrepancy through alignment techniques, and controlling the ideal joint error, we can achieve better performance on the target domain. This theoretical foundation guides the design of practical domain adaptation algorithms.